Luca Matteis

Apps Script lets you build powerful HTTP APIs

Wed, 12 Dec 2012 06:03:00 -0500

I heard of Apps Script a while ago. Sometimes an interesting news item appears on my Google+ feed, showing some cool functionality that you can achieve with these so called “scripts”. I never dug deeper than that, I felt it was just some automation that you could do with your spreadsheets, it was nothing special.

I couldn’t be more wrong.

You write server-side JavaScript that runs on Google’s servers. You even get a nice HTML5 interface for writing such code. This script can do a whole lot of things. It can use most of Google’s APIs: CloudSQL, BigQuery, Spreadsheets, you name it. The APIs are quite simple to use. The very cool thing is that you can choose to execute such script using a simple HTTP request, and it will respond promptly. In essence a Script can become a tiny web server.

For example, imagine I have this code in my Script:

function doGet(e) {
  var output = ContentService.createTextOutput()
  output.setMimeType(ContentService.MimeType.TEXT)
  output.setContent('Hello World!')
  return output
}

Whenever you save the project, you get a URL you can use to make this script run (such as https://script.google.com/macros/s/AKfy…).

As you can see the doGet(e) function is called whenever my script is accessed through an HTTP GET request. The e argument contains information about the request (query parameters and such). And in the content I’m simply setting a mime-type for the response - this will be set directly in my HTTP response header. You can of course use doPost(e) as well for incoming POST requests; yes you can upload files too!

Storage

Cool so now we have a little HTTP server that can answer to our requests, and can do all kinds of funky things with Google’s services. Let’s try and deal with persistency a bit. Like say we want to store some data, so that subsequent requests can access this information!

Well that couldn’t be easier thanks to ScriptDB, a minimal API Google has available specifically for this functionality. It’s basically a JavaScript object storage - any data-structure available in JS can also be stored in ScriptDB. I love it! I feel like all storage abstractions should directly map to the language’s standard data-structure. Here’s an example of how ScriptDB works (yes, that’s semicolon-less JavaScript):

var sam = {
    name: 'Sam',
    age: 33,
    hobbies: ['soccer', 'movies'],
    son: {
      name: 'Dimitri',
      age: 18
    }
}
var db = ScriptDb.getMyDb()
db.save(sam)

It’s important to note that each script gets its own database, so the persistency happens at the Script’s level. The limitation are a bit annoying. Only 50mb for regular account. But I think there’s ways to increase that, like if you get a Google Apps account. This is not meant for any binary data - there’s Google Drive for that - so 50mb might be enough for most apps.

Of course if you need a large database, then BigQuery would be a better fit. Personally I prefer the simple API approach of ScriptDB.

Let’s go a bit deeper

Well that was easy! We created a web-server that stored stuff in Google’s database. What now? How do we use all these goodies? One thing that comes to mind is that I’d want to build a JSON API of some sort. So let’s start by outputting some JSON!

function doGet(e) {
  var output = ContentService.createTextOutput()
  output.setMimeType(ContentService.MimeType.JSON)
  var content = { Foo: 'Bar' }
  output.setContent(JSON.stringify(content))
  return output
}

So whenever I access my deployed Script URL I should get a nice JSON (with correct mime-type headers): {“Foo”:”Bar”}

This is very useful, but wouldn’t it be more awesome if I could use this data directly from within my HTML website? Unfortunately I cannot directly make an Ajax request to this Script’s URL because it lives on Google’s domain, and cross-site Ajax requests are a well known security issue, and are not allowed in web-browsers.

Don’t worry, JSONP to the rescue. If you wrap the JSON inside a regular function, this data can be fetched directly within your HTML page, no matter what domain it is on.

var callback = e.parameter.callback
var jsonString = JSON.stringify(content)
output.setContent(callback + '(' + jsonString + ');')

And then you call your Script URL adding the ?callback=test query parameter: https://script.google.com/macros/s/AK../exec?callback=test

Abstractions getting higher and higher

So we can write HTTP JSON APIs using server-side JavaScript and the richness of Google’s Services. This is mind blowing. The easiness of use and the complexity that it abstracts, is outstanding.

This allows us to build really complex apps, with less and less effort. I imagine myself concentrating on the user interface of my product. Building all the HTML/CSS that I need, and going deep on the user experience without worrying much about the backend. When I need persistency I can simply write an Apps Script, with a JSON API, and immediately feed into Google’s most powerful services.

Comments on Hacker News

Lisboa!

Wed, 03 Oct 2012 12:49:00 -0400

Lisbon was great. I went there to attend LXJS, which turned out to be a really awesome conference about JavaScript. What I liked about the conference is that it was very casual, but at the same time it had lots of quality speakers, the top-notch of the Node.js community.

Lisboa is a small city compared to Rome, but it delivers very efficiently. A public transport system that allows you to reach most of the city with a daily pass of just 5 euros.

They really had great food at this bar. Typical Portuguese cuisine

Some skateboarders and an interesting pink guy

Portuguese beauty? Maybe not Portuguese :)

Got my new toy today. A Fujifilm X10. Was lucky enough to shoot...

Mon, 17 Sep 2012 15:09:00 -0400

Got my new toy today. A Fujifilm X10. Was lucky enough to shoot a praying mantis laying on my mosquito tent. It was actually praying!

Forget About The Backend Stack

Mon, 17 Sep 2012 08:18:00 -0400

I think I’ve shown how important user interface is for me. Most of my web development happens in the Browser. This is essentially where I concentrate most of my time, trying to make my interface work well on all platforms and devices.

When I’m finished with this part, then I have to worry about the data, the authentication, and everything that revolves around persistency. I rely on a backend - which is basically a server located somewhere - for the state of my application.

And this sucks, because now I have to go find a good server where I can keep all this state on. Sometimes I use App Engine, which works great and it’s mostly free, sometimes I use Heroku running some Node.js server.

But it’s still a different code base which I have to maintain. And it’s kind of unrelated to my app if you think about it. My app is my interface, which I’ve built using HTML/CSS and JavaScript. I shouldn’t have to worry about how to develop and maintain all of the state for my app in another codebase - my interface already manages that.

With powerful frontend frameworks such as AngularJS and BackboneJS, I can literally build the entire logics of my app using very well engineered frontend code. I can abstract the state of my app using Models, and allow all of this logic to exist directly in my browser, and not somewhere on my backend.

This way I can literally choose different backends for different purposes. For example, geo-spatial data is better stored on an instance of GeoCouch. My frontend app will therefore know how to communicate with this instance whenever it needs that type of data. I do the same for full text search. I have a backend setup just for this - and I query everything I need using old school Ajax, which is of course abstracted underneath an MVC (Model-View-Controller) layer.

Thanks to new tech coming along such CORS and JSONP (ok not so new) I think this is quickly becoming the new face of web development. Allowing you to forget about your backend stack, and concentrate more on your frontend, which is usually the thing you should be worrying more about anyway.

Comments on HN

Being a perv

Fri, 14 Sep 2012 08:56:00 -0400

Being a perv

New Camera

Sun, 09 Sep 2012 07:23:00 -0400

I’ve been looking for a new camera. The one I’ve been using on my current iPhone 3gs is bad. It can take decent pictures with good light, but in places with little light, it’s just really bad. The new iPhone (5) is going to come out next week, and that’s going to have a pretty decent camera on it - I’m not looking for anything fancy, just as long as it can take decent pictures, I’m happy.

Zeus shot with my iPhone 3gs

Interesting thing is that you can also buy lenses for the iPhone, which will dramatically improve picture quality.

That seems like quite an exaggerated lens, but it’s certainly doable to use your iPhone to take decent quality pictures.

Cibarsi.com

Thu, 06 Sep 2012 05:07:00 -0400

After about a couple of months of really intense work, I’m happy to show off one of my latest creations: cibarsi.com

Essentially it’s a site that allows small restaurant owners to build their own site, without any technical skills. Think of it as about.me for restaurants. The issue is that if you own a restaurant and you want to communicate and interact with your online users, you need a website.

But the whole “building a website” scenario sucks. You’re stuck with having to contact professionals to build it, and you end up spending thousands of euros in the process. You end up with something that you basically have no control of, and every time you want to make specific changes to your menu or anything else, you have to waste time and energy to contact your “webmaster”.

(Little sign I found in a small restaurant in New York. This is when the idea of Cibarsi sparked in my head)

Cibarsi gets rid of all this extra boilerplate and puts you in charge of everything. With a very basic interface you can edit your menu, change the color schema, add new images, etc. And in the end you get this very basic webpage where you can show off your restaurant and make it visible to your online clients.

It’s an Italian thing, and restaurants over here are a big deal. It means that there’s lots of money floating around in the restaurant community. And for small restaurant owners to spend 5-10 euros a month on something they have total control of, is a pretty good deal.

Awesome trip on top of the Vesuvius with some friends! The ride...

Tue, 04 Sep 2012 03:31:00 -0400

Awesome trip on top of the Vesuvius with some friends! The ride up the hill was not that bad. Then we also ended up visiting Pompeii.

Comparing The Best Agile Tools

Tue, 31 Jul 2012 07:51:00 -0400

Recently I’ve been trying to track down some tools that would help me and my team with the process of building software. There’s so many things involved with development, that it gets really easy to forget to do things.

Sometimes you simply get lazy and skip certain steps - maybe you forget to test certain parts of your app, or you write a feature just for the sake of getting it out there, without any proper architecture behind it.

This is when I think Agile Methodologies become really important. I’d like to think of them as a set of rules that you follow in order for you (and your users) to stay happy.

The idea is that you organize yourself, and your team, so that these rules became a daily routine. That everything flows and becomes this transparent thing in your company, where you just do it without much effort. Similarly to how you got used to opening your Text Editor to write something, you’ll have to get used to Agile Methodologies to build software.

As for anything that you’re trying to learn, the best idea is to lessen the burden and give yourself a basic environment to help you train in. Agile tools give you this workflow, with rules to follow in order to build your app, so it’s a good start to choose a good tool with solid features.

I know some of you are probably saying that being Agile isn’t about the tools you use but about how you do things. However, as an agile beginner myself, I think using something with strict workflows involved, would help me, and my colleagues, switch to the agile mindset more easily.

Tools available

There are so many tools available that it would’ve taken me years to review all of them, so instead I decided to obtain some of the best names in the industry, and see what they have to offer.

Pivotal Tracker. They provide a very guided workflow that emphasizes rapid delivery and lots of feedback. Each iteration allows you to show your clients exactly what has been delivered for that given sprint. This level of guidance is important to become trained in continuous deployment and allow it to be part of your development process. It gives you a backlog to organize user stories - these are requirements expressed from the customer’s point of view (not from yours) - and a very intuitive interface for managing their priority.

GreenHopper (Atlassian). Offers same SCRUM principles as Pivotal. Also very guided sprints. It seems like it has less features compared to PT, but overall they’re quite similar. You can export very nice graphs to visualize team process and identify bottlenecks.

Blossom. This seems quite promising. It’s in its early stages. Can’t seem to find much about how it manages SCRUM. I can’t find a way to setup sprints, so this feature is probably left for you to setup, which isn’t ideal.

Basecamp. This tool looks amazing for organizing your high-level work. But it doesn’t go into the details of actual coding and, for this case, setting up sprints. Ideally an agile tool should provide mechanisms to guide you through the SCRUM process, so not being able to satisfy this need is a negative aspect if you’re looking for something with very guided workflows.

Trello. They simply provide a nice virtual board where you can drag items around and it kind of puts everybody on the same page. This is nice, but again, if you want a guided workflow for SCRUM, this doesn’t give you much.

GitHub. Like Basecamp and Trello, GitHub leaves you in charge of setting up your own workflow for doing SCRUM.

Low level comparison

If you’re looking for something with really guided workflows, my list breaks down to just two tools.

            Pivotal     GreenHopper
            Tracker     (Atlassian)

guided
workflow       x             

user
stories        x             x
(bug 
tracking)

backlogs       x             x

manual
sprint                       x

user
assignment     x             x

automatic
sprint         x
calculation

charts         x             x

estimating
stories        x

git/svn
integration    x             x

GitHub
integration    x

third-party             (can only integrate
integrations   x         with Atlassian tools) 

            (through        (through Atlassian
help-desk   third-party         tools)
            integrations)

All in all, Pivotal Tracker just seems to be much more of a complete product. The video on their website shows you in great detail the process of managing stories and turning them into working software.

With weekly iterations and constant communication between developers and product owners, it truly covers the most important aspects of Agile development, through a simple and interactive workflow.

Finding a product that will accomodate all of your needs is impossible. You need to find what works for you and your team. Personally, if you’re getting started with Agile, I think Pivotal is a really good choice. Combined with GitHub for code versioning, and User Voice for users’ feedback, you can fully maximize your productivity with simple and interactive processes.

Comments on Hacker News.

Interface Is All That Matters

Tue, 08 May 2012 11:19:00 -0400

Whenever I begin writing a new Web Application I get all freaked out about choosing my development stack. What database? What language? How should I design my API? Will I need to make it performant? All this incredible amount of bullshit that doesn’t really matter at the end of the day, all that matters is the final product and the user experience, and I really want to concentrate on that instead of the technicalities.

Don’t get me wrong. I love talking technology. Trying to figure out what things will make my life easier in the long run is a really interesting topic, but it’s also very subjective. It’s hard to predict what’s going to happen to your app 2 or 3 months after you developed it. Users might have different needs and you might end up having to re-think your system. Plus technology changes so rapidly that you’ll eventually end up with something that is outdated.

So I decided to not worry about the future. I decided to develop apps using technologies that will make my life as simple as possible, today. Basically I try to avoid anything that gets in the way of me working on the user experience.

Here’s a great quote from Bret Victor about how important User Interface is:

Interface matters to me more than anything else, and it always has. I just never realized that. I’ve spent a lot of time over the years desperately trying to think of a “thing” to change the world. I now know why the search was fruitless — things don’t change the world. People change the world by using things. The focus must be on the “using”, not the “thing”. Now that I’m looking through the right end of the binoculars, I can see a lot more clearly, and there are projects and possibilities that genuinely interest me deeply.

What stack should you use?

This is surely the million dollar question. Essentially I think that you should use whatever you feel most comfortable with. For example, I like working with JSON - I find it to be a perfect data structure for representing data and I can easily interact with it using any language - so I’ll try and use a database that supports JSON (CouchDB or MongoDB). I also like very much JavaScript, and since we still can’t program computers using other senses such as our voice or vision, we must use a programming language, so I choose JS as it allows me to work with JSON in a very straightforward way.

There you go. Find reasons why you should use certain technologies compared to others. Even if it’s simply because you like Ruby better than PHP - it really doesn’t matter, just be smart.

Use the Cloud!

The Cloud has made it even easier for us to abstract the technicalities from our development. It’s amazing how you can essentially feed into the power of complicated and powerful databases (such as Google’s BigQuery) through a simple REST call. This is a cheap and smart solution for storing your data.

Also, why buy an entire Amazon server that you basically have to maintain, when you could easily use services independently. Need search? Amazon Search! Need a Database? Google BigQuery. Need a Web Server? IrisCouch. This allows you to rapidly build your interface and rely on these cloud services for specific functionalities, without being bothered with tedious technology decisions.

Gene Banks' Data

Sun, 25 Mar 2012 10:34:00 -0400

Gene banks, as described by Wikipedia, are a type of biorepository which preserve genetic material. In plants, this could be by freezing cuts from the plant, or stocking the seeds. Gene banks hold a very important value for our future, specifically with our planet undergoing environmental changes, allowing us to access and use specific seeds that would otherwise be lost in nature.

The value of these seeds is obviously important, but the ability to access these seeds is also critical. In fact, without a proper information access system these seeds would be lost in gene banks the same way they would’ve been lost in nature. Fortunately the top gene banks in the world do a good job in storing and managing their data, through internal software systems maintained by IT units.

However you can imagine that not all gene banks do a good job in managing their data. Some systems are as ugly as Excel spreadsheets or Word documents, located on regular desktop computers, copied around different machines, without any proper backup process or disaster recovery routine. Imagine losing this data: the location the seeds were collected in; the date; the characteristics of the plant. Without this data the seeds become useless, so it is very important that gene banks take responsibility in managing their data correctly.

Also another important issue is that this data might not be accessible through the internet. Either the gene banks have little internet connectivity, or they simply don’t have the software to allow them to publish their data online.

Accessing data across different gene banks

All this inconsistency between different gene banks information systems, is not beneficial for people that are trying to access this data. A system must be built that would gather all the data from all the different gene banks and make it accessible in a consistent way.

This is not an easy task. It’s like locating and gathering all the information on the internet about movies, and making it searchable in a consistent way. You could scrape data from different sites such as imdb and rottentomatoes, but you would still need to normalize part of the data. For example imdb uses stars for rating movies; rottentomatoes uses numbers. So you either convert stars into numbers, or you’re stuck with two pieces of information that are semantically the same, but syntactically different.

Possible solutions

If gene banks use the same seed management system, instead of something they built on their own, then the data is bound to be stored using a standard schema. Gathering and searching this data would then be simple and it would make it much more accessible.

This would be the ideal scenario, but unfortunately forcing everyone to use a specific system will not work. There’s always going to be someone more comfortable with building their own solutions. This is how business has always worked. A single system cannot accomodate everyone’s needs.

Another interesting approach would be to adopt a standard schema and have the gene banks publish their data online using microformats. This is something already being adopted for other information across the web, so it makes sense to be using it for the data held inside gene banks as well.

JavaScript On The Server

Thu, 04 Aug 2011 17:23:00 -0400

Over the past couple of years the language that has really made an impression over the others has been JavaScript. We can find clues of its burst of popularity everywhere; GitHub’s most famous and HackerNews’ most discussed language. It’s obvious JavaScript has made an impact on how we develop and interact with websites.

But why? Why are all these cool things coming out from this particularly annoying little scripting language? Why is everyone so excited about it? Frankly, I don’t think it’s part of the language’s syntax - as your Computer Scientist teacher in college might have told you, a programming language is just a tool that we use to perform a task, and the syntax is just a way to let us perform that task and it shouldn’t get in our way. Well, then it must be something else, right? Particularly for JavaScript, I would say its popularity is mostly due to the fact that it’s embedded in the browser, which is by far the most used application on a personal computer.

This little but important fact leads to an enormous amount of information related to the language being constantly shared over the internet. Think of blogs, code snippets, questions, tutorials etc. All this makes a language extremely popular. The amount of data on the internet related to a particular language is also proportionally related to the amount of programmers you’ll find that know how to use that language. This leads to more developers and therefore more innovation.

There you go, the secret JavaScript ingredient is the casual existence of the Web Browser on peoples computers that made interpreting scripts possible without the overhead of installing compilers or IDEs.

However, even with this extreme jump in popularity, JavaScript was always referred to as a client-side language or frontend language, it took a while before it entered the main-stream. Web-development has always been a matter of developing with JavaScript on the client-side and then using some other language on the server. This methodology is really inconsistent to me, but apparently people are still accustomed to it.

Of course I’m not saying that people should drop their Java/PHP/Python environment - technologies and languages should be used wherever they are best fit - I’m saying that people should consider trying out JavaScript on the server. If you think about it, you end up with a code-base written in one language; this points to all sorts of maintainability improvements on the overall project.

What are your thoughts on the subject? Will your next project use JavaScript on the server or will you use the same old PHP framework? Try and innovate, use different things and different languages, all the time. Don’t get stuck with a boring Java framework.