Google App Engine limitations

Most of the discussion about Google App Engine seems to focus on how it allows you to scale your app, however I find it most useful for small client apps where we want a reliable platform while avoiding any ongoing hosting fee. For large apps paying for hosting would not be a problem.

These are some of the downsides I have found using Google App Engine:

• Slow - if your app has not been accessed recently (last minute) then it can take up to 10 seconds to load for the user
• Pure Python/Java code only - this prevents using a lot of good libraries, most importantly for me lxml
• CPU quota easily gets exhausted when uploading data
• Proxies not supported, which makes apps that rely on external websites risky. For example the Twitter API has a per IP quota which you would be sharing with all other GAE apps.
• Blocked in some countries, such as Turkey
• Indexes - the free quota is 1 GB but often over half of this is taken up by indexes
• Maximum 1000 records per query - no longer a limitation!
• 20 second request limit, so often need the overhead of using Task Queues

Despite these problems I still find Google App Engine a fantastic platform and a pleasure to develop on.

Increase your Google App Engine quotas for free

Google App Engine provides generous free quotas for your app and additional paid quotas.
I always enable billing for my GAE apps even though I rarely exhaust the free quotas. Enabling billing and setting paid quotas does not mean you have to pay anything and in fact increases what you get for free.

Here is a screenshot of the billing panel:

GAE lets you allocate a daily budget to the various resources, with the minimum permitted budget being USD $1. When you exhaust a free quota you will only be charged for the budget allocated to it. In the above screenshot I have allocated all my budget to emailing, but since my app does use the Mail API I can be confident this free quota will never be exhausted and I will never pay a cent. For another app that does use Mail I have allocated all the budget to Bandwidth Out instead.

Now with billing enabled my app:

• can access the Blobstore API to store larger amounts of data
• enjoys much higher free limits for the Mail, Task Queue, and UrlFetch API's - for example by default an app can make 7000 Mail API calls but with billing enabled this limit jumps to 1,700,000 calls
• has a higher per minute CPU limit, which I find particularly useful when uploading a mass of records to the Datastore

So in summary you can enable billing to extend your free quotas without risk of paying.

Why Google App Engine

In the previous post I covered three alternative approaches to regularly scrape a website for a client, with the most common one being in the form of a web application. However hosting the web application on either my own or the clients server has problems.

My solution is to host the application on a neutral third party platform - Google App Engine (GAE). Here is my overview of deploying on GAE:

Pros:

• provides a stable and consistent platform that I can use for multiple applications
• both the customer and I can login and manage it, so we do not need to expose our servers
• has generous free quotas, which I rarely exhaust

Cons:

• only supports pure Python (or Java), so libraries that rely on C such as lxml are not supported (yet)
• limitations on maximum job time and interacting with the database
• have to trust Google with storing our scraped data

Often deploying on GAE works well for both the client and me, but it is not always practical/possible. I'm still looking for a silver bullet!