Google interview

Recently I was invited to Sydney to interview for a developer position.

I waited in the reception for a 1pm start. Apart from a tire swing, which everyone ignored anyway, this could have been any office. Nothing like the Googleplex in Mountain View.

I was led to a small room for the interviews. Along the way I passed rows of coders at work and an effigy of Sarah Palin. Poor taste.

I then had back to back technical interviews until past 5pm. Lunch was not provided as I had assumed, so I was feeling weak by the final interview.

By this time most people had already left for the day.

After the last interview I was escorted out. No tour. No snacks. No schwag.

So in conclusion I don't feel at all tempted to leave my freelancing + travel lifestyle.

Google App Engine limitations

Most of the discussion about Google App Engine seems to focus on how it allows you to scale your app, however I find it most useful for small client apps where we want a reliable platform while avoiding any ongoing hosting fee. For large apps paying for hosting would not be a problem.

These are some of the downsides I have found using Google App Engine:

• Slow - if your app has not been accessed recently (last minute) then it can take up to 10 seconds to load for the user
• Pure Python/Java code only - this prevents using a lot of good libraries, most importantly for me lxml
• CPU quota easily gets exhausted when uploading data
• Proxies not supported, which makes apps that rely on external websites risky. For example the Twitter API has a per IP quota which you would be sharing with all other GAE apps.
• Blocked in some countries, such as Turkey
• Indexes - the free quota is 1 GB but often over half of this is taken up by indexes
• Maximum 1000 records per query - no longer a limitation!
• 20 second request limit, so often need the overhead of using Task Queues

Despite these problems I still find Google App Engine a fantastic platform and a pleasure to develop on.

Using Google Translate to crawl a website

I wrote previously about using Google Cache to crawl a website. Sometimes, for whatever reason, Google Cache does not include a webpage so it is good to have backup options.

One option is using Google Translate, which let's you translate a webpage into another language. If the source language is selected as something you know it is not (eg Dutch) then no translation will take place and you will just get back the original content.

I added a function to download a URL via Google Translate and Google Cache to my webscraping library. Here is an example:

from webscraping import download, xpath

D = download.Download()
url = 'http://sitescraper.net/faq'
html1 = D.get(url) # download directly
html2 = D.gcache_get(url) # download via Google Cache
html3 = D.gtrans_get(url) # download via Google Translate
for html in (html1, html2, html3):
    print xpath.get(html, '//title')

This example downloads the same webpage directly, via Google Cache, and via Google Translate. Then it parses the title to show the same webpage has been downloaded. The output when run is:

Frequently asked questions | SiteScraper 

Frequently asked questions | SiteScraper 

Frequently asked questions | SiteScraper

Using Google Cache to crawl a website

Occasionally I come across a website that blocks your IP after only a few requests. If the website contains a lot of data then downloading it quickly would take an expensive amount of proxies.

Fortunately there is an alternative - Google.

If a website doesn't exist in Google's search results then for most people it doesn't exist at all. Websites want visitors so will usually be happy for Google to crawl their content. This means Google has likely already downloaded all the web pages we want. And after downloading Google makes much of the content available through their cache.

So instead of downloading a URL we want directly we can download it indirectly via Google Cache: http://www.google.com/search?&q=cache%3Ahttp%3A//sitescraper.net
This way the source website can not block you and does not even know you are crawling their content.

Google Storage

Often the datasets I scrape are too big to send via email and would take up too much space on my web server, so I upload them to Google Storage.
Here is an example snippet to create a folder on GS, upload a file, and then download it:

>>> gsutil mb gs://bucket_name
>>> gsutil ls
gs://bucket_name
>>> gsutil cp path/to/file.ext gs://bucket_name
>>> gsutil ls gs://bucket_name
file.ext
>>> gsutil cp gs://bucket_name/file.ext file_copy.ext

How to protect your data

You spent time and money collecting the data in your website so you want to prevent someone else downloading and reusing it. However you still want Google to index your website so that people can find you.

This is a common problem. Below I will outline some strategies to protect your data.

Restrict

Firstly if your data really is valuable then perhaps it shouldn't be all publicly available. Often websites will display the basic data to standard users / search engines and the more valuable data (such as emails) only to logged in users. Then the website can easily track and control how much valuable data each account is accessing.

If requiring accounts isn't practical and you want search engines to crawl it then realistically you can't prevent it being scraped, but you can discourage scrapers by setting a high enough barrier.

Obfuscate

Scrapers typically work by downloading the HTML for a URL and then extracting out the desired content. To make this process harder you can obfuscate your valuable data.

The simplest way to obfuscate your data is have it encoded on the server and then dynamically decoded with JavaScript in the client's browser. The scraper would then need to decode this JavaScript to extract the original data. This is not difficult for an experienced scraper, but would atleast provide a small barrier.

A better way is to encapsulate the key data within images or flash. Optical Character Recognition (OCR) techniques would then need to be used to extract the original data, which require a lot of effort to do accurately. (Make sure the URL of the image does not reveal the original data, as one website did!) The free OCR tools that I have tested are at best 80% accurate, which makes the resulting data useless.
The tradeoff with encoding data in images images is there will be more data for the client to download and they prevent genuine users from conveniently copying the text. For example people often display their email address within an image to combat spammers, which then forces everyone else to type it out manually.

Challenge

A popular way to prevent automated scrapers is by forcing users to pass a CAPTCHA. For example Google does this when it gets too many search requests from the same IP within a timeframe. To avoid the CAPTCHA the scraper could proceed slowly, but they probably can't afford to wait. To speed up this rate they may purchase multiple anonymous proxies to provide multiple IP addresses, but that is expensive - 10 anonymous proxies will cost ~$30 / month to rent. The CAPTCHA can also be solved automatically by a service like deathbycaptcha.com. This takes some effort to setup so would only be implemented by experienced scrapers for valuable data.

CAPTCHA is not a good solution for protecting your content - they annoy genuine users, can be bypassed by a determined scraper, and additionally make it difficult for the Google Bot to index your website properly. They are only a good solution when being indexed by Google is not a priority and you want to stop most scrapers.

Corrupt

If you are suspicious of an IP that is accessing your website you could block the IP, but then they would know they are detected and try a different approach. Instead you could allow the IP to continue downloading but return incorrect text or figures. This should be done subtly so that is not clear which data is correct and their entire data set will be corrupted. Perhaps they won't notice and you will be able to track them down later by searching for "purple monkey dishwasher" or whatever other content was inserted!

Structure

Another factor that makes sites easy to scrape is when they use a URL structure like:
domain/product/product_title/product_id
For example these two URLs point to the same content on Amazon:
http://www.amazon.com/Lets-Go-Australia-10th-Inc/dp/0312385757
http://www.amazon.com/FAKE_TITLE/dp/0312385757
The title is just to make the URL look pretty. This makes the site easy to crawl because the scraper can just iterate through all the ID's (in this case ISBN's). If the title here had to be consistent with the product ID then it would take more work to scrape.

Google

All of the above strategies could be ignored for the Google Bot to ensure your website is properly indexed. Be aware that anyone could pretend to be the Google Bot by setting their user-agent to "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", so to be confident you should also verify the IP address via a reversed DNS lookup. Be warned that Google has been known to punish websites that display different content for their bot to regular users.

 
In the next post I will take the opposite point of view of someone trying to scrape a website.