Taking advantage of mobile interfaces

Sometimes a website will have multiple versions: one for regular users with a modern browser, a HTML version for browsers that don't support JavaScript, and a simplified version for mobile users.

For example Gmail has:

• gmail.com
• mail.google.com/maiOl/h/
• m.gmail.com

All three of these interfaces will display the content of your emails but use different layouts and features. The main entrance at gmail.com is well known for its use of AJAX to load content dynamically without refreshing the page. This leads to a better user experience but makes web automation or scraping harder.

On the other hand the static HTML interface has fewer features and is less efficient for users, but much easier to automate or scrape because all the content is available when the page loads.

So before scraping a website check for its HTML or mobile version, which should be easier to scrape.

To find the HTML version try disabling JavaScript in your browser and see what happens.
To find the mobile version try adding the "m" subdomain (domain.com -> m.domain.com) or using a mobile user-agent.

Parsing Flash with Swiffy

Google has released a tool called Swiffy for parsing Flash files into HTML5. This is relevant to web scraping because content embedded in Flash is a pain to extract, as I wrote about earlier.

I tried some test files and found the results no more useful for parsing text content than the output produced by swf2html (Linux version). Some neat example conversions are available here. Currently Swiffy supports ActionScript 2.0 and works best with Flash 5, which was released back in 2000 so there is still a lot of work to do.

Google App Engine limitations

Most of the discussion about Google App Engine seems to focus on how it allows you to scale your app, however I find it most useful for small client apps where we want a reliable platform while avoiding any ongoing hosting fee. For large apps paying for hosting would not be a problem.

These are some of the downsides I have found using Google App Engine:

• Slow - if your app has not been accessed recently (last minute) then it can take up to 10 seconds to load for the user
• Pure Python/Java code only - this prevents using a lot of good libraries, most importantly for me lxml
• CPU quota easily gets exhausted when uploading data
• Proxies not supported, which makes apps that rely on external websites risky. For example the Twitter API has a per IP quota which you would be sharing with all other GAE apps.
• Blocked in some countries, such as Turkey
• Indexes - the free quota is 1 GB but often over half of this is taken up by indexes
• Maximum 1000 records per query - no longer a limitation!
• 20 second request limit, so often need the overhead of using Task Queues

Despite these problems I still find Google App Engine a fantastic platform and a pleasure to develop on.

Using Google Translate to crawl a website

I wrote previously about using Google Cache to crawl a website. Sometimes, for whatever reason, Google Cache does not include a webpage so it is good to have backup options.

One option is using Google Translate, which let's you translate a webpage into another language. If the source language is selected as something you know it is not (eg Dutch) then no translation will take place and you will just get back the original content.

I added a function to download a URL via Google Translate and Google Cache to my webscraping library. Here is an example:

from webscraping import download, xpath

D = download.Download()
url = 'http://sitescraper.net/faq'
html1 = D.get(url) # download directly
html2 = D.gcache_get(url) # download via Google Cache
html3 = D.gtrans_get(url) # download via Google Translate
for html in (html1, html2, html3):
    print xpath.get(html, '//title')

This example downloads the same webpage directly, via Google Cache, and via Google Translate. Then it parses the title to show the same webpage has been downloaded. The output when run is:

Frequently asked questions | SiteScraper 

Frequently asked questions | SiteScraper 

Frequently asked questions | SiteScraper

Using Google Cache to crawl a website

Occasionally I come across a website that blocks your IP after only a few requests. If the website contains a lot of data then downloading it quickly would take an expensive amount of proxies.

Fortunately there is an alternative - Google.

If a website doesn't exist in Google's search results then for most people it doesn't exist at all. Websites want visitors so will usually be happy for Google to crawl their content. This means Google has likely already downloaded all the web pages we want. And after downloading Google makes much of the content available through their cache.

So instead of downloading a URL we want directly we can download it indirectly via Google Cache: http://www.google.com/search?&q=cache%3Ahttp%3A//sitescraper.net
This way the source website can not block you and does not even know you are crawling their content.

Google Storage

Often the datasets I scrape are too big to send via email and would take up too much space on my web server, so I upload them to Google Storage.
Here is an example snippet to create a folder on GS, upload a file, and then download it:

>>> gsutil mb gs://bucket_name
>>> gsutil ls
gs://bucket_name
>>> gsutil cp path/to/file.ext gs://bucket_name
>>> gsutil ls gs://bucket_name
file.ext
>>> gsutil cp gs://bucket_name/file.ext file_copy.ext

The SiteScraper module

A few years ago I developed the sitescraper library for automatically scraping website data based on example cases:

>>> from sitescraper import sitescraper>>> ss = sitescraper()
>>> url = 'http://www.amazon.com/s/ref=nb_ss_gw?url=search-alias%3Daps&field-keywords=python&x=0&y=0'
>>> data = ["Amazon.com: python", ["Learning Python, 3rd Edition", 
     "Programming in Python 3: A Complete Introduction to the Python Language (Developer's Library)", 
     "Python in a Nutshell, Second Edition (In a Nutshell (O'Reilly))"]]
>>> ss.add(url, data)
>>> # we can add multiple example cases, but this is a simple example so 1 will do (I generally use 3)
>>> # ss.add(url2, data2) 
>>> ss.scrape('http://www.amazon.com/s/ref=nb_ss_gw?url=search-alias%3Daps&field-keywords=linux&x=0&y=0')
["Amazon.com: linux", ["A Practical Guide to Linux(R) Commands, Editors, and Shell Programming", "Linux Pocket Guide", "Linux in a Nutshell (In a Nutshell (O'Reilly))", 'Practical Guide to Ubuntu Linux (Versions 8.10 and 8.04), A (2nd Edition)', 'Linux Bible, 2008 Edition: Boot up to Ubuntu, Fedora, KNOPPIX, Debian, openSUSE, and 11 Other Distributions']]

See this paper for more info.

It was designed for scraping websites overtime where their layout may change. Unfortunately I don't use it much these days because most of my projects are one-off scrapes.