Why is my scraper returning inconsistent results and timing out?

Question

I've been building a scraper in python (using mechanize) to get a list of banks off of:

https://connectonline.asic.gov.au/RegistrySearch/faces/landing/ProfessionalRegisters.jspx

The scraper is: https://github.com/bianca/mapaustralianbanks

1) I keep getting inconsistent results. Sometimes I query and get data, sometimes I don't. What is the best way of making the python script more robust so that it keeps trying in order to eventually get consistent results? I was thinking of some sort of settimeout before requerying, but I was wondering if there was something about the server that I'm making requests to that is the reason why my data is inconsistent...

2) ~2 hours in, the script just stops scraping. I've tried to set a timeout in mechanize to give up after 60 seconds, but that isn't working. Any ideas?

Answer 1

Some tips for scrapers:

• log: don't stop at "sometimes I get data and sometimes I don't" -- when you get unexpected results in the response, log it so you can learn what happened. Specifically when you are being throttled for making too many requests, you will often get an explicit message saying so.
• cache: write your code to save files you retrieve and use saved versions before making a URL call.

Python users can take advantage of some toolkits which make these two things easier. I've had some good experience with scrapelib and scrapekit. I do not know how to insert seamless caching into Mechanize—last time I wrote a scraper with Mechanize I just did it manually, which is basically what @philshem suggested.

Answer 2

You may get better answers on CodeReview, but just glancing at your code I have one suggestion that is to split the codes into a scraping and parsing steps. This will help with reproducibility and also by decoupling the steps you can easier debug.

• Scraping - the goal should be to save HTML files saved on the disk with names describing what they are and when they were collected (or a metadata lookup). Running a crawler on one page for 2 hours (with robots off) is asking for them to block your IP. You should consider doing the scraping in chunks, and then slowing each request down. If there are around 15 Registers, then run your code with a parameter to get one at a time. Getting all the data over the course of 24 hours is OK since this data is not so dynamic. But you don't want your code to run for 24 hours, you want it to run 24 times for less than one hour each time. The total number of HTML files shouldn't change very much, so that is a simple metric to see how complete your scraping is.

• Parsing - once you can reliably download HTML files, the scrapings should be straightforward (from what I could tell, all the HTML files should be similar). Because the HTML files are downloaded, you can debug the parser by running it as much as you want, without having to re-run the scraper and run into network issues (i.e. server denying access).

---

Even if you have separate codes for each step that are functionalized with input variables (especially for the scraper), you can easily join them with one launcher. In this case there would be 3 python files (launcher.py, my_scraper.py, and my launcher.py)

import glob, time # standard libs

import my_scraper # my custom scraper
for item in list_of_registries: # get one at a time
    my_scraper.main(item) # accepts a variable based on the website
    time.sleep(300)  
    # break # for debugging, put in a break statement

import my_parser
table = []
for html_file in glob.glob('*.html):
    table += my_parser.main(html_file)  # accepts a blob of html, returns the structured data

---

One question - you have in your code from requests import session but I don't see where that session is established. By creating a session you might find the server is less likely to deny your many rapid requests. In leaving a tab open for 15 minutes with that page, my browser showed me this: