You may get better answers on CodeReview, but just glancing at your code I have one suggestion that is to split the codes into a scraping and parsing steps. This will help with reproducibility and also by decoupling the steps you can easier debug.
Scraping - the goal should be to save HTML files saved on the disk with names describing what they are and when they were collected (or a metadata lookup). Running a crawler on one page for 2 hours (with robots off) is asking for them to block your IP. You should consider doing the scraping in chunks, and then slowing each request down. If there are around 15 Registers, then run your code with a parameter to get one at a time. Getting all the data over the course of 24 hours is OK since this data is not so dynamic. But you don't want your code to run for 24 hours, you want it to run 24 times for less than one hour each time. The total number of HTML files shouldn't change very much, so that is a simple metric to see how complete your scraping is.
Parsing - once you can reliably download HTML files, the scrapings should be straightforward (from what I could tell, all the HTML files should be similar). Because the HTML files are downloaded, you can debug the parser by running it as much as you want, without having to re-run the scraper and run into network issues (i.e. server denying access).
Even if you have separate codes for each step that are functionalized with input variables (especially for the scraper), you can easily join them with one launcher. In this case there would be 3 python files (launcher.py, my_scraper.py, and my launcher.py)
import glob, time # standard libs
import my_scraper # my custom scraper
for item in list_of_registries: # get one at a time
my_scraper.main(item) # accepts a variable based on the website
time.sleep(300)
# break # for debugging, put in a break statement
import my_parser
table = []
for html_file in glob.glob('*.html):
table += my_parser.main(html_file) # accepts a blob of html, returns the structured data
One question - you have in your code from requests import session
but I don't see where that session is established. By creating a session you might find the server is less likely to deny your many rapid requests. In leaving a tab open for 15 minutes with that page, my browser showed me this:
