I have a case where I'm building a connector between a federated search engine, and an archive that's serving all of their data via HTTP. They've been kind enough to have their pipeline generate a series of index files for each day that I can use to get the metadata that I need to allow people to search through their data.
Often for situations like this, I'd use
wget to get directory listings ... but I'm running into a few problems:
- Some of the directories are huge ... it's taking me minutes for the 'day' directories (and this spans a decade).
- They have the autoindex options set so that they also have links to sort the directory in different ways.
- I need to check for signs of re-processing, so I'll need to re-crawl, it's not just a one-off crawl.
I was hoping to use
wget to skip over the sorting links, as more recent versions allow rejection patterns, instead of just rejecting file extensions:
Unfortunately, it seems that the 'exclude' option in wget doesn't do quite what I'd have expected:
2015-03-25 14:35:33 (6.30 MB/s) - `index.html?C=N;O=D' saved  Removing index.html?C=N;O=D since it should be rejected.
... so it downloads the files, and then deletes them ... rather than avoiding them in the first place.
Is anyone aware of a tool that can be used for scraping that processes a reject list before attempting to download things?