I have a case where I'm building a connector between a federated search engine, and an archive that's serving all of their data via HTTP. They've been kind enough to have their pipeline generate a series of index files for each day that I can use to get the metadata that I need to allow people to search through their data.
Often for situations like this, I'd use wget
to get directory listings ... but I'm running into a few problems:
- Some of the directories are huge ... it's taking me minutes for the 'day' directories (and this spans a decade).
- They have the autoindex options set so that they also have links to sort the directory in different ways.
- I need to check for signs of re-processing, so I'll need to re-crawl, it's not just a one-off crawl.
I was hoping to use wget
to skip over the sorting links, as more recent versions allow rejection patterns, instead of just rejecting file extensions:
-R '*index.html[?]C=?;O=?'
Unfortunately, it seems that the 'exclude' option in wget doesn't do quite what I'd have expected:
2015-03-25 14:35:33 (6.30 MB/s) - `index.html?C=N;O=D' saved [2635] Removing index.html?C=N;O=D since it should be rejected.
... so it downloads the files, and then deletes them ... rather than avoiding them in the first place.
Is anyone aware of a tool that can be used for scraping that processes a reject list before attempting to download things?
--reject-regex
option?--regex-type pcre --reject-regex '.*index[.]html[?]C=\w;O=\w'
but it's not even deleting them. Will try some more after my meeting in 2 min.index.html*
files. I circumvented that by using the--spider
option, then filtered the resulting url list and used wget with the filtered list to download the files. From your question I am not quite sure if that is even an option.Spider mode enabled. Check if remote file exists.
, but it still does the 'download, but then delete' behavior. I really don't want to contacting the remote webserver at all. I guess I could set up an HTTP proxy, and have that do the filtering. Your mention of--spider
also led me to : stackoverflow.com/q/2804467/143791 ... which sounds like what I need, but I suspect I might need to do--convert-links
and write a wrapper to only pull one URL at a time.