Exclusion lists when crawling web directories

Question

I have a case where I'm building a connector between a federated search engine, and an archive that's serving all of their data via HTTP. They've been kind enough to have their pipeline generate a series of index files for each day that I can use to get the metadata that I need to allow people to search through their data.

Often for situations like this, I'd use wget to get directory listings ... but I'm running into a few problems:

1. Some of the directories are huge ... it's taking me minutes for the 'day' directories (and this spans a decade).
2. They have the autoindex options set so that they also have links to sort the directory in different ways.
3. I need to check for signs of re-processing, so I'll need to re-crawl, it's not just a one-off crawl.

I was hoping to use wget to skip over the sorting links, as more recent versions allow rejection patterns, instead of just rejecting file extensions:

-R '*index.html[?]C=?;O=?'

Unfortunately, it seems that the 'exclude' option in wget doesn't do quite what I'd have expected:

2015-03-25 14:35:33 (6.30 MB/s) - `index.html?C=N;O=D' saved [2635]

Removing index.html?C=N;O=D since it should be rejected.

... so it downloads the files, and then deletes them ... rather than avoiding them in the first place.

Is anyone aware of a tool that can be used for scraping that processes a reject list before attempting to download things?

Answer 1

Using --reject-regex "\?.=.;.=." seems to work for me to avoid the extra listings provided by Apache. The full command-line I use to get all the files from an Apache directory listing is:

wget --recursive --no-parent --timestamping --no-directories --reject-regex "\?.=.;.=." http://example.com/some/dir/

Answer 2

This doesn't solve the problem entirely, but I'll add it here for anyone else who has the same issue. (but not mark it as the answer, in case someone else has a better solution)

There's a program xidel which allows you to retrieve a URL and extract information from the page using XPath queries. See this response to a similar question on Stack Overflow.

The problem is that it doesn't do two things:

• recursive retrieval -- you'll have to write your own script process the output and do whatever logic you need to determine what links to follow.
• return complete URLs -- the example shows how to extract what's in //a/@href, but this could be an absolute, relative or root-relative instance. It may also need to take //base/@href into consideration if one exists.

Even with those problems, I found that it saved me considerable time over trying to get wget to do what I needed. I called xidel in a perl script, and then used curl -s -f -z (silent, no file on failure, send if-modifed since) to do the file retrieval once I identified the proper URLs to check for updates. I probably could've used LWP::Simple's mirror() to avoid calling out to curl.