I have a case where I'm building a connector between a federated search engine, and an archive that's serving all of their data via HTTP. They've been kind enough to have their pipeline generate a series of index files for each day that I can use to get the metadata that I need to allow people to search through their data.

Often for situations like this, I'd use wget to get directory listings ... but I'm running into a few problems:

  1. Some of the directories are huge ... it's taking me minutes for the 'day' directories (and this spans a decade).
  2. They have the autoindex options set so that they also have links to sort the directory in different ways.
  3. I need to check for signs of re-processing, so I'll need to re-crawl, it's not just a one-off crawl.

I was hoping to use wget to skip over the sorting links, as more recent versions allow rejection patterns, instead of just rejecting file extensions:

-R '*index.html[?]C=?;O=?'

Unfortunately, it seems that the 'exclude' option in wget doesn't do quite what I'd have expected:

2015-03-25 14:35:33 (6.30 MB/s) - `index.html?C=N;O=D' saved [2635]

Removing index.html?C=N;O=D since it should be rejected.

... so it downloads the files, and then deletes them ... rather than avoiding them in the first place.

Is anyone aware of a tool that can be used for scraping that processes a reject list before attempting to download things?

  • Does this also happen with the --reject-regex option?
    – Kersten
    Mar 26, 2015 at 14:25
  • @Kersten : hmm ... it seems that I'm running an older wget that doesn't support that option. Compiling the most recent wget now.
    – Joe
    Mar 26, 2015 at 14:45
  • @Kersten : I tried --regex-type pcre --reject-regex '.*index[.]html[?]C=\w;O=\w' but it's not even deleting them. Will try some more after my meeting in 2 min.
    – Joe
    Mar 26, 2015 at 14:58
  • I should add that I had the same problem with the index.html* files. I circumvented that by using the --spider option, then filtered the resulting url list and used wget with the filtered list to download the files. From your question I am not quite sure if that is even an option.
    – Kersten
    Mar 26, 2015 at 15:39
  • @Kersten : I tried it, and got the message of Spider mode enabled. Check if remote file exists., but it still does the 'download, but then delete' behavior. I really don't want to contacting the remote webserver at all. I guess I could set up an HTTP proxy, and have that do the filtering. Your mention of --spider also led me to : stackoverflow.com/q/2804467/143791 ... which sounds like what I need, but I suspect I might need to do --convert-links and write a wrapper to only pull one URL at a time.
    – Joe
    Mar 26, 2015 at 16:22

2 Answers 2


Using --reject-regex "\?.=.;.=." seems to work for me to avoid the extra listings provided by Apache. The full command-line I use to get all the files from an Apache directory listing is:

wget --recursive --no-parent --timestamping --no-directories --reject-regex "\?.=.;.=." http://example.com/some/dir/

This doesn't solve the problem entirely, but I'll add it here for anyone else who has the same issue. (but not mark it as the answer, in case someone else has a better solution)

There's a program xidel which allows you to retrieve a URL and extract information from the page using XPath queries. See this response to a similar question on Stack Overflow.

The problem is that it doesn't do two things:

  • recursive retrieval -- you'll have to write your own script process the output and do whatever logic you need to determine what links to follow.
  • return complete URLs -- the example shows how to extract what's in //a/@href, but this could be an absolute, relative or root-relative instance. It may also need to take //base/@href into consideration if one exists.

Even with those problems, I found that it saved me considerable time over trying to get wget to do what I needed. I called xidel in a perl script, and then used curl -s -f -z (silent, no file on failure, send if-modifed since) to do the file retrieval once I identified the proper URLs to check for updates. I probably could've used LWP::Simple's mirror() to avoid calling out to curl.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.