how to search archive.org for PDF files on a captured website between some date range

Question

I am trying to search health insurance companies' website content on http://archive.org/ during the period of 2005-2008 for legal documents related to pre-existing medical conditions.

most of these websites contain forms that users could have filled out for a cost estimate or rate quote, but http:// form submissions rarely work on archive.org historic pages, so my search dead-ends there.

I suspect that most of the actual text content I am looking for is hosted in legal agreements in PDF files, so the optimal search would simply be to grab all captured files ending with .pdf but I am unclear how to do that for a specific website for a specific date range? example websites would be https://www.anthem.com/ or http://www.kaiserpermanente.org/

Answer 1

here's how i automated the process within R

setwd( "C:/path/to/PDFs" )

year <- 2008

domain_name <- "anthem.com"

archive_org_captures <-
    readLines( 
        paste0(
            "http://web.archive.org/cdx/search/cdx?url=" , 
            domain_name , 
            "/about/&matchType=host&limit=10000000&from=" ,
            year , 
            "&to=" ,
            year ,
            "&timestamp=4" 
        )
    )


all_responses <- grep( "pdf" , archive_org_captures , value = TRUE )

all_pdfs <- unique( gsub( "(.*) (.*) (.*) (.*) (.*) (.*)" , "\\2" , grep( "pdf" , all_responses , value = TRUE ) ) )

for( this_pdf in all_pdfs ){

    this_url <- paste0( "https://web.archive.org/web/" , year , "/" , this_pdf )

    try( download.file( this_url , basename( this_pdf ) , mode = 'wb' ) , silent = TRUE )

}

Answer 2

Here is one way, although the API probably has more efficient methods.

You can append * to the end of a URL in the Wayback Machine and it will return all of the saved URLs for that domain. The link below does this, although you can't see the asterisk because markdown is dumb. This is actually a great case for why its dumb. After wasting a few minutes trying to get the asterisk in this paragraph to render to no avail, now its not showing in the URL/example. Anyways, the link below does the complete domain search for Anthem on the Wayback Machine.
http://web.archive.org/web//https://www.anthem.com/

Sometimes there are so many results, your browser will choke...just let it breath until its functional again. Firefox and Chrome often fire alerts warning of the unresponsive tab; in Chrome, don't select kill, select wait; in Firefox the alerts are typically regarding unresponsive JavaScript files, again don't kill, wait.

You'll know its functional again when you see options under the header declaring the number of URLs that have been captured for this domain. Directly underneath this, there are two options, show number of entries, and filter results. In the filter input, type "pdf" and the results will only display URLs that match your filter.

A word of warning here, ".pdf" filter search doesn't mean you are getting all the PDFs that have been saved. Some sites save URLs that force downloads, some have other tricks; the point is, it is not 100% accurate, so be aware of that. Also, some URLs will have pdf in them, like icons, document roots, etc.; again, don't be lulled into false security. Its best to poke around manually until you feel confident with the results.

After filtering the Anthem example, the Wayback Machine says its has 7368 entries (filtered from 135865 total entries). How you go about downloading/accessing them can take many routes, although this makes another great case for using the API.

One way to accomplish this non-technically that I use, but only for small results (1 page or less of results) is to utilize the Download Them All extension for Firefox; run Download Them All, select the appropriate filter, and click download them all.

Another word of warning here, Download Them All is also not foolproof, and will not automatically get every PDF everytime; the same filtering rules apply here as they do in the Wayback Machine. Download Them All cannot distinguish between URLs that don't have proper file extensions without some tinkering. They do allow some RegEx as well, but that is another discussion in itself.

To break this all down into a time frame, you can search by year in the Wayback Machine by prepending a date, as well as a variable, to the domain you would like searched.

http://web.archive.org/web/2002*/https://www.anthem.com/

is a search across anthem.com for 2002.

Sidenote: pretty odd that asterisk worked...

Lastly, I have to add that even though the Wayback Machine has X number of hits, those don't equate to X number of desired results. Some are references that redirect, some 404, and some I still don't have a strong grasp on what is happening, except that they simply do not exist on the Wayback Machine's servers. The question touches on this when referencing documents locked behind forms/technology that requires the original sources infrastructure to complete functionality.

Answer 3

Using the official wayback_machine_downloader tool:

wayback_machine_downloader https://www.anthem.com --from 2005 --to 2008 --only "/\.(pdf)$/i"