Many questions are posted here in search of a specific dataset that is either no longer online, or has died from linkrot, or a combination of the two. What is the best way to get around this? Or rather, can these datasets be recovered?

up vote 9 down vote accepted

The Memento Web and the Wayback Machine are two possible solutions:

Wayback Machine

The Wayback Machine by the Internet Archive is your best friend for all things that were once online, and even some things that still are, if you want to compare changes.

From Wikipedia:

The Wayback Machine is a digital archive of the World Wide Web and other information on the Internet created by the Internet Archive, a non-profit organization, based in San Francisco, California. It was set up by Brewster Kahle and Bruce Gilliat, and is maintained with content from Alexa Internet. The service enables users to see archived versions of web pages across time, which the Archive calls a "three dimensional index."

If you have a dead link, simply paste it into the search input element of Wayback Machine.

Alternatively, you can type https://web.archive.org/web/*/DEAD_LINK'S_URL directly into the address bar of your favorite browser.

Memento Web

Los Alamos National Lab has been offering the Memento Web project which unifies search across the archives.

Other

Some search engines may provide you with a cached version of the page where the dataset resided (possibly, the dataset files as well).

If the steps above fail, you may consider

  1. looking for papers which used the dataset (at arXiv or in academic journals),
  2. contacting their authors,
  3. looking at various bulletin boards for those who might have downloaded the dataset and contacting them.

You can also have a look at http://academictorrents.com to see if your dataset is present there.

  • In addition to this, a number of countries have implemented "digital legal deposit" and introduced national web archives. In the UK, for example, there is the UK Web Archive, which allows free access to archived web pages over the internet where licensing allows, and full access to the restricted parts of the archive from the British Library reading room in London and a handful of other access points around the UK. – Jez Mar 19 '15 at 13:35

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.