Many questions are posted here in search of a specific dataset that is either no longer online, or has died from linkrot, or a combination of the two. What is the best way to get around this? Or rather, can these datasets be recovered?
I just posted a related question on the underlying causes: opendata.stackexchange.com/questions/5319/…– Daniel MietchenMay 25, 2015 at 14:38
The Memento Web and the Wayback Machine are two possible solutions:
The Wayback Machine by the Internet Archive is your best friend for all things that were once online, and even some things that still are, if you want to compare changes.
The Wayback Machine is a digital archive of the World Wide Web and other information on the Internet created by the Internet Archive, a non-profit organization, based in San Francisco, California. It was set up by Brewster Kahle and Bruce Gilliat, and is maintained with content from Alexa Internet. The service enables users to see archived versions of web pages across time, which the Archive calls a "three dimensional index."
If you have a dead link, simply paste it into the search input element of Wayback Machine.
Alternatively, you can type https://web.archive.org/web/*/DEAD_LINK'S_URL directly into the address bar of your favorite browser.
Los Alamos National Lab has been offering the Memento Web project which unifies search across the archives.
Some search engines may provide you with a cached version of the page where the dataset resided (possibly, the dataset files as well).
If the steps above fail, you may consider
- looking for papers which used the dataset (at arXiv or in academic journals),
- contacting their authors,
- looking at various bulletin boards for those who might have downloaded the dataset and contacting them.
You can also have a look at http://academictorrents.com to see if your dataset is present there.
In addition to this, a number of countries have implemented "digital legal deposit" and introduced national web archives. In the UK, for example, there is the UK Web Archive, which allows free access to archived web pages over the internet where licensing allows, and full access to the restricted parts of the archive from the British Library reading room in London and a handful of other access points around the UK.– JezMar 19, 2015 at 13:35