Skip to main content
Share Your Experience: Take the 2024 Developer Survey
6 votes
Accepted

how to search archive.org for PDF files on a captured website between some date range

here's how i automated the process within R setwd( "C:/path/to/PDFs" ) year <- 2008 domain_name <- "anthem.com" archive_org_captures <- readLines( paste0( "http://...
Anthony Damico's user avatar
4 votes

how to search archive.org for PDF files on a captured website between some date range

Here is one way, although the API probably has more efficient methods. You can append * to the end of a URL in the Wayback Machine and it will return all of the saved URLs for that domain. The link ...
albert's user avatar
  • 11.9k
3 votes

The list of restricted second-level domains such as .co.uk and .ac.jp

This "restricted domain" is known an effective top-level domain (eTLD)", or a suffix. By Mozilla: "A "public suffix" is one under which Internet users can (or historically could) directly register ...
Anton Tarasenko's user avatar
3 votes

Places to get large (volume wise) datasets reasonably well formatted and free to share?

USPTO has a Bulk Data sub domain which has many datasets that fit this description for you to pick through. Here's a description of two: Patent Grant Single-Page TIFF Images (July 31, 1790 - ...
albert's user avatar
  • 11.9k
3 votes
Accepted

What tool can I use to manage a researcher network bibliography?

ORCID provides a persistent digital identifier that distinguishes you from every other researcher and, through integration in key research workflows such as manuscript and grant submission, supports ...
albert's user avatar
  • 11.9k
3 votes

Datasets: Is there any public data-set for FAQ (Frequently asked questions) in different domains?

What about StackOverflow's own dataset? https://archive.org/details/stackexchange
robinvdvleuten's user avatar
2 votes

Data about what kind of entities appear on public web pages

Not concrete answers, but here are some thoughts/places to find this: Common Crawl is a large corpus of crawled data, Web Data Commons are deep dives into the data. Read up on which are the most used ...
albert's user avatar
  • 11.9k
2 votes
Accepted

Lobby meetings of European Commission - Integrity Watch

With a hint from a person from Transparency International and some further browsing through the source code this link eventually made it work: http://static.tttp.eu/doi/data/meeting.csv
moezelot's user avatar
  • 153
2 votes

List of political blogs

There is a blog that lists political blogs based in the US: http://politicalbloglistings.blogspot.ch/ Categories (count of listed blogs): Conservative (108) Liberal (70) Non/Bi-Partisan or ...
philshem's user avatar
  • 17.6k
2 votes

The "right to mine" and scraping the web in EU

The article in the LSE blog you have pointed out seems to be a good overview, the problems seem to be not only UK specific: Even with some legal barriers now removed, technical barriers remain ...
2 votes

how to search archive.org for PDF files on a captured website between some date range

Using the official wayback_machine_downloader tool: wayback_machine_downloader https://www.anthem.com --from 2005 --to 2008 --only "/\.(pdf)$/i"
Matt Sephton's user avatar
2 votes

How do I get historical data from coinmarketcap.com?

Here's a quick python3 script to get the table from each html page as a local csv file. basically, you have to create a loop over all dates and then make a web requests. the python library pandas has ...
philshem's user avatar
  • 17.6k
1 vote
Accepted

Catalog of all websites on the entire internet

Regarding git repo: it's not stale, it was updated about a month ago. Still, there aren't newer/larger and available datasets just yet. If you stumble upon one - just drop a line here, I'll merge it ...
Bohdan Turkynewych's user avatar
1 vote

Online Dictionary for Scraping

i found a website with 10,000 words that I'm using for my project. mabey this will work? https://www.mit.edu/~ecprice/wordlist.10000
hayden misiura's user avatar
1 vote

Online Dictionary for Scraping

You can use data from Wiktionary. Wikimedia provides data dumps for all its websites here (ctrl+f "wiktionary"), that way you don't have to scrape. Although the dumps seem to be regular, if you want ...
Peeyush Kushwaha's user avatar
1 vote

Places to get large (volume wise) datasets reasonably well formatted and free to share?

See the following for starters: Engima.io Public Data data.world Awesome Public Datasets
Carl V. Lewis's user avatar
1 vote

How to obtain airline data

I don't know the scope of airports you are after (US only, EU or WW) but I can't think of any public sources. I think that you will need to find a comparison site like kayak or expedia then use a ...
Marcus D's user avatar
  • 1,109
1 vote

How to obtain airline data

Daily airfares from where to where on which airline? Every flight everywhere or a specific flight on a specific airline...The data is public just go to a carriers site and do a query. If you a trying ...
risail's user avatar
  • 260
1 vote

Twitter crawlers for tweets, retweets and social network

With www.followthehashtag.com twitter analytics service you will be able to get Twitter data for free up to 9 days back or track some thousand tweets or, in premium version since 2006, with no total ...
Enrique R Rivera's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible