10 votes
Accepted

Scraping pages to get prices can be considered as OpenData

There's a big difference between “available to the public” and “belonging to the public”. By accessing a privately-owned website, you are accepting its terms and conditions. These conditions typically ...
scruss's user avatar
  • 555
6 votes
Accepted

how to search archive.org for PDF files on a captured website between some date range

here's how i automated the process within R setwd( "C:/path/to/PDFs" ) year <- 2008 domain_name <- "anthem.com" archive_org_captures <- readLines( paste0( "http://...
Anthony Damico's user avatar
4 votes

how to search archive.org for PDF files on a captured website between some date range

Here is one way, although the API probably has more efficient methods. You can append * to the end of a URL in the Wayback Machine and it will return all of the saved URLs for that domain. The link ...
albert's user avatar
  • 11.9k
3 votes

The list of restricted second-level domains such as .co.uk and .ac.jp

This "restricted domain" is known an effective top-level domain (eTLD)", or a suffix. By Mozilla: "A "public suffix" is one under which Internet users can (or historically could) directly register ...
Anton Tarasenko's user avatar
3 votes

Places to get large (volume wise) datasets reasonably well formatted and free to share?

USPTO has a Bulk Data sub domain which has many datasets that fit this description for you to pick through. Here's a description of two: Patent Grant Single-Page TIFF Images (July 31, 1790 - ...
albert's user avatar
  • 11.9k
3 votes
Accepted

What tool can I use to manage a researcher network bibliography?

ORCID provides a persistent digital identifier that distinguishes you from every other researcher and, through integration in key research workflows such as manuscript and grant submission, supports ...
albert's user avatar
  • 11.9k
3 votes

Datasets: Is there any public data-set for FAQ (Frequently asked questions) in different domains?

What about StackOverflow's own dataset? https://archive.org/details/stackexchange
robinvdvleuten's user avatar
2 votes

List of political blogs

There is a blog that lists political blogs based in the US: http://politicalbloglistings.blogspot.ch/ Categories (count of listed blogs): Conservative (108) Liberal (70) Non/Bi-Partisan or ...
philshem's user avatar
  • 17.6k
2 votes

The "right to mine" and scraping the web in EU

The article in the LSE blog you have pointed out seems to be a good overview, the problems seem to be not only UK specific: Even with some legal barriers now removed, technical barriers remain ...
2 votes

how to search archive.org for PDF files on a captured website between some date range

Using the official wayback_machine_downloader tool: wayback_machine_downloader https://www.anthem.com --from 2005 --to 2008 --only "/\.(pdf)$/i"
Matt Sephton's user avatar
2 votes

Data about what kind of entities appear on public web pages

Not concrete answers, but here are some thoughts/places to find this: Common Crawl is a large corpus of crawled data, Web Data Commons are deep dives into the data. Read up on which are the most used ...
albert's user avatar
  • 11.9k
2 votes
Accepted

Lobby meetings of European Commission - Integrity Watch

With a hint from a person from Transparency International and some further browsing through the source code this link eventually made it work: http://static.tttp.eu/doi/data/meeting.csv
moezelot's user avatar
  • 153
2 votes

Scraping pages to get prices can be considered as OpenData

I think it depends of the nature of the site, not only the terms and conditions. In the case of privately owned health medicine information, it could be an exception because of the nature itself of ...
Offray's user avatar
  • 121
2 votes

How do I get historical data from coinmarketcap.com?

Here's a quick python3 script to get the table from each html page as a local csv file. basically, you have to create a loop over all dates and then make a web requests. the python library pandas has ...
philshem's user avatar
  • 17.6k
1 vote
Accepted

Catalog of all websites on the entire internet

Regarding git repo: it's not stale, it was updated about a month ago. Still, there aren't newer/larger and available datasets just yet. If you stumble upon one - just drop a line here, I'll merge it ...
Bohdan Turkynewych's user avatar
1 vote

Online Dictionary for Scraping

i found a website with 10,000 words that I'm using for my project. mabey this will work? https://www.mit.edu/~ecprice/wordlist.10000
hayden misiura's user avatar
1 vote

Online Dictionary for Scraping

You can use data from Wiktionary. Wikimedia provides data dumps for all its websites here (ctrl+f "wiktionary"), that way you don't have to scrape. Although the dumps seem to be regular, if you want ...
Peeyush Kushwaha's user avatar
1 vote

Places to get large (volume wise) datasets reasonably well formatted and free to share?

See the following for starters: Engima.io Public Data data.world Awesome Public Datasets
Carl V. Lewis's user avatar
1 vote

How to obtain airline data

I don't know the scope of airports you are after (US only, EU or WW) but I can't think of any public sources. I think that you will need to find a comparison site like kayak or expedia then use a ...
Marcus D's user avatar
  • 1,119
1 vote

How to obtain airline data

Daily airfares from where to where on which airline? Every flight everywhere or a specific flight on a specific airline...The data is public just go to a carriers site and do a query. If you a trying ...
risail's user avatar
  • 260
1 vote

Twitter crawlers for tweets, retweets and social network

With www.followthehashtag.com twitter analytics service you will be able to get Twitter data for free up to 9 days back or track some thousand tweets or, in premium version since 2006, with no total ...
Enrique R Rivera's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible