10
votes
Accepted
Scraping pages to get prices can be considered as OpenData
There's a big difference between “available to the public” and “belonging to the public”. By accessing a privately-owned website, you are accepting its terms and conditions. These conditions typically ...
- 555
6
votes
Accepted
how to search archive.org for PDF files on a captured website between some date range
here's how i automated the process within R
setwd( "C:/path/to/PDFs" )
year <- 2008
domain_name <- "anthem.com"
archive_org_captures <-
readLines(
paste0(
"http://...
- 1,480
4
votes
Accepted
Data sets for predicting home value
You're probably going to need to get information from many jurisdictions - at least in the US, there is not a good/granular sources for property tax, crime, and other information for all states in one ...
- 5,174
4
votes
how to search archive.org for PDF files on a captured website between some date range
Here is one way, although the API probably has more efficient methods.
You can append * to the end of a URL in the Wayback Machine and it will return all of the saved URLs for that domain. The link ...
- 11.9k
4
votes
Exclusion lists when crawling web directories
Using --reject-regex "\?.=.;.=." seems to work for me to avoid the extra listings provided by Apache. The full command-line I use to get all the files from an Apache directory listing is:
wget --...
- 141
3
votes
The list of restricted second-level domains such as .co.uk and .ac.jp
This "restricted domain" is known an effective top-level domain (eTLD)", or a suffix. By Mozilla:
"A "public suffix" is one under which Internet users can (or historically could) directly register ...
- 3,641
3
votes
Places to get large (volume wise) datasets reasonably well formatted and free to share?
USPTO has a Bulk Data sub domain which has many datasets that fit this description for you to pick through. Here's a description of two:
Patent Grant Single-Page TIFF Images (July 31, 1790 - ...
- 11.9k
3
votes
Accepted
What tool can I use to manage a researcher network bibliography?
ORCID provides a persistent digital identifier that distinguishes you from every other researcher and, through integration in key research workflows such as manuscript and grant submission, supports ...
- 11.9k
3
votes
Creating data from web tables with import.io failed - other tools?
Data Scraping Studio can do this and it's free software.
I just created a scraping agent using this tool and extracted all tables into one using a REGEX parser. The extracted data should be exactly ...
- 131
3
votes
Datasets: Is there any public data-set for FAQ (Frequently asked questions) in different domains?
What about StackOverflow's own dataset? https://archive.org/details/stackexchange
- 131
2
votes
List of political blogs
There is a blog that lists political blogs based in the US:
http://politicalbloglistings.blogspot.ch/
Categories (count of listed blogs):
Conservative (108)
Liberal (70)
Non/Bi-Partisan or ...
- 17.6k
2
votes
Employer's website is "web scraping" from another site, can I be liable even though I did not write that code?
I would have to disagree with the other answer. It has got to do with a lot more than that. Did your employer at the time know his website was scraping? If so, you are in no trouble at all. Writing ...
- 349
2
votes
Employer's website is "web scraping" from another site, can I be liable even though I did not write that code?
Firstly, the information provided in my answers is for general information purposes only and is not intended to be legal advice for individual situations and should not be relied on in taking legal ...
- 379
2
votes
Creating data from web tables with import.io failed - other tools?
you can use google docs importHtml functionality to get what you want. add =ImportHtml("URL", "table", num) into a spread sheet, replacing URL with the HTML document's URL, and num with the HTML <...
- 11.9k
2
votes
The "right to mine" and scraping the web in EU
The article in the LSE blog you have pointed out seems to be a good overview, the problems seem to be not only UK specific:
Even with some legal barriers now removed, technical barriers remain
...
2
votes
how to search archive.org for PDF files on a captured website between some date range
Using the official wayback_machine_downloader tool:
wayback_machine_downloader https://www.anthem.com --from 2005 --to 2008 --only "/\.(pdf)$/i"
- 121
2
votes
Data about what kind of entities appear on public web pages
Not concrete answers, but here are some thoughts/places to find this:
Common Crawl is a large corpus of crawled data, Web Data Commons are deep dives into the data.
Read up on which are the most used ...
- 11.9k
2
votes
Accepted
Lobby meetings of European Commission - Integrity Watch
With a hint from a person from Transparency International and some further browsing through the source code this link eventually made it work:
http://static.tttp.eu/doi/data/meeting.csv
- 153
2
votes
Scraping pages to get prices can be considered as OpenData
I think it depends of the nature of the site, not only the terms and conditions. In the case of privately owned health medicine information, it could be an exception because of the nature itself of ...
- 121
2
votes
How do I get historical data from coinmarketcap.com?
Here's a quick python3 script to get the table from each html page as a local csv file.
basically, you have to create a loop over all dates and then make a web requests. the python library pandas has ...
- 17.6k
1
vote
Accepted
Catalog of all websites on the entire internet
Regarding git repo: it's not stale, it was updated about a month ago. Still,
there aren't newer/larger and available datasets just yet. If you stumble upon one - just drop a line here, I'll merge it ...
1
vote
Online Dictionary for Scraping
i found a website with 10,000 words that I'm using for my project. mabey this will work?
https://www.mit.edu/~ecprice/wordlist.10000
1
vote
Online Dictionary for Scraping
You can use data from Wiktionary.
Wikimedia provides data dumps for all its websites here (ctrl+f "wiktionary"), that way you don't have to scrape.
Although the dumps seem to be regular, if you want ...
- 111
1
vote
Places to get large (volume wise) datasets reasonably well formatted and free to share?
See the following for starters:
Engima.io Public Data
data.world
Awesome Public Datasets
- 579
1
vote
How to obtain airline data
I don't know the scope of airports you are after (US only, EU or WW) but I can't think of any public sources.
I think that you will need to find a comparison site like kayak or expedia then use a ...
- 1,104
1
vote
How to obtain airline data
Daily airfares from where to where on which airline? Every flight everywhere or a specific flight on a specific airline...The data is public just go to a carriers site and do a query. If you a trying ...
- 260
1
vote
Data sets for predicting home value
Zillow has a free API that will return their estimate of average home prices. You can get the data down to zipcode and neighborhood level.
http://www.zillow.com/howto/api/APIOverview.htm
- 8,647
1
vote
Data sets for predicting home value
I guess you are looking for data like this
https://archive.ics.uci.edu/ml/datasets/Housing
- 121
1
vote
african newpapers for the past 20 years for machine learning
I've only just seen your query, Krishna, so am not certain whether you still need the info, but Code for Africa might be able to help for machine readable archives of newspaper content from Kenya, ...
Only top scored, non community-wiki answers of a minimum length are eligible