17

APIs are often offered by websites so that developers can use the web-based data for apps, without having the uncertainty and difficulty of scraping the HTML. But it's not necessary to use the data to build apps, and this means that APIs can be a great source of data for research and analysis. Just to name a few types of API data: weather forecasts, ...


12

You can find plenty of summary data, but I have not seen any publicly available raw counter data. Here's some summaries: Federal Highway Administration Traffic Volume Trends is a monthly report based on hourly traffic count data reported by the States. These data are collected at approximately 4,000 continuous traffic counting locations nationwide and are ...


10

There's a big difference between “available to the public” and “belonging to the public”. By accessing a privately-owned website, you are accepting its terms and conditions. These conditions typically preclude scraping and aggregation. Here's an example from Amazon's (Canada) Licence and Access terms: Subject to your compliance with these Conditions of Use ...


8

If the crawled data doesn't need to be very recent, the Internet Archive provides 80 terabytes of archived web crawl data from 2011 for research. Unfortunately, they don't say under which license they release the data, so it might not be Open Data as defined by the Open Definition.


6

here's how i automated the process within R setwd( "C:/path/to/PDFs" ) year <- 2008 domain_name <- "anthem.com" archive_org_captures <- readLines( paste0( "http://web.archive.org/cdx/search/cdx?url=" , domain_name , "/about/&matchType=host&limit=10000000&from=" , year , ...


5

This site helps to calculate the cost of living in different parts of the United States. There's a caveat about the fidelity and comprehensiveness of the data ("Consider the results a minimum cost threshold that serves as a benchmark, but only that."). For licensing, the site references that it is part of the Living Wage Project. However, in looking ...


5

First, you should check for any license for using the site to see if you can use their data for any reason (commercial and non-commercial). Second, you should read the terms of use and find anything about scraping website. Many websites don't allow scraping. Third, if non of the above is a problem, you should use a programming language to crawl and scrap ...


5

I found this website (http://www.datasciencetoolkit.org/). There is an API collection in there with great tools, especially to "clean" and prepare your open datasets for analysis and visualization. A list of the tools: Street Address to Coordinates Street Address to Location calculates the latitude/longitude coordinates for a postal address. ...


4

Alexa is a division of Amazon. They still have an API, but you need to access it as a paid service through AWS: Alexa Web Information Service http://aws.amazon.com/awis/ and Alexa Top Sites http://aws.amazon.com/alexatopsites/


4

I know that 1-2 years ago, Alexa had a free API for this on data.alexa.com. Now it returns a 404 error. There are a few software out there that promise data from Alexa but actually they scrap the website which is illegal. The only way I know is via Amazon web servers but it is a paid service. 0.15$/1000 requests. You can see details here AWS. The price ...


4

Microformats are far more implemented than other "technologies", and plenty of big companies, pushing out a lot of data, implement them. There's no great advertisement for them, so you have to figure it out, although there are plug-ins that will tell you when a microformat can be mined/consumed/interacted with. I would assume you could do the same with ...


4

It depends what data you want to gather. Twitter is a major source of data that is API-accessible. They have a REST API and a streaming API. There are also a lot of wrappers to make it easier to use those API's from your language of choice. For Twitter and other social media websites, I would suggest looking at the book "Mining the Social Web" But this ...


4

Have you looked at: http://datasift.com/ it is where I would start. http://gnip.com/ would be another good option.


4

You can download the top 1 million (ZIP) sites from Alexa.


4

If you are interested to just scrape some data from web pages, then like @magdmartin wrote, you just need to write some code to download the HTML (python requests package) and then to parse the HTML (BeautifulSoup in this case). import requests from bs4 import BeautifulSoup import re r = requests.get('https://www.canlii.org/en/ca/scc/doc/2007/2007scc4/...


4

You may get better answers on CodeReview, but just glancing at your code I have one suggestion that is to split the codes into a scraping and parsing steps. This will help with reproducibility and also by decoupling the steps you can easier debug. Scraping - the goal should be to save HTML files saved on the disk with names describing what they are and when ...


4

Some tips for scrapers: log: don't stop at "sometimes I get data and sometimes I don't" -- when you get unexpected results in the response, log it so you can learn what happened. Specifically when you are being throttled for making too many requests, you will often get an explicit message saying so. cache: write your code to save files you retrieve and use ...


4

What about StackOverflow's own dataset? https://archive.org/details/stackexchange


4

You're probably going to need to get information from many jurisdictions - at least in the US, there is not a good/granular sources for property tax, crime, and other information for all states in one dataset. For crime data, check out FBI UCR reports which are usually not granular. Each city may (hopefully) report their own data. Try google and/or http://...


3

In the UK the Computer Misuse Act from 1990 (section 1) states that it is an offence to access a computer where: the access to the data is unauthorised the offender knows that it is unauthorised This may extend to web scraping because, in some circumstances, you are accessing the website intending to use the data in a way that is not authorised by the ...


3

Here's a good starting place (compiled by Columbia University): Nigeria: http://library.columbia.edu/locations/global/virtual-libraries/african_studies/countries/nigeria/online.html Tanzania: http://library.columbia.edu/locations/global/virtual-libraries/african_studies/countries/tanzania/online.html


3

From the ACM terms of usage page To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Send written requests for republication to ACM Publications, Copyright & Permissions at the address above or fax +1 (212) 869-0481 or email permissions@acm.org. Thus, I believe ...


3

If you are affiliated with an academic institution that subscribes to IEEE/ACM material, talk to your library. They may be able to negotiate access on your behalf. Chances are fair it isn't the first such request they've heard.


3

I'm pretty sure there aren't, but your best bet would probably be to ask on the Common Crawl mailing list. That's a place where people who are interested in this kind of stuff hang out and it's low enough traffic that a polite request wouldn't be out of line -- the answer would probably be of interest to the entire community there.


3

The best example I have heard of is Real-time traffic monitoring using mobile phone data (PDF). The idea is to derive road traffic velocity from the position data that the mobile phones within cars "generate" when moving from one base station to the next. The frequency of these base station handshakes approximates the travel velocity of the car. Practical ...


3

TubeKit might be of interest to you: TubeKit is a toolkit for creating YouTube crawlers. It allows one to build one's own crawler that can crawl YouTube based on a set of seed queries and collect up to 16 different attributes. The tool is open source (licensed under CC BY-NC-SA*) and has been developed for research purposes. * Creative Commons advise ...


3

On Linux use youtube-dl command to download videos straight off of YouTube. To install youtube-dl run sudo pip install youtube-dl To download an entire channel: youtube-dl -citw ytuser:<USER> It is open source, and well-maintained, modifications in YouTube are usually taken care of very quickly. Answer stolen from josten at https://askubuntu.com/...


3

This doesn't solve the problem entirely, but I'll add it here for anyone else who has the same issue. (but not mark it as the answer, in case someone else has a better solution) There's a program xidel which allows you to retrieve a URL and extract information from the page using XPath queries. See this response to a similar question on Stack Overflow. ...


3

Using --reject-regex "\?.=.;.=." seems to work for me to avoid the extra listings provided by Apache. The full command-line I use to get all the files from an Apache directory listing is: wget --recursive --no-parent --timestamping --no-directories --reject-regex "\?.=.;.=." http://example.com/some/dir/


3

Here's a link to datasets (Excel spreadsheets) from a NYU professor whom has been keeping corporate finance data on major corporations for 20 years: http://pages.stern.nyu.edu/~adamodar/New_Home_Page/data.html Filter the companies with classification "Software (Internet)", there are 759 of them. I generated the list at https://gist.github.com/nicolas-raoul/...


Only top voted, non community-wiki answers of a minimum length are eligible