APIs are often offered by websites so that developers can use the web-based data for apps, without having the uncertainty and difficulty of scraping the HTML. But it's not necessary to use the data to build apps, and this means that APIs can be a great source of data for research and analysis. Just to name a few types of API data: weather forecasts, ...
You can find plenty of summary data, but I have not seen any publicly available raw counter data. Here's some summaries:
Federal Highway Administration
Traffic Volume Trends is a monthly report based on hourly traffic count data reported by the States. These data are collected at approximately 4,000 continuous traffic counting locations nationwide and are ...
There's a big difference between “available to the public” and “belonging to the public”. By accessing a privately-owned website, you are accepting its terms and conditions. These conditions typically preclude scraping and aggregation. Here's an example from Amazon's (Canada) Licence and Access terms:
Subject to your compliance with these Conditions
of Use ...
If the crawled data doesn't need to be very recent, the Internet Archive provides 80 terabytes of archived web crawl data from 2011 for research. Unfortunately, they don't say under which license they release the data, so it might not be Open Data as defined by the Open Definition.
here's how i automated the process within R
setwd( "C:/path/to/PDFs" )
year <- 2008
domain_name <- "anthem.com"
This site helps to calculate the cost of living in different parts of the United States. There's a caveat about the fidelity and comprehensiveness of the data ("Consider the results a minimum cost threshold that serves as a benchmark, but only that.").
For licensing, the site references that it is part of the Living Wage Project. However, in looking ...
First, you should check for any license for using the site to see if you can use their data for any reason (commercial and non-commercial).
Third, if non of the above is a problem, you should use a programming language to crawl and scrap ...
I found this website (http://www.datasciencetoolkit.org/). There is an API collection in there with great tools, especially to "clean" and prepare your open datasets for analysis and visualization.
A list of the tools:
Street Address to Coordinates
Street Address to Location calculates the latitude/longitude
coordinates for a postal address. ...
Alexa is a division of Amazon. They still have an API, but you need to access it as a paid service through AWS:
Alexa Web Information Service
Alexa Top Sites
I know that 1-2 years ago, Alexa had a free API for this on data.alexa.com. Now it returns a 404 error.
There are a few software out there that promise data from Alexa but actually they scrap the website which is illegal.
The only way I know is via Amazon web servers but it is a paid service. 0.15$/1000 requests. You can see details here AWS.
The price ...
Microformats are far more implemented than other "technologies", and plenty of big companies, pushing out a lot of data, implement them. There's no great advertisement for them, so you have to figure it out, although there are plug-ins that will tell you when a microformat can be mined/consumed/interacted with.
I would assume you could do the same with ...
It depends what data you want to gather. Twitter is a major source of data that is API-accessible. They have a REST API and a streaming API. There are also a lot of wrappers to make it easier to use those API's from your language of choice. For Twitter and other social media websites, I would suggest looking at the book "Mining the Social Web"
But this ...
If you are interested to just scrape some data from web pages, then like @magdmartin wrote, you just need to write some code to download the HTML (python requests package) and then to parse the HTML (BeautifulSoup in this case).
from bs4 import BeautifulSoup
r = requests.get('https://www.canlii.org/en/ca/scc/doc/2007/2007scc4/...
You may get better answers on CodeReview, but just glancing at your code I have one suggestion that is to split the codes into a scraping and parsing steps. This will help with reproducibility and also by decoupling the steps you can easier debug.
Scraping - the goal should be to save HTML files saved on the disk with names describing what they are and when ...
Some tips for scrapers:
log: don't stop at "sometimes I get data and sometimes I don't" -- when you get unexpected results in the response, log it so you can learn what happened. Specifically when you are being throttled for making too many requests, you will often get an explicit message saying so.
cache: write your code to save files you retrieve and use ...
You're probably going to need to get information from many jurisdictions - at least in the US, there is not a good/granular sources for property tax, crime, and other information for all states in one dataset.
For crime data, check out FBI UCR reports which are usually not granular. Each city may (hopefully) report their own data. Try google and/or http://...
California Performance measurement system (PEMS) has 5 minute and raw sensor data, as well as many levels of aggregation: http://pems.dot.ca.gov/
EDIT: Been looking through this and it's an awesome source. The raw data is every 30 seconds on every single freeway sensor in California which leads to 10's of millions of records every day. And you can download ...
In the UK the Computer Misuse Act from 1990 (section 1) states that it is an offence to access a computer where:
the access to the data is unauthorised
the offender knows that it is unauthorised
This may extend to web scraping because, in some circumstances, you are accessing the website intending to use the data in a way that is not authorised by the ...
Here's a good starting place (compiled by Columbia University):
From the ACM terms of usage page
To copy otherwise, to republish, to post on servers, or to
redistribute to lists, requires prior specific permission and/or a
fee. Send written requests for republication to ACM Publications,
Copyright & Permissions at the address above or fax +1 (212) 869-0481
or email email@example.com.
Thus, I believe ...
If you are affiliated with an academic institution that subscribes to IEEE/ACM material, talk to your library. They may be able to negotiate access on your behalf. Chances are fair it isn't the first such request they've heard.
I'm pretty sure there aren't, but your best bet would probably be to ask on the Common Crawl mailing list. That's a place where people who are interested in this kind of stuff hang out and it's low enough traffic that a polite request wouldn't be out of line -- the answer would probably be of interest to the entire community there.
The best example I have heard of is Real-time traffic monitoring using mobile
phone data (PDF).
The idea is to derive road traffic velocity from the position data that the mobile phones within cars "generate" when moving from one base station to the next. The frequency of these base station handshakes approximates the travel velocity of the car. Practical ...
TubeKit might be of interest to you:
TubeKit is a toolkit for creating YouTube crawlers. It allows one to build one's own crawler that can crawl YouTube based on a set of seed queries and collect up to 16 different attributes.
The tool is open source (licensed under CC BY-NC-SA*) and has been developed for research purposes.
* Creative Commons advise ...
On Linux use youtube-dl command to download videos straight off of YouTube.
To install youtube-dl run sudo pip install youtube-dl
To download an entire channel:
youtube-dl -citw ytuser:<USER>
It is open source, and well-maintained, modifications in YouTube are usually taken care of very quickly.
Answer stolen from josten at https://askubuntu.com/...
This doesn't solve the problem entirely, but I'll add it here for anyone else who has the same issue. (but not mark it as the answer, in case someone else has a better solution)
There's a program xidel which allows you to retrieve a URL and extract information from the page using XPath queries. See this response to a similar question on Stack Overflow.
Using --reject-regex "\?.=.;.=." seems to work for me to avoid the extra listings provided by Apache. The full command-line I use to get all the files from an Apache directory listing is:
wget --recursive --no-parent --timestamping --no-directories --reject-regex "\?.=.;.=." http://example.com/some/dir/