There are many web-based Application Programming Interfaces (API) that allow data-hungry people to get their hands on all sorts of data. It's a great way to collect data as-needed, instead of looking for an existing database or data set.
But how can I use web-based APIs to gather data?
APIs are often offered by websites so that developers can use the web-based data for apps, without having the uncertainty and difficulty of scraping the HTML. But it's not necessary to use the data to build apps, and this means that APIs can be a great source of data for research and analysis. Just to name a few types of API data: weather forecasts, historical stock prices, the twitter live-stream, wikipedia page views, ... and the list goes on and on.
Full list of APIs: ProgrammableWeb has a comprehensive list of APIs (more than 10,000). The list is so comprehensive that it's hard to know where to start. In the answers below you'll find details about popular APIs as well as some instructions for how to connect to data.
Authentication: Some APIs are free and open and don't require authentication, although they will often block your IP address if you make too many requests or request too fast. Other APIs require authentication but are free for basic usage. Each API has its own terms of service, and can restrict unlimited access by either capping the total calls per time period (i.e. 1000 per day), or by using a rate-limiting quota (i.e. 1 call per second).
Data formats: Data from APIs usually flows via a RESTful interface, and common formats are JSON, XML, and sometimes CSV (comma separated values).
Software and code: With a correct URL/URI, anyone can access the raw data from the browser (for example, JSON, XML, or even a CSV file). Here is a handy overview for non-programmers. Programming skills are usually needed to repeat or automate a process. There are some open source tools such as KNIME that allow non-programmers to connect to a RESTful interface and then collect and analyze data. For Python, see the Open Data Python Guide for examples of how to read and use the data files described above.
Wikipedia page views:
Wikipedia page views are useful for many things, including predicting changes in a stock price (paper)
A flood of views to a company's Wikipedia page may be a sign that their stock price is about to plummet. (news article)
You can access the raw data without any authentication, either by download or via an API.
It helps very much to first verify that you page and title exists. It does.
Sample python code to access page views for two articles ('Advisor' and 'Adviser') for all months between 2008 and 2014 (my reason). Note that days that don't exist (i.e. June 31st) will return 0 (zero) page views.
Weather data (historical):
Stock quotes
The Internet Archive
I found this website (http://www.datasciencetoolkit.org/). There is an API collection in there with great tools, especially to "clean" and prepare your open datasets for analysis and visualization.
A list of the tools:
Street Address to Coordinates
Street Address to Location calculates the latitude/longitude coordinates for a postal address. Currently only the US and UK have street-level detail.
Google-style Geocoder
Are you currently using Google's geocoding API and want to switch? Replace maps.googleapis.com with the address of a DSTK server and your code should work without changes.
Coordinates to Political Areas
Returns the country, region, state, county, constituencies and neighborhood a point is inside.
Text to Sentiment
Estimates whether a piece of text comment is complimentary or negative.
Coordinates to Statistics
Returns the population density, elevation, terrain, climate, and other conditions at a point.
Geodict
Geodict pulls country, city and region names from unstructured English text, and returns their coordinates.
IP Address to Coordinates
IP Address to Location calculates country, state, city and latitude/longitude coordinates for IP addresses.
Text to Sentences
Removes the parts of the text that seem to be boilerplate, leaving the real sentences.
HTML to Text
Returns the full text that would actually be displayed in the browser when an HTML document was rendered.
HTML to Story
Takes an HTML document representing a news article or similar page, and extracts just the story text.
Text to People
Spots text fragments that look like people's names or titles, and guesses their gender where possible.
Text to Times
Spots text fragments that look like times or dates, and converts them into a standard form.
File to Text
Converts PDFs, Word Documents, Excel Spreadsheets to text. Recovers text from JPEG, PNG or TIFF images of scanned documents.
It depends what data you want to gather. Twitter is a major source of data that is API-accessible. They have a REST API and a streaming API. There are also a lot of wrappers to make it easier to use those API's from your language of choice. For Twitter and other social media websites, I would suggest looking at the book "Mining the Social Web"
But this is just scratching the surface, and there are a lot of websites with APIs, and which you use depends on what type of data you are looking for.
Microformats are far more implemented than other "technologies", and plenty of big companies, pushing out a lot of data, implement them. There's no great advertisement for them, so you have to figure it out, although there are plug-ins that will tell you when a microformat can be mined/consumed/interacted with.
I would assume you could do the same with microdata/rdfa that's been implemented. So to your question, microformats + microdata + rfda are by far the most implemented APIs.
Also typically you're not going to have many problems/restrictions with API limits. The markup is the API. You just have to parse it, and there are libraries for that all over GitHub. Optimus would be the most popular/famous
http://microformatique.com/optimus/
and here's microformats parser for indieweb (which is something we all need to check out):
https://github.com/indieweb/php-mf2
commoncrawl.org has set up webcommonsdata, for getting slices of the web's data, and they specifically touch on crawling for microformats/rdfa in their posts as well.
Over half of the "semantic world" will fight me to death for promoting markup and microformats, but they're so easy, and they're all around us; and they are specifically what you are asking for. They provide a web API, and it's free and open. I don't want credit for the answer, I just cannot emphasize how crucial microformats are, how easy they are to implement, and why no one has an excuse to not use them. Most solid webdevs have been publishing them for years; it's near second-hand knowledge, or rather, the classes just make up patterns I embed into practically everything I do.
