Take the 2-minute tour ×
Open Data Stack Exchange is a question and answer site for developers and researchers interested in open data. It's 100% free, no registration required.

There are many web-based Application Programming Interfaces (API) that allow data-hungry people to get their hands on all sorts of data. It's a great way to collect data as-needed, instead of looking for an existing database or data set.

But how can I use web-based APIs to gather data?

share|improve this question
2  
What exactly is your question? –  vanthome Mar 6 at 14:55
add comment

4 Answers 4

APIs are often offered by websites so that developers can use the web-based data for apps, without having the uncertainty and difficulty of scraping the HTML. But it's not necessary to use the data to build apps, and this means that APIs can be a great source of data for research and analysis. Just to name a few types of API data: weather forecasts, historical stock prices, the twitter live-stream, wikipedia page views, ... and the list goes on and on.

Full list of APIs: ProgrammableWeb has a comprehensive list of APIs (more than 10,000). The list is so comprehensive that it's hard to know where to start. In the answers below you'll find details about popular APIs as well as some instructions for how to connect to data.

Authentication: Some APIs are free and open and don't require authentication, although they will often block your IP address if you make too many requests or request too fast. Other APIs require authentication but are free for basic usage. Each API has its own terms of service, and can restrict unlimited access by either capping the total calls per time period (i.e. 1000 per day), or by using a rate-limiting quota (i.e. 1 call per second).

Data formats: Data from APIs usually flows via a RESTful interface, and common formats are JSON, XML, and sometimes CSV (comma separated values).

Software and code: With a correct URL/URI, anyone can access the raw data from the browser (for example, JSON, XML, or even a CSV file). Here is a handy overview for non-programmers. Programming skills are usually needed to repeat or automate a process. There are some open source tools such as KNIME that allow non-programmers to connect to a RESTful interface and then collect and analyze data. For Python, see the Open Data Python Guide for examples of how to read and use the data files described above.


Here are just a handful of examples of the diverse and useful web-based API data

Wikipedia page views:

  • stats.grok.se

    • Wikipedia page views are useful for many things, including predicting changes in a stock price (paper)

      A flood of views to a company's Wikipedia page may be a sign that their stock price is about to plummet. (news article)

    • You can access the raw data without any authentication, either by download or via an API.

    • For the API, you construct a URL like this: http://stats.grok.se/json/en/201306/Data , which would give you the page views in JSON format for the article 'Data' from June, 2013.
    • It helps very much to first verify that you page and title exists. It does.

    • Sample python code to access page views for two articles ('Advisor' and 'Adviser') for all months between 2008 and 2014 (my reason). Note that days that don't exist (i.e. June 31st) will return 0 (zero) page views.


Weather data (historical):

  • Wunderground.com

    • 500 free API calls per day (with authentication), and offers historical detailed daily weather data going back to 1997 for my location (details).
    • International weather, so it is not only restricted to the US.
    • Data comes in JSON format (example data, click "show response").
    • Sample python code for current and historical weather.

Stock quotes


The Internet Archive


Google Maps Geolocation API

  • Pass search string and receive latitude, longitude and other data.
  • 100 free requests per 24 hours (no authentication required)
  • Example lookup
  • Python code sample

Freebase API:

  • Freebase is a structured database based on community editors.
  • Owned by Google and is the source of their Knowledge Graph.
  • Freely available for commercial and non-commercial (with Creative Commons Attribution License)
  • Data can be accessed via browser, API, RDF endpoint, and database dump.
    • For API access, authentication is required via Google's API Console
    • Once authenticated, user can read, search and download.
    • One example of Python library: freebase

Although library not needed - sample code designed to retreive a list of golf courses, in an attempt to answer this OD question. In the end, OSM answer via @Oleg was better.

import json, urllib
api_key = open(".api_key").read()
service_url = 'https://www.googleapis.com/freebase/v1/search'

query = ''
filte = '(any type:/sports/golf_course )'
params = {'query': query, 'key': api_key, 'filter': filte}
url = service_url + '?' + urllib.urlencode(params)
response = json.loads(urllib.urlopen(url).read())

print response.get(u'hits','')
for result in response[u'result']:
    print result.get(u'name'),result.get(u'mid','') 

The output looks like this:

Old Course at St Andrews /m/02d78x
Pebble Beach Golf Links /m/06vwtf
....

Where the Machine ID (/m/id) is then typically used to to make other queries or find more information.

share|improve this answer
add comment

I found this website (http://www.datasciencetoolkit.org/). There is an API collection in there with great tools, especially to "clean" and prepare your open datasets for analysis and visualization.

A list of the tools:

  • Street Address to Coordinates

Street Address to Location calculates the latitude/longitude coordinates for a postal address. Currently only the US and UK have street-level detail.

  • Google-style Geocoder

Are you currently using Google's geocoding API and want to switch? Replace maps.googleapis.com with the address of a DSTK server and your code should work without changes.

  • Coordinates to Political Areas

Returns the country, region, state, county, constituencies and neighborhood a point is inside.

  • Text to Sentiment

Estimates whether a piece of text comment is complimentary or negative.

  • Coordinates to Statistics

Returns the population density, elevation, terrain, climate, and other conditions at a point.

  • Geodict

Geodict pulls country, city and region names from unstructured English text, and returns their coordinates.

  • IP Address to Coordinates

IP Address to Location calculates country, state, city and latitude/longitude coordinates for IP addresses.

  • Text to Sentences

Removes the parts of the text that seem to be boilerplate, leaving the real sentences.

  • HTML to Text

Returns the full text that would actually be displayed in the browser when an HTML document was rendered.

  • HTML to Story

Takes an HTML document representing a news article or similar page, and extracts just the story text.

  • Text to People

Spots text fragments that look like people's names or titles, and guesses their gender where possible.

  • Text to Times

Spots text fragments that look like times or dates, and converts them into a standard form.

  • File to Text

Converts PDFs, Word Documents, Excel Spreadsheets to text. Recovers text from JPEG, PNG or TIFF images of scanned documents.

share|improve this answer
add comment

microformats are far more implemented than other "technologies", and plenty of big companies, pushing out alot of data implement them. there's no great advertisement for them, so you have to figure it out, although there are plug-ins that will tell you when a microformat can be mined/consumed/interacted with..
i would assume you could do the same with microdata/rdfa thats been implemented.
so to your question, microformats + microdata + rfda are by far the most implemented apis...
also typically you're not going to have many problems/restrictions with api limits...the markup is the api. you just have to parse it, and there are libraries for that all over github...optimus would be the most popular/famous
http://microformatique.com/optimus/
and here's microformats parser for indieweb (which is something we all need to check out):
https://github.com/indieweb/php-mf2

commoncrawl.org has set up webcommonsdata, for getting slices of the web's data, and they specifically touch on crawling for microformats/rdfa in their posts as well.

over half of the "semantic world" will fight me to death for promoting markup and microformats, but they're so easy, and they're all around us; and they are specifically what you are asking for. they provide a web api. and its free and open. i don't want credit for the answer, i just cannot emphasize how crucial microformats are, how easy they are to implement, and why no one has an excuse to not use them. most solid webdevs have been publishing them for years; its near second-hand knowledge, or rather, the classes just make up patterns i embed into practically everything i do. lol..."i embed....markup classes" says the faux hackz0r!

share|improve this answer
    
And if you're planning on publishing something using microformats, see schema.org –  Joe Apr 8 at 15:17
    
not at all, quite the contrary, microformats.org. schema needs microformats, pretty sure they haven't adopted any of schema's though –  albert Apr 8 at 18:42
add comment

It depends what data you want to gather. Twitter is a major source of data that is API-accessible. They have a REST API and a streaming API. There are also a lot of wrappers to make it easier to use those API's from your language of choice. For Twitter and other social media websites, I would suggest looking at the book "Mining the Social Web"

But this is just scratching the surface, and there are a lot of websites with APIs, and which you use depends on what type of data you are looking for.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.