Take the 2-minute tour ×
Open Data Stack Exchange is a question and answer site for developers and researchers interested in open data. It's 100% free, no registration required.

While there are many open databases available, is there a database or the project that would contain the information of such databases?

In other words, is there an open meta-database of open databases?

share|improve this question
1  
What you're calling a 'meta-database', the library community calls a 'registry'. The science community calls it a 'metadata registry', but that's a problematic term as it has a completely different meaning to the DCMI folks. –  Joe May 13 '13 at 1:46
1  
@Joe CKAN calls it 'data catalog(ue)'. –  Patrick Hoefler May 14 '13 at 8:44
2  
Time after time, somebody compiles a list of sources, and then stops updating. This is sad... –  Deer Hunter May 25 '13 at 21:18
    
@DeerHunter That's what we call "standing on the shoulders of giants" ... –  Patrick Hoefler May 30 '13 at 21:12
    
knoema.com These guys have lots of datasets from open sources with metadata info around it like topic, source, region etc. Its easy to preview, browse and download as well. More details in my answer. HTH –  Mr.Hunt Dec 26 '13 at 10:22
show 1 more comment

17 Answers

up vote 43 down vote accepted

Currently there doesn't seem to exist a comprehensive database of open datasets. However, there are some places to start if one wanted to create such a meta-database:

share|improve this answer
add comment

There's also http://datacatalogs.org/ which maintains a list of open data portals.

You can get a full machine readable list (JSON) from there via the API: http://datacatalogs.org/api/search/dataset

That's only the first 10, to get 100 do: http://datacatalogs.org/api/search/dataset?limit=100

And with full info: http://datacatalogs.org/api/search/dataset?limit=100&all_fields=1

share|improve this answer
    
Thanks, I added datacatalogs.org to my answer :) –  Patrick Hoefler May 12 '13 at 21:45
add comment

Quandl (http://www.quandl.com) is an index of open datasets and sources, you can view a list of all of the sources here: http://www.quandl.com/about/sources

It not only pulls them into one place for easier access, but provides an API for each dataset and packages for them to be pulled into the analysis tool of choice.

share|improve this answer
add comment

Actually, with the data portals everyone is talking about, people forgot about Google. We Google for most things in life, why not data?

Does no-one remember the early days of the web, when we had portals that neatly catalogued web pages. It made sense to have people manually curating lists of pages, carefully categorizing them. Portals were cool. Much better than dumb search engines. But we all know which won.

Actually Google right now doesn't work too badly if you know the name of the dataset you're looking for. "But a search engine is no good for browsing through the topics, or providing useful search faceting, or all those other features data.gov.uk has." I hear you say.

Well y'know they have a big new idea recently that might enable all those features. They want everyone publishing a dataset to publish a metadata record alongside it on their server. (The format is called 'schema.org'.) Then when googlebot comes to spider the site, it finds the metadata and can then do all the topic browsing, faceting etc. that the average data portal does. But Google has the added advantage that it indexes all the data that doesn't have metadata too, and you can bet they can make a good attempt at guessing categories for those datasets too.

So in a couple of years Google (or Yahoo or Bing) might just run the finding data space. But not make the tools or metadata open. Tell me I'm wrong...

share|improve this answer
    
Portals => structured data | search engines => unstructured. We want data for algorithms not humans. So we need structured portals until such time as machines can read the web as well as we can. –  GreyCloud May 29 '13 at 15:57
    
Interesting, although all the schema.org meta that Google harvests will be structured and the rest could be provided in a structured way, if that provides value. –  D Read May 29 '13 at 20:05
    
search engines use structured data...see microformats, the most used semantic web format alive –  albert Feb 25 at 7:44
add comment

As far as I know, the most complete dataset index is IOGDS: International Open Government Dataset Search. It contains info about >1 million datasets.

Check it out at http://logd.tw.rpi.edu/node/9903

share|improve this answer
add comment

The Guardian data website has a variety and insights in to UK, European and World data. The site also includes data visualisationa and application you can use. Very enjoyable. http://www.guardian.co.uk/data

share|improve this answer
add comment

The Registry of Research Data Repositories just issued a press release announcing their existance, but their scope isn't necessarily all data, only 'research data'.

I assume they also contain closed data, based on one of their metadata fields:

Type of access to data: open

share|improve this answer
add comment

There are some great resources here. I would like to add http://knoema.com to this list.

They are possibly the most comprehensive and constantly updating resource. All their data is free to use and links back to source. Plus the data exploration tools and search engine is pretty useful.

Their dataset explorer is here: http://knoema.com/data

I found that they have some open data portals as well like http://opendataforafrica.org

Also found that they have good presence over chrome webstore: https://chrome.google.com/webstore/search/knoema

PS: I know some guys who work here and I think it is a good addition to list of resources here.

share|improve this answer
add comment

The "Datasets" and/or "OpenData" sub-reddits on reddit.com comprise something like that. for example.

share|improve this answer
add comment

Our open-date site has links to about 550 useful datasets from US and Canadian government and a handful from the UN. The US federal government datasets are sorted by department.

http://www.opengeocode.org/cude1.1/index.php

share|improve this answer
    
I know this is an odd question ... but what do you consider a 'dataset' to be? –  Joe Dec 26 '13 at 17:57
    
A dataset is a general term for anything that is a downloadable set of records that can be parsed and imported into a database. Examples include CSV files, tab-delimited files, XML/RDF, JSON objects, etc. I hope that helps. –  Andrew - OpenGeoCode Dec 26 '13 at 18:16
    
not as much as you might think ... I've been tracking language used to discuss scientific data systems, and your definition isn't clear if you'd consider a time-series of files (eg, one file per month or year) to be one dataset, or multiple datasets. –  Joe Dec 26 '13 at 18:26
    
Here's my opinion: A dataset has a collection period (e.g., monthly, yearly) and a dissemination date (when released). A time-series of files would be considered separate datasets. A revision to an existing dataset with an new release date would be a new dataset. –  Andrew - OpenGeoCode Dec 26 '13 at 18:37
    
Wow ... okay, I'll just keep citing the Renear, Sacchi & Wicket paper (Definitions of Dataset in the Scientific and Technical Literature), in which they effective came to the conclusion that there are some similarities, but every field has their own definition. (I think your definition is more similar to earth science, which makes sense if you deal with geocoding) –  Joe Dec 26 '13 at 18:58
show 1 more comment

The only thing that comes to mind is programmableweb... I guess you could also include this site although its still in closed beta.

share|improve this answer
1  
ProgrammableWeb is a collection of Web APIs, but to my knowledge it's not possible to only list open APIs. The second link might be a little self-referential ;) –  Patrick Hoefler May 12 '13 at 13:27
    
@PatrickHoefler good point. –  John May 12 '13 at 13:31
add comment

Have you looked into Linked Open Data (http://vimeo.com/36752317)? http://lod-cloud.net/ has a interactive diagram of open data sets (as at September 2011).

share|improve this answer
    
Unfortunately, clicking through an image map isn't quite a usable database of databases. –  David James May 24 '13 at 21:02
    
The data behind the LOD Cloud image is contained in The Data Hub, mentioned by Patrick –  D Read May 29 '13 at 20:08
add comment

Enigma.io is a repository for data that host all the data in its raw form on its own servers which allows you to search within the data as well download it through a REST API.

share|improve this answer
    
Enigma is a paid service as I can see. Am I right? If yes, it would be wise to update your answer and include this info. –  Anastasios Ventouris Jan 8 at 16:41
add comment

Also, this Quora thread lists quite a few others.

share|improve this answer
add comment
share|improve this answer
add comment

Google offers a couple more structured ways to search just on data sets beyond the main Google search function mentioned above:

1) A Googler on Twitter has suggested "try[ing] Google custom search (@googlecse) on pages that mention a http://schema.org/Dataset - http://datasets.schema-labs.appspot.com/" This search tool allows you to search for data sets that have been tagged using the Schema.org tag for data sets.

2) Google Fusion tables allows the public to search for tables that are either uploaded into Fusion tables or tables that Google has found on the web that could be uploaded into Fusion tables. In their words, "Web pages sometimes display high-quality structured data in a table. Many of these tables appear in Google Tables search results, dramatically expanding your ability to locate structured data. Once you find a good table, you may decide to import it to Fusion Tables."

share|improve this answer
add comment

I've compiled an index/catalog of government open data portals around the world. So far, there are over 575 sites in the catalog. I've setup the catalog for crowdsourcing, so feel free to send us your suggested portals to add to the catalog.

http://www.opengeocode.org/opendata/

You can download the index as a CSV file as well:

http://www.opengeocode.org/opendata/opendata.csv

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.