159

While there are many open databases available, is there a database or the project that would contain the information of such databases?

In other words, is there an open meta-database of open databases?

Improve this question
5

29 Answers 29

Reset to default
100

The short answer

DataPortals.org (previously DataCatalogs.org) provides a comprehensive list of open data portals from around the world. Their (meta-)data is in the public domain and available for download as CSV and JSON.

The longer answer

Data that is somehow related is usually grouped in datasets or databases, contained in files (e.g. CSV or spreadsheets) or some kind of database management system, which might be accessible via an API.

In the context of Open Data, data portals, data catalogs, or data hubs make it easier to find these datasets or databases.

A great example of such a data portal is the Datahub, which currently lists more than 4,500 open datasets.

However, there are already hundreds of data portals. A few prominent examples are the official data portals of the US (data.gov), the UK (data.gov.uk), or the European Union (open-data.europa.eu).

This is where DataPortal.org comes in: It is a data portal of data portals.

To sum it up:

    DataPortals.org --> Specific Data Portal --> Specific Data Set --> Open Data
Improve this answer
2
  • Let's not forget OpenDataMonitor: opendatamonitor.eu
    – Ulrich
    Commented Aug 5, 2015 at 12:49
  • OpenDataMonitor was promising for a while - 218 european data portals listed (compared to 520 at dataportals.org for the whole world). But the ODM project ended about Sept/October 2015 - funding finished, no website updates. It's not clear if ODI continue to update their spreadsheet data. There is a Share-Alike clause on their data so it can't be merged into DataPortals.org data.
    – D Read
    Commented Aug 30, 2016 at 14:57
35

There's also http://datacatalogs.org/ which maintains a list of open data portals.

You can get a full machine readable list (JSON) from there via the API: http://datacatalogs.org/api/search/dataset

That's only the first 10, to get 100 do: http://datacatalogs.org/api/search/dataset?limit=100

And with full info: http://datacatalogs.org/api/search/dataset?limit=100&all_fields=1

Improve this answer
0
18

Quandl is an index of datasets. It includes open datasets avaliable for free as well as premium databases which are only available for a cost.

It not only pulls them into one place for easier access, but provides an API for each dataset and packages for them to be pulled into the analysis tool of choice.

Improve this answer
1
  • quandl.com/resources/data-sources lists its sources - 180 at present, mainly business, economic and international development organizations. The site is closed source and for-profit, using a freemium model.
    – D Read
    Commented Mar 12, 2015 at 9:46
15

As of February 2019, you'll find a list of 2600+ open data portals around the world on OpenDataSoft's website. It lists as of now 2600+ portals ranging from the US (obviously) to Afghanistan.

The story behind it:

Working for a SaaS company in need of loads of structured data, our team started to compile a list of all open data portals around the world as a go-to resource.

We ended up with a list of more than 1600 portals. We gathered our own listings, scrapped third-party datasets (the likes of dataportals.org), cleaned the whole thing (lot of elbow grease + Clojure) and created a list (w/ Ruby).

Instead of keeping it in a dusty corner of our computers, we thought we'll share it with the open data community / data geeks.

Major work was done on cleaning the data (many duplicates), adding geo-coordinates to every portals, crossing off the list all dead portals/urls...

First, we used the dataset generated to create a website called opendatainception.io where you can now browse data on a map. You can browse data by navigating or through the search box. When typing a query there, the data will automatically refine on the map.

Second, we shared the list as a go-to resource for everyone. At first we were building the list as an HTML list from the dataset with some Ruby script. It was a kinda pain and not always super reliable. To be more efficient and reflect the changes instantly as we were making them, we went for some open source widgets instead (built w/ angular).

Now, the page displays a dynamic list, always synced up with the dataset. You still can look for countries and stuff.

You can find the list here and we explained the how-to there.

Hope that'll help!

If you have any feedback or want to add a portal, give us a shout!

enter image description here

Improve this answer
13

As far as I know, the most complete dataset index is IOGDS: International Open Government Dataset Search. It contains info about >1 million datasets.

Check it out at http://logd.tw.rpi.edu/node/9903

Improve this answer
0
11

Actually, with the data portals everyone is talking about, people forgot about Google. We Google for most things in life, why not data?

Does no-one remember the early days of the web, when we had portals that neatly catalogued web pages. It made sense to have people manually curating lists of pages, carefully categorizing them. Portals were cool. Much better than dumb search engines. But we all know which won.

Actually Google right now doesn't work too badly if you know the name of the dataset you're looking for. "But a search engine is no good for browsing through the topics, or providing useful search faceting, or all those other features data.gov.uk has." I hear you say.

Well y'know they have a big new idea recently that might enable all those features. They want everyone publishing a dataset to publish a metadata record alongside it on their server. (The format is called 'schema.org'.) Then when googlebot comes to spider the site, it finds the metadata and can then do all the topic browsing, faceting etc. that the average data portal does. But Google has the added advantage that it indexes all the data that doesn't have metadata too, and you can bet they can make a good attempt at guessing categories for those datasets too.

So in a couple of years Google (or Yahoo or Bing) might just run the finding data space. But not make the tools or metadata open. Tell me I'm wrong...

Improve this answer
6
  • 3
    Portals => structured data | search engines => unstructured. We want data for algorithms not humans. So we need structured portals until such time as machines can read the web as well as we can.
    – GreyCloud
    Commented May 29, 2013 at 15:57
  • Interesting, although all the schema.org meta that Google harvests will be structured and the rest could be provided in a structured way, if that provides value.
    – D Read
    Commented May 29, 2013 at 20:05
  • 2
    search engines use structured data...see microformats, the most used semantic web format alive
    – albert
    Commented Feb 25, 2014 at 7:44
  • 1
    google.com/publicdata/directory
    – Sun
    Commented Mar 11, 2015 at 17:25
  • @sunk818 Google Public Data Explorer is very limited in scope, rather than the comprehensive catalogue of the world's data that I suggested they might choose to do. GPDE has only 136 datasets, for which they done the manual work of putting into a database, so that they can offer visualizations. I can't see any activity on it since launch in 2010.
    – D Read
    Commented Mar 12, 2015 at 9:52
9

Our open-date site has links to about 600 useful datasets from US and Canadian government and a handful from the UN. The US federal government datasets are sorted by department.

http://www.opengeocode.org/cude1.1/index.php

We [opengeocode.org] maintain a catalog of open data portals across the world. We've categorized them under the categories:

  1. Data Portal (multi-category)
  2. Transparency Portal (budget, payroll, expenditures, etc)
  3. GIS/Gazetteer (maps, geolocation data, etc)
  4. Census/Demographics
  5. Health
  6. Education

http://www.opengeocode.org/opendata/

Improve this answer
6
  • I know this is an odd question ... but what do you consider a 'dataset' to be?
    – Joe
    Commented Dec 26, 2013 at 17:57
  • A dataset is a general term for anything that is a downloadable set of records that can be parsed and imported into a database. Examples include CSV files, tab-delimited files, XML/RDF, JSON objects, etc. I hope that helps.
    – Andrew - OpenGeoCode
    Commented Dec 26, 2013 at 18:16
  • not as much as you might think ... I've been tracking language used to discuss scientific data systems, and your definition isn't clear if you'd consider a time-series of files (eg, one file per month or year) to be one dataset, or multiple datasets.
    – Joe
    Commented Dec 26, 2013 at 18:26
  • Here's my opinion: A dataset has a collection period (e.g., monthly, yearly) and a dissemination date (when released). A time-series of files would be considered separate datasets. A revision to an existing dataset with an new release date would be a new dataset.
    – Andrew - OpenGeoCode
    Commented Dec 26, 2013 at 18:37
  • Wow ... okay, I'll just keep citing the Renear, Sacchi & Wicket paper (Definitions of Dataset in the Scientific and Technical Literature), in which they effective came to the conclusion that there are some similarities, but every field has their own definition. (I think your definition is more similar to earth science, which makes sense if you deal with geocoding)
    – Joe
    Commented Dec 26, 2013 at 18:58
8

The Guardian data website has a variety and insights in to UK, European and World data. The site also includes data visualisationa and application you can use. Very enjoyable. http://www.guardian.co.uk/data

Improve this answer
8

databib.org

re3data.org

recently merged: Registry of Research Data Repositories i.e. RE3

"The consolidated registry contains information for more than 1,130 data repositories that are accessed by over 5,000 unique visitors each month. On average, 10 new repositories are added every week."

"A new REST-API is currently being beta tested that provides the full list of repositories and enables retrieval of full, structured records describing individual repositories.' 09 MARCH 2015

One repository may contain, for instance, 100,000 data descriptions. Research Data Australia contains 100k data descriptions, but many of these are mediated access i.e. contact the researcher.

Improve this answer
8

There are some great resources here. I would like to add http://knoema.com to this list.

They are possibly the most comprehensive and constantly updating resource. All their data is free to use and links back to source. Plus the data exploration tools and search engine is pretty useful.

Their dataset explorer is here: http://knoema.com/data

I found that they have some open data portals as well like http://opendataforafrica.org

Also found that they have good presence over chrome webstore: https://chrome.google.com/webstore/search/knoema

PS: I know some guys who work here and I think it is a good addition to list of resources here.

Update: The Data Browser is now replaced by a Data Atlas which is more intuitive when looking for data. http://knoema.com/atlas. It is also easy to explore and share visualizations from dataset pages like here I compared China & USA GDP on line chart http://knoema.com/IMFWEO2015Oct/imf-world-economic-outlook-weo-october-2015?accesskey=rlauzhf

Improve this answer
7

Another system of interest for your list would be the OKFN Global Open Data Index:

https://index.okfn.org/place/

It provides national and sub-national catalog listings and categorization for a variety of data types, and an assessment of how open each country is. Similar to DataPortals.org, and run by the same people.

Improve this answer
5

Please refer the link https://www.udacity.com/wiki/ml/resources

In the above given link there is a link which points to many sites where you can pull the data from open database.

Improve this answer
5

Also, this Quora thread lists quite a few others.

Improve this answer
4

Have you looked into Linked Open Data (http://vimeo.com/36752317)? http://lod-cloud.net/ has a interactive diagram of open data sets (as at September 2011).

Improve this answer
2
  • Unfortunately, clicking through an image map isn't quite a usable database of databases.
    – David J.
    Commented May 24, 2013 at 21:02
  • 1
    The data behind the LOD Cloud image is contained in The Data Hub, mentioned by Patrick
    – D Read
    Commented May 29, 2013 at 20:08
4

The Registry of Research Data Repositories just issued a press release announcing their existance, but their scope isn't necessarily all data, only 'research data'.

I assume they also contain closed data, based on one of their metadata fields:

Type of access to data: open

Improve this answer
4
Improve this answer
4

I've compiled an index/catalog of government open data portals around the world. So far, there are over 700 sites in the catalog. I've setup the catalog for crowdsourcing, so feel free to send us your suggested portals to add to the catalog.

http://www.opengeocode.org/opendata/

You can download the index as a CSV file as well:

http://www.opengeocode.org/opendata/opendata.csv

Improve this answer
2
  • 1
    Is there a reason this is seperate from your other answer to this question? Also, please stop constantly updating the number -- you're triggering moderator flags that the answer has been edited too many times.
    – Joe
    Commented Jun 27, 2014 at 15:29
  • @Joe - thanks for letting me know how the edit updates work.
    – Andrew - OpenGeoCode
    Commented Jun 27, 2014 at 15:39
3

The only thing that comes to mind is programmableweb... I guess you could also include this site although its still in closed beta.

Improve this answer
2
  • 1
    ProgrammableWeb is a collection of Web APIs, but to my knowledge it's not possible to only list open APIs. The second link might be a little self-referential ;)
    – Patrick Hoefler
    Commented May 12, 2013 at 13:27
  • @PatrickHoefler good point.
    – John
    Commented May 12, 2013 at 13:31
3

Google offers a couple more structured ways to search just on data sets beyond the main Google search function mentioned above:

1) A Googler on Twitter has suggested "try[ing] Google custom search (@googlecse) on pages that mention a http://schema.org/Dataset - http://datasets.schema-labs.appspot.com/" This search tool allows you to search for data sets that have been tagged using the Schema.org tag for data sets.

2) Google Fusion tables allows the public to search for tables that are either uploaded into Fusion tables or tables that Google has found on the web that could be uploaded into Fusion tables. In their words, "Web pages sometimes display high-quality structured data in a table. Many of these tables appear in Google Tables search results, dramatically expanding your ability to locate structured data. Once you find a good table, you may decide to import it to Fusion Tables."

Improve this answer
3

The "Datasets" and/or "OpenData" sub-reddits on reddit.com comprise something like that. for example.

Improve this answer
3

I didn't see these mentioned.

  1. Rockstar Data Sources on Github.
  2. List of free and open journals. Mostly scientific journals.
Improve this answer
3

Enigma Public is a repository for data that host all the data in its raw form on its own servers which allows you to search within the data as well download it through a REST API. All data is available under creative commons licensing.

Improve this answer
2
  • 2
    Enigma is a paid service as I can see. Am I right? If yes, it would be wise to update your answer and include this info.
    – Tasos
    Commented Jan 8, 2014 at 16:41
  • Hey Tasos, the site has changed its model somewhat in the past few years, but it is actually completely free for non-commercial use.
    – eveahe
    Commented Sep 25, 2017 at 16:00
3

Google Dataset Search

https://toolbox.google.com/datasetsearch

A recent tool in beta.

Improve this answer
2

The Open Knowledge Foundation has a registry of packaged datasets that are easier to use - Core Datasets as Data Packages.

The data is hosted on https://github.com/datasets and the registry can be found in core_list.txt.

Important, commonly-used datasets in high quality, easy-to-use & open form

  • High Quality & Reliable - Sourcing, normalizing and quality checking a set of key reference and indicator datasets such as country codes, currencies, GDP and population.

  • Standard Form & Bulk Access - All datasets provided in a standardized form and can be accessed in bulk as CSV together with a simple JSON schema.

  • Versioned & Packaged - All data is in data packages and is versioned using git so all changes are visible and data can be collaboratively maintained.

Improve this answer
1

Google Custom Search for Data

Google Custom Search across 200+ data publishers and data repositories:

It's compatible with Google's search operators. You can

  • exclude sources with -site:domain.com
  • exclude keywords with -keyword
  • find by extension ext:csv

(Disclaimer: I maintain this CSE. Your suggestions for new data sources are welcomed.)

Improve this answer
1

Check out data.world - we're a social network for data people, and are building the world’s most collaborative, abundant, and meaningful data resource. If you don't see what you're looking for, you can create a dataset page and add a 'contributors-wanted' tag to let other users know you're looking for that data and would like to collaborate. Hope this helps!

Improve this answer
1

Add to the list the Github repo Awesomedata/awesome-public-datasets

(and also check out the Complementary Collections section, which links other lists/resources.)

At the time of this answer, the categories are:

  • Agriculture
  • Biology
  • Climate+Weather
  • ComplexNetworks
  • ComputerNetworks
  • DataChallenges
  • EarthScience
  • Economics
  • Education
  • Energy
  • Finance
  • GIS
  • Government
  • Healthcare
  • ImageProcessing
  • MachineLearning
  • Museums
  • NaturalLanguage
  • Neuroscience
  • Physics
  • ProstateCancer
  • Psychology+Cognition
  • PublicDomains
  • SearchEngines
  • SocialNetworks
  • SocialSciences
  • Software
  • Sports
  • TimeSeries
  • Transportation
  • eSports
Improve this answer
0

You can check out World Bank Open Data portal for open datasets on global economic development.

Improve this answer
0

It depends on what you are looking, but for shared research dataset, zenodo is becoming more and more popular. The data are not all open, but you can easily filter for open data, which gives you 50000+ results.

https://zenodo.org/search?page=1&size=20&q=dataset&access_right=open

Improve this answer