While there are many open databases available, is there a database or the project that would contain the information of such databases?
In other words, is there an open meta-database of open databases?
DataPortals.org (previously DataCatalogs.org) provides a comprehensive list of open data portals from around the world. Their (meta-)data is in the public domain and available for download as CSV and JSON.
Data that is somehow related is usually grouped in datasets or databases, contained in files (e.g. CSV or spreadsheets) or some kind of database management system, which might be accessible via an API.
In the context of Open Data, data portals, data catalogs, or data hubs make it easier to find these datasets or databases.
This is where DataPortal.org comes in: It is a data portal of data portals.
To sum it up:
DataPortals.org --> Specific Data Portal --> Specific Data Set --> Open Data
There's also http://datacatalogs.org/ which maintains a list of open data portals.
You can get a full machine readable list (JSON) from there via the API: http://datacatalogs.org/api/search/dataset
That's only the first 10, to get 100 do: http://datacatalogs.org/api/search/dataset?limit=100
And with full info: http://datacatalogs.org/api/search/dataset?limit=100&all_fields=1
It not only pulls them into one place for easier access, but provides an API for each dataset and packages for them to be pulled into the analysis tool of choice.
As of May 2016, you'll find a list of 2500+ open data portals around the world on OpenDataSoft's website. It lists as of now 2500+ portals ranging from the US (obviously) to Afghanistan.
The story behind it:
Working for a SaaS company in need of loads of structured data, our team started to compile a list of all open data portals around the world as a go-to resource.
We ended up with a list of more than 1600 portals. We gathered our own listings, scrapped third-party datasets (the likes of dataportals.org), cleaned the whole thing (lot of elbow grease + Clojure) and created a list (w/ Ruby).
Instead of keeping it in a dusty corner of our computers, we thought we'll share it with the open data community / data geeks.
Major work was done on cleaning the data (many duplicates), adding geo-coordinates to every portals, crossing off the list all dead portals/urls...
First, we used the dataset generated to create a website called opendatainception.io where you can now browse data on a map. You can browse data by navigating or through the search box. When typing a query there, the data will automatically refine on the map.
Second, we shared the list as a go-to resource for everyone. At first we were building the list as an HTML list from the dataset with some Ruby script. It was a kinda pain and not always super reliable. To be more efficient and reflect the changes instantly as we were making them, we went for some open source widgets instead (built w/ angular).
Now, the page displays a dynamic list, always synced up with the dataset. You still can look for countries and stuff.
You can find the list here and we explained the how-to there.
Hope that'll help!
If you have any feedback or want to add a portal, give us a shout!
Actually, with the data portals everyone is talking about, people forgot about Google. We Google for most things in life, why not data?
Does no-one remember the early days of the web, when we had portals that neatly catalogued web pages. It made sense to have people manually curating lists of pages, carefully categorizing them. Portals were cool. Much better than dumb search engines. But we all know which won.
Actually Google right now doesn't work too badly if you know the name of the dataset you're looking for. "But a search engine is no good for browsing through the topics, or providing useful search faceting, or all those other features data.gov.uk has." I hear you say.
Well y'know they have a big new idea recently that might enable all those features. They want everyone publishing a dataset to publish a metadata record alongside it on their server. (The format is called 'schema.org'.) Then when googlebot comes to spider the site, it finds the metadata and can then do all the topic browsing, faceting etc. that the average data portal does. But Google has the added advantage that it indexes all the data that doesn't have metadata too, and you can bet they can make a good attempt at guessing categories for those datasets too.
So in a couple of years Google (or Yahoo or Bing) might just run the finding data space. But not make the tools or metadata open. Tell me I'm wrong...
Our open-date site has links to about 600 useful datasets from US and Canadian government and a handful from the UN. The US federal government datasets are sorted by department.
We [opengeocode.org] maintain a catalog of open data portals across the world. We've categorized them under the categories:
recently merged: Registry of Research Data Repositories i.e. RE3
"The consolidated registry contains information for more than 1,130 data repositories that are accessed by over 5,000 unique visitors each month. On average, 10 new repositories are added every week."
"A new REST-API is currently being beta tested that provides the full list of repositories and enables retrieval of full, structured records describing individual repositories.' 09 MARCH 2015
One repository may contain, for instance, 100,000 data descriptions. Research Data Australia contains 100k data descriptions, but many of these are mediated access i.e. contact the researcher.
There are some great resources here. I would like to add http://knoema.com to this list.
They are possibly the most comprehensive and constantly updating resource. All their data is free to use and links back to source. Plus the data exploration tools and search engine is pretty useful.
Their dataset explorer is here: http://knoema.com/data
I found that they have some open data portals as well like http://opendataforafrica.org
Also found that they have good presence over chrome webstore: https://chrome.google.com/webstore/search/knoema
PS: I know some guys who work here and I think it is a good addition to list of resources here.
Update: The Data Browser is now replaced by a Data Atlas which is more intuitive when looking for data. http://knoema.com/atlas. It is also easy to explore and share visualizations from dataset pages like here I compared China & USA GDP on line chart http://knoema.com/IMFWEO2015Oct/imf-world-economic-outlook-weo-october-2015?accesskey=rlauzhf
Another system of interest for your list would be the OKFN Global Open Data Index:
It provides national and sub-national catalog listings and categorization for a variety of data types, and an assessment of how open each country is. Similar to DataPortals.org, and run by the same people.
Also, this Quora thread lists quite a few others.
I've compiled an index/catalog of government open data portals around the world. So far, there are over 700 sites in the catalog. I've setup the catalog for crowdsourcing, so feel free to send us your suggested portals to add to the catalog.
You can download the index as a CSV file as well:
Google offers a couple more structured ways to search just on data sets beyond the main Google search function mentioned above:
1) A Googler on Twitter has suggested "try[ing] Google custom search (@googlecse) on pages that mention a http://schema.org/Dataset - http://datasets.schema-labs.appspot.com/" This search tool allows you to search for data sets that have been tagged using the Schema.org tag for data sets.
2) Google Fusion tables allows the public to search for tables that are either uploaded into Fusion tables or tables that Google has found on the web that could be uploaded into Fusion tables. In their words, "Web pages sometimes display high-quality structured data in a table. Many of these tables appear in Google Tables search results, dramatically expanding your ability to locate structured data. Once you find a good table, you may decide to import it to Fusion Tables."
The Open Knowledge Foundation has a registry of packaged datasets that are easier to use - Core Datasets as Data Packages.
Important, commonly-used datasets in high quality, easy-to-use & open form
High Quality & Reliable - Sourcing, normalizing and quality checking a set of key reference and indicator datasets such as country codes, currencies, GDP and population.
Standard Form & Bulk Access - All datasets provided in a standardized form and can be accessed in bulk as CSV together with a simple JSON schema.
Versioned & Packaged - All data is in data packages and is versioned using git so all changes are visible and data can be collaboratively maintained.
Google Custom Search across 200+ data publishers and data repositories:
It's compatible with Google's search operators. You can
(Disclaimer: I maintain this CSE. Your suggestions for new data sources are welcomed.)
Check out data.world - we're a social network for data people, and are building the world’s most collaborative, abundant, and meaningful data resource. If you don't see what you're looking for, you can create a dataset page and add a 'contributors-wanted' tag to let other users know you're looking for that data and would like to collaborate. Hope this helps!