I am interested in how typical open data users, such as journalists, researchers, companies, developers, and others, find out about new open data sets today. For example, how do sources like search engines, re-distributors of open data, tech media, and government open data platforms compare? Is there any research into this question? (This question seems particularly relevant to this site, since many people have posted questions about how to find particular data sets.)
-
It's a combination; I believe no individual answer covers it yet. New releases generally occur according to a schedule, so following social media, e.g. a RSS feed, is popular. For journalists, see our slide 32 here.– UlrichFeb 3, 2014 at 15:01
-
They don't look for open datasets as such. They are interested in certain areas, follow what happens there, and then bump into announcements, links, tweets, ...– user4293Mar 8, 2017 at 9:36
Add a comment
|
12 Answers
How to stay up to date with UK government data releases:
- Office for National Statistics release calendar
- Parliamentary releases mailing list
- Planning alerts mailing list
- Press releases
- RSS feeds
For example:
The Office for National Statistics release calendar is excellent because it allows you to see weeks in advance what data is going to be published. http://www.statistics.gov.uk/hub/release-calendar/index.html
Parliamentary releases mailing list lets you select exactly which committees and types of reports you want to be alerted to, and if you want them immediately or daily. https://subscriptions.parliament.uk/accounts/UKPARLIAMENT/subscriber/new?
Planning data has a great email alert, excellent for local data journalism. http://www.planningportal.gov.uk/inyourarea/
For datasets where a single authoritative source exists (for instance, a government office often fulfills this role for data concerning its jurisdiction), then it's helpful to find an RSS feed or revisions page that lists additions/updates to datasets.
I've also found that Google Alerts are helpful for this task, even beyond open data. Once you've found an existing dataset using a particular query, the Alert will email you (based on your settings) whenever another similar piece of content is posted.
Personally, I often start with an open question, and than look at potential data providers (Statistical Office, etc) or using google/yahoo with the filetype:xls keyword.
DuckDuckGo Search for "data filetype:xls"
Google Search for "data filetype:xls"
Very helpful and great search method. Adding to it, swap out "filetype:xls" for "filetype:json", "filetype:csv", "filetype:xlsx", etc.
r/datasets Reddit sub is pretty helpful/has some rather interesting sets.
Data Elixir is a weekly newsletter of curated data science news and resources from around the web. Free for data lovers.
Data Is Plural - A weekly newsletter of useful/colorful datasets
Kaggle's Datasets are pretty extensive, and supported by a vibrant data science community.
Awesome Public Datasets is a maintained list of open data.
KDnuggets Datasets has datasets for data mining and data science.
ArcGIS Open Datasets has a lot of data; there is no feed for it all (that I can see), but typically you can subscribe to users/groups/terms RSS feeds in ArcGIS too.
US National Park Service (NPS) Data Repository
datahub.io has many datasets by a lot of organizations; instances of both have RSS feeds you can subscribe to.
Open Data Stack Exchange's datahub.io instance
Data Packaged Core Datasets via OKFN's Data Team on GitHub
BuzzFeedNews' everything Repository is an index of all of BuzzFeedNews' open-source data, analysis, libraries, tools, and guides.
Browse Studies Database provides a tool for diving into the database of studies found on the Journalist's Resource, a part of Harvard's Shorenstein Center on Media, Politics, and Public Policy
'Finding' data is a really hot topic. (Would love to up-vote but am too new).
It is something I am asked a lot about at work. We call the process 'Data Landscaping'.
In terms of sites http://www.programmableweb.com/ can be useful - but often comes a bit light. We would then start with data marketplaces and data providers (http://www.quandl.com/ for example). And of course, search engines and following the right accounts on twitter. Building Data Curation Experience plays a huge part.
We often have to start with exploring what a client as already. I use a 'magic quadrant' with axis of 'Distant and Close data' and 'Dark and Light' data. Distant data 'you know is out there, but don't have in your organisation'. Close data 'you have access to directly'. Dark data 'you don't really know what to do with' Light data 'you know how to use to benefit your organisation'.
I then map against 'The Data Journey' (I am producing a white paper on this - message me if you would like a copy). Which shows how data moves and can be put to work.
The combination of these two assets help show gaps in the landscape and raise questions to focus where to look next. I have not, as yet, seen any research into how people are finding new data.
Great question!! :)
I'm starting to think that nobody really has a systematic way of finding datasets. With that in mind, I recently made OpenPrism so I could search 100 data portals at once.
-
Your site links to datacatalogs.org ; which claims to have 337 data catalogs ... is there a reason you only have 100? (is it just an issue of 'portals' having more than one 'catalog'?)– JoeAug 16, 2013 at 15:47
-
Not all of the catalogs have the same search APIs. I've only implemented the search call for CKAN, Junar, and Socrata, and they account for a bit over 100 portals. Aug 16, 2013 at 17:12
I've also found DataBullet.in and Enigma Public Data Explorer to be helpful.
Also, a recent social network called data.world has tons of interesting stuff.
-
1
In research, one common way is to simply read other research articles and see what datasets they have used. Also, when a dataset is created by a research lab, the latter usually advertises to their colleagues (research meeting, email, conference, etc.) and might publish a paper presenting the new dataset (1).
(1) E.g. Saeed, Mohammed, et al. "MIMIC II: a massive temporal ICU patient database to support research in intelligent patient monitoring." Computers in Cardiology, 2002. IEEE, 2002.
answered Feb 1, 2014 at 17:08
Data.world has some interesting open datasets, and allows you to look at recent additions.
-
-
My apologies. Is it considered good etiquette to delete this comment? Or should I edit it? I'm relatively new here– wirefireFeb 15, 2017 at 21:22
-
2No worries. I am relatively new to the site as well and do not know if there is a "correct" way to do it. Maybe others can comment on this. Then again, if it was my answer I would either delete it or update with additional information not mentioned yet. Feb 15, 2017 at 21:28
As an addendum, even though it may not sound like the most obvious source, NICAR (National Institute for Computer-Assisted Reporting) maintains a very active listserv of data journalists corresponding with one another and sharing new data as it's released in real-time nearly. If something's happening in current events, a data journalist somewhere is looking for and sharing data on the topic. You can sign up for the ListServ on NICAR's website.
-
joined this a few weeks ago and it is by far on the top of my list now. just for datasets...techniques, ideas, etc. everyone should lurk.– albertFeb 24, 2017 at 6:26
Jeremy Singer-Vine, BuzzFeed's data editor, runs a pretty good mailing list of "interesting" data sets: Data Is Plural. It's not an exhaustive collection, but it covers a variety of topics.
re3data includes many research data repositories, however it also has a number of non-academic sources like the World Bank, UN.