I am interested in how typical open data users, such as journalists, researchers, companies, developers, and others, find out about new open data sets today. For example, how do sources like search engines, re-distributors of open data, tech media, and government open data platforms compare? Is there any research into this question? (This question seems particularly relevant to this site, since many people have posted questions about how to find particular data sets.)
How to stay up to date with UK government data releases:
- Office for National Statistics release calendar
- Parliamentary releases mailing list
- Planning alerts mailing list
- Press releases
- RSS feeds
The Office for National Statistics release calendar is excellent because it allows you to see weeks in advance what data is going to be published. http://www.statistics.gov.uk/hub/release-calendar/index.html
Parliamentary releases mailing list lets you select exactly which committees and types of reports you want to be alerted to, and if you want them immediately or daily. https://subscriptions.parliament.uk/accounts/UKPARLIAMENT/subscriber/new?
Planning data has a great email alert, excellent for local data journalism. http://www.planningportal.gov.uk/inyourarea/
For datasets where a single authoritative source exists (for instance, a government office often fulfills this role for data concerning its jurisdiction), then it's helpful to find an RSS feed or revisions page that lists additions/updates to datasets.
I've also found that Google Alerts are helpful for this task, even beyond open data. Once you've found an existing dataset using a particular query, the Alert will email you (based on your settings) whenever another similar piece of content is posted.
Personally, I often start with an open question, and than look at potential data providers (Statistical Office, etc) or using google/yahoo with the filetype:xls keyword.
DuckDuckGo Search for "data filetype:xls"
Google Search for "data filetype:xls"
Very helpful and great search method. Adding to it, swap out "filetype:xls" for "filetype:json", "filetype:csv", "filetype:xlsx", etc.
r/datasets Reddit sub is pretty helpful/has some rather interesting sets.
Data Elixir is a weekly newsletter of curated data science news and resources from around the web. Free for data lovers.
Kaggle's Datasets are pretty extensive, and supported by a vibrant data science community.
Awesome Public Datasets is a maintained list of open data.
KDnuggets Datasets has datasets for data mining and data science.
ArcGIS Open Datasets has a lot of data; there is no feed for it all (that I can see), but typically you can subscribe to users/groups/terms RSS feeds in ArcGIS too.
BuzzFeedNews' everything Repository is an index of all of BuzzFeedNews' open-source data, analysis, libraries, tools, and guides.
Browse Studies Database provides a tool for diving into the database of studies found on the Journalist's Resource, a part of Harvard's Shorenstein Center on Media, Politics, and Public Policy
'Finding' data is a really hot topic. (Would love to up-vote but am too new).
It is something I am asked a lot about at work. We call the process 'Data Landscaping'.
In terms of sites http://www.programmableweb.com/ can be useful - but often comes a bit light. We would then start with data marketplaces and data providers (http://www.quandl.com/ for example). And of course, search engines and following the right accounts on twitter. Building Data Curation Experience plays a huge part.
We often have to start with exploring what a client as already. I use a 'magic quadrant' with axis of 'Distant and Close data' and 'Dark and Light' data. Distant data 'you know is out there, but don't have in your organisation'. Close data 'you have access to directly'. Dark data 'you don't really know what to do with' Light data 'you know how to use to benefit your organisation'.
I then map against 'The Data Journey' (I am producing a white paper on this - message me if you would like a copy). Which shows how data moves and can be put to work.
The combination of these two assets help show gaps in the landscape and raise questions to focus where to look next. I have not, as yet, seen any research into how people are finding new data.
Great question!! :)
I'm starting to think that nobody really has a systematic way of finding datasets. With that in mind, I recently made OpenPrism so I could search 100 data portals at once.
In research, one common way is to simply read other research articles and see what datasets they have used. Also, when a dataset is created by a research lab, the latter usually advertises to their colleagues (research meeting, email, conference, etc.) and might publish a paper presenting the new dataset (1).
(1) E.g. Saeed, Mohammed, et al. "MIMIC II: a massive temporal ICU patient database to support research in intelligent patient monitoring." Computers in Cardiology, 2002. IEEE, 2002.
Data.world has some interesting open datasets, and allows you to look at recent additions.
As an addendum, even though it may not sound like the most obvious source, NICAR (National Institute for Computer-Assisted Reporting) maintains a very active listserv of data journalists corresponding with one another and sharing new data as it's released in real-time nearly. If something's happening in current events, a data journalist somewhere is looking for and sharing data on the topic. You can sign up for the ListServ on NICAR's website.
re3data includes many research data repositories, however it also has a number of non-academic sources like the World Bank, UN.