27

I am interested in how typical open data users, such as journalists, researchers, companies, developers, and others, find out about new open data sets today. For example, how do sources like search engines, re-distributors of open data, tech media, and government open data platforms compare? Is there any research into this question? (This question seems particularly relevant to this site, since many people have posted questions about how to find particular data sets.)

  • It's a combination; I believe no individual answer covers it yet. New releases generally occur according to a schedule, so following social media, e.g. a RSS feed, is popular. For journalists, see our slide 32 here. – Ulrich Feb 3 '14 at 15:01
  • They don't look for open datasets as such. They are interested in certain areas, follow what happens there, and then bump into announcements, links, tweets, ... – Jan Doggen Mar 8 '17 at 9:36

12 Answers 12

7

How to stay up to date with UK government data releases:

  • Office for National Statistics release calendar
  • Parliamentary releases mailing list
  • Planning alerts mailing list
  • Press releases
  • RSS feeds
  • Twitter

For example:

  1. The Office for National Statistics release calendar is excellent because it allows you to see weeks in advance what data is going to be published. http://www.statistics.gov.uk/hub/release-calendar/index.html

  2. Parliamentary releases mailing list lets you select exactly which committees and types of reports you want to be alerted to, and if you want them immediately or daily. https://subscriptions.parliament.uk/accounts/UKPARLIAMENT/subscriber/new?

  3. Planning data has a great email alert, excellent for local data journalism. http://www.planningportal.gov.uk/inyourarea/

6

For datasets where a single authoritative source exists (for instance, a government office often fulfills this role for data concerning its jurisdiction), then it's helpful to find an RSS feed or revisions page that lists additions/updates to datasets.

I've also found that Google Alerts are helpful for this task, even beyond open data. Once you've found an existing dataset using a particular query, the Alert will email you (based on your settings) whenever another similar piece of content is posted.

6

Personally, I often start with an open question, and than look at potential data providers (Statistical Office, etc) or using google/yahoo with the filetype:xls keyword.
DuckDuckGo Search for "data filetype:xls"
Google Search for "data filetype:xls"

Very helpful and great search method. Adding to it, swap out "filetype:xls" for "filetype:json", "filetype:csv", "filetype:xlsx", etc.

6

r/datasets Reddit sub is pretty helpful/has some rather interesting sets.

Data Elixir is a weekly newsletter of curated data science news and resources from around the web. Free for data lovers.

Data Is Plural - A weekly newsletter of useful/colorful datasets

Kaggle's Datasets are pretty extensive, and supported by a vibrant data science community.

Awesome Public Datasets is a maintained list of open data.

KDnuggets Datasets has datasets for data mining and data science.

ArcGIS Open Datasets has a lot of data; there is no feed for it all (that I can see), but typically you can subscribe to users/groups/terms RSS feeds in ArcGIS too.

US National Park Service (NPS) Data Repository

datahub.io has many datasets by a lot of organizations; instances of both have RSS feeds you can subscribe to.

Open Data Stack Exchange's datahub.io instance

Data Packaged Core Datasets via OKFN's Data Team on GitHub

BuzzFeedNews' everything Repository is an index of all of BuzzFeedNews' open-source data, analysis, libraries, tools, and guides.

DataCrunch Podcast

Browse Studies Database provides a tool for diving into the database of studies found on the Journalist's Resource, a part of Harvard's Shorenstein Center on Media, Politics, and Public Policy

5

'Finding' data is a really hot topic. (Would love to up-vote but am too new).

It is something I am asked a lot about at work. We call the process 'Data Landscaping'.

In terms of sites http://www.programmableweb.com/ can be useful - but often comes a bit light. We would then start with data marketplaces and data providers (http://www.quandl.com/ for example). And of course, search engines and following the right accounts on twitter. Building Data Curation Experience plays a huge part.

We often have to start with exploring what a client as already. I use a 'magic quadrant' with axis of 'Distant and Close data' and 'Dark and Light' data. Distant data 'you know is out there, but don't have in your organisation'. Close data 'you have access to directly'. Dark data 'you don't really know what to do with' Light data 'you know how to use to benefit your organisation'.

I then map against 'The Data Journey' (I am producing a white paper on this - message me if you would like a copy). Which shows how data moves and can be put to work.

The combination of these two assets help show gaps in the landscape and raise questions to focus where to look next. I have not, as yet, seen any research into how people are finding new data.

Great question!! :)

5

I'm starting to think that nobody really has a systematic way of finding datasets. With that in mind, I recently made OpenPrism so I could search 100 data portals at once.

  • Your site links to datacatalogs.org ; which claims to have 337 data catalogs ... is there a reason you only have 100? (is it just an issue of 'portals' having more than one 'catalog'?) – Joe Aug 16 '13 at 15:47
  • Not all of the catalogs have the same search APIs. I've only implemented the search call for CKAN, Junar, and Socrata, and they account for a bit over 100 portals. – Thomas Levine Aug 16 '13 at 17:12
5

I've also found DataBullet.in and Enigma Public Data Explorer to be helpful.

Also, a recent social network called data.world has tons of interesting stuff.

  • 1
    its pretty ironic data bulletin has no rss feed – albert Feb 24 '17 at 6:26
3

In research, one common way is to simply read other research articles and see what datasets they have used. Also, when a dataset is created by a research lab, the latter usually advertises to their colleagues (research meeting, email, conference, etc.) and might publish a paper presenting the new dataset (1).

(1) E.g. Saeed, Mohammed, et al. "MIMIC II: a massive temporal ICU patient database to support research in intelligent patient monitoring." Computers in Cardiology, 2002. IEEE, 2002.

3

Data.world has some interesting open datasets, and allows you to look at recent additions.

  • Data.world is already mentioned above by Carl V. Lewis' answer. – eigenvector Feb 15 '17 at 7:36
  • My apologies. Is it considered good etiquette to delete this comment? Or should I edit it? I'm relatively new here – wirefire Feb 15 '17 at 21:22
  • 2
    No worries. I am relatively new to the site as well and do not know if there is a "correct" way to do it. Maybe others can comment on this. Then again, if it was my answer I would either delete it or update with additional information not mentioned yet. – eigenvector Feb 15 '17 at 21:28
3

As an addendum, even though it may not sound like the most obvious source, NICAR (National Institute for Computer-Assisted Reporting) maintains a very active listserv of data journalists corresponding with one another and sharing new data as it's released in real-time nearly. If something's happening in current events, a data journalist somewhere is looking for and sharing data on the topic. You can sign up for the ListServ on NICAR's website.

  • joined this a few weeks ago and it is by far on the top of my list now. just for datasets...techniques, ideas, etc. everyone should lurk. – albert Feb 24 '17 at 6:26
3

Jeremy Singer-Vine, BuzzFeed's data editor, runs a pretty good mailing list of "interesting" data sets: Data Is Plural. It's not an exhaustive collection, but it covers a variety of topics.

3

re3data includes many research data repositories, however it also has a number of non-academic sources like the World Bank, UN.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.