37

This is a very simple question:

Suppose that I have some sort of specialized data, perhaps that I've collected myself or been a part of the collection. And suppose that nothing prevents me from handing this data out to people. In what method should I go about distributing/storing this data so that others will be able to find it and use it, whenever this time may be?

For example, as an undergrad I used to sell out my R programming capabilities to all the soft-science research groups. Sometimes, they would have really interesting data that they had collected. One that sticks out to me now: a group had spent 6 months traveling Central America, keeping records on aspects of water quality and personal hygiene, etc. I did their statistical analysis, they wrote their paper and project, and then they thought nothing more of their data. They ended up letting me keep it and distribute it however I wanted.

I didn't know what to do.

But as a fluke, 15 months later (again doing some R for a group), it turns out that I ran across someone who was interested and gave them a copy. And so it was passed on.

How might one go about best making others aware of open data availability?

22

Suppose that I have some sort of specialized data, perhaps that I've collected myself or been a part of the collection. And suppose that nothing prevents me from handing this data out to people. In what method should I go about distributing/storing this data so that others will be able to find it and use it, whenever this time may be?

Targeting specialised repositories as per @Joe's answer is indeed an excellent way to go about disseminating data, but what if no such specialised repository exists or you do not wish to target only one specific community in particular?

A methodology to expose Open Data using generic principles is the 5-star Open Data scheme originally proposed by Tim Berners-Lee here.

The core rationale of 5-star Open Data is that you make your data more easily accessible, processable and interoperable with each successive star:

Put your data on the Web in some format with an Open Licence. People can access it through their browsers and spend some time to figure out how they can download/access/process/use it. (Avoid problems for your client like this.)

★★ Put your data in a machine-processable format. For example, having a table in Excel is better than having a snapshot printed in PDFs or images because people can download it and start running experiments over it. (Avoid problems like this.)

★★★ Use non-proprietary formats. For example, providing data as a CSV is often better than as an Excel file because CSV can be directly processed by a wider range of (free/open source) tools and programming languages. (Can't find anyone complaining about Excel on here yet but, e.g., this is a similar problem.)

★★★★ Use URIs to denote things. For example, let's say you provide a bunch of pollution measures for cities and somebody would like to specifically reference the pollution measure for London. Assigning a URI for London in your local data provides a global unique identifier for that city that people can reference and point to. There are, for example, related proposals for embedding URI fragment identifiers in CSV files. (Avoid problems like this or this.)

★★★★★ Link your data to other data to provide context. So you have created a URI for London in your data and people can point to it. However, which London are you referring to? London, England or London, Ontario? If you link your local URI for London to the Wikipedia page about the London to which you refer (or, even better, to the DBpedia URI for the specific place to which you prefer), this provides context as to what you mean. (Avoid problems like this.)

The shift from ★★★ to ★★★★(★) is quite an ambitious one and technical proposals are still being made on how best to achieve this, but five star Open Data is great because now your data are available on the Web under open licences with open structured formats where everything of importance is given a URI that can be referenced and linked across the Web, allowing for future discovery and re-use. A common methodology to create five star Open Data (again proposed by Tim Berners-Lee) is Linked Data, which assumes RDF as a common interoperable data format. But if that all sounds too much, getting as far as ★★★ data is still great.

Again, you can check out this description of 5 Star Open Data for more information and a related question here.

A useful resource for the generic cataloguing of Open Datasets is the CKAN project, where the related DataHub repository is a great place to list and publicise your dataset. You can check out a bunch of 5-star Open Datasets here.

  • How much wider an audience does each additional star gain you? Would I be better off paying for an additional star for my anonymously hosted data or just getting it uploaded to Dryad or Figshare (or any of the places suggested by Joe) where people would actually find it and use it? – Tom Morris Aug 8 '13 at 3:48
  • The stars are really about making your data as generically discoverable on the Web as possible. If the domain-specific upload sites suggested by Joe are sufficiently well-known for people to know where to look, this would be a good option, but still consumers need to manually find, download and potentially convert the data. The five stars are about letting machines do that for you and without requiring a centralised domain-relevant upload site. – badroit Aug 9 '13 at 21:05
13

I'd recommend looking to see how that specific scientific discipline handles their data.

Some common methods include:

  • Discipline Registry -- you continue hosting the data elsewhere, but notify the service of the availability of the data, and characteristics so that other people can tell if it's of value to them. (eg, Anthropological, Heliophysics)
  • Discipline Repository -- like the registry, but you give them the data to share with others. (eg, the PDS nodes, Dryad)
  • Discipline Database -- a type of repository where they have a proscribed format that you need to put the data into before submitting it to them. (eg, PetDB)
  • Federation -- you continue hosting the code, and register it with the central authority, but you have to expose the data using a proscribed API. (eg, IVOA)

Note that these are general categories ... I didn't give the Protein Data Bank as an example as it's sort of between a repository and a database; data.gov can act as a federation in some cases but is mostly a repository and it's not for a specific discipline.

... and in other cases, there's nothing formal. For instance, in some 'big science' fields, where there's only a few new datasets released per year, I've seen the common practice be to put a little blurb in the relevant professional society's newsletter, or post to a mailing list.

For your data, I'd suggest starting with the Environmental Information Exchange Network. They deal with water quality, but I don't know what their submittal processes are. They're a federation, but you might be able to talk to one of the local member nodes, and have them take the data.

It's also worth noting that some of the disciplines have specific metadata standards -- if you can't describe the data sufficiently for their needs, they're just not interested. This seems strange, as you'd assume that all data is valuable, but if it doesn't have sufficient documentation for that group to cite it as supporting evidence it's of no use to them. (although, that's not to say that there won't be other groups interested in the data with less vigorous standards)

  • There's also the supposedly haphazard way of self-hosting the set. – Deer Hunter Jun 13 '13 at 14:42
  • 1
    Agreed with Joe, but also remember to include a license that you or your organization can feel comfortable with. It doesn't matter where or how you publish the data, if it is unlicensed, the default all rights reserved will restrict other people to use it. – Alvaro Graves Jun 13 '13 at 15:30
  • @DeerHunter : that'd fall into the 'registry' or 'federation' models ... but just putting it up on a website without letting the appropriate community would fail the 'making others aware' bit of the question. – Joe Jun 13 '13 at 15:42
  • Most registries or repositories should ask about the rights for their metadata. Some registries will accept closed data, most discipline repositories won't (unless there are privacy issues in the whole field, such as medical or endangered animals). Federations vary -- the one I'm affiliated with (Virtual Solar Observatory) doesn't even want restricted data in the network. ... and I'd also question the default being 'all rights reserved', as 'data' doesn't necessarily get the same protection as creative works in many jurisdictions. – Joe Jun 13 '13 at 15:52
  • That's right, should, but I don't know if all of them onforce that. IANAL, but as far as I know every work that do not specify a license has all its rights reserved for most of the countries, independent if it is a picture, a document or a dataset. – Alvaro Graves Jun 13 '13 at 15:55
5

UPDATE:

datahub.io is:

a free, powerful data management platform from the Open Knowledge Foundation, based on the CKAN data management system.

Datasets can be added to as part of the organization OpenData StackExchange.

Additionally, participation in the "organization" as a follower, member or admin is encouraged!

Discussion on meta site


It was mentioned in the comments of another answer, but I'll expand here.

GitHub is a great solution for hosting datasets (it's not just for code!). Here is a OpenData discussion regarding GitHub and datasets: link.

Some benefits:

  • Version control and forking (for when people want to contribute data or files)
  • Accessible without registration
  • Not domain specific (i.e. only found by biologists)
  • Allows any and all formats (and multiple formats for same data)
  • Can host data AND sample (R) code for using the data.

Here is a blog post from OKFN with some tips.

Some options:

  • Post as your own, independent repository.
  • Post your own, but list in the OKFN registry and text list.
  • You can also post your data as part of their collection (link).
  • following...but i don't see where to become a member....i'm looking around but if you figure it out, please share – albert Apr 10 '15 at 11:32
  • 1
    @albert - I made a meta post here that requests the username of people who want to be members/admins – philshem Apr 10 '15 at 11:51
3

Academic Torrents is a platform to share research releated dataset and paper through a peer to peer network. Academci Torrents host a list of tracer so other can search for the torrent.

Data are stored on your local machine / own server and get distributer as other download and start to seed it back.

2

WikiStats? I keep hoping that I will stumble across a page like wikipedia, but full of public datasets - a place where I could upload the open/public data which I find most useful... btw, I live in Central America and I would be very interested to check out 6 months of water hygiene data...

  • or also a github site dedicated to such things could be appealing... – Matt Lemmon Jun 20 '13 at 21:34
  • wow there are tons of open-data-related githubs... I will be browsing through them in the upcoming days. – Matt Lemmon Jun 20 '13 at 21:42
  • I have created a basic open dataset library [on github](github.com/MattLemmon/OPEN-DATA). It is ready to be tested out if anyone has data they feel like sharing. – Matt Lemmon Jun 21 '13 at 0:01
  • meant to say on github – Matt Lemmon Jun 21 '13 at 0:25
  • I found this discussion about what it would take to set up a public data git repository. – Matt Lemmon Jun 22 '13 at 20:11
1

In this particular case I would also contact the UN Entity that operates in the area the survey was conducted - my sense is that it would be welcome. UNICEF does a lot of work in the WATSAN area (water and sanitation) and the UN coordination body for Central America is the Regional Office for Latin America and the Caribbean (ROLAC).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.