We’re rewarding the question askers & reputations are being recalculated! Read more.
92

The short answer DataPortals.org (previously DataCatalogs.org) provides a comprehensive list of open data portals from around the world. Their (meta-)data is in the public domain and available for download as CSV and JSON. The longer answer Data that is somehow related is usually grouped in datasets or databases, contained in files (e.g. CSV or ...


35

There's also http://datacatalogs.org/ which maintains a list of open data portals. You can get a full machine readable list (JSON) from there via the API: http://datacatalogs.org/api/search/dataset That's only the first 10, to get 100 do: http://datacatalogs.org/api/search/dataset?limit=100 And with full info: http://datacatalogs.org/api/search/dataset?...


25

I have worked for numerous companies in the past who have had a policy of not using any software, libraries, or datasets that would impose requirements on their product, this would include the attribution clause in the CC-BY licence. Placing a dataset in the public domain will maximise its potential audience; whether or not this is more desirable than ...


22

Suppose that I have some sort of specialized data, perhaps that I've collected myself or been a part of the collection. And suppose that nothing prevents me from handing this data out to people. In what method should I go about distributing/storing this data so that others will be able to find it and use it, whenever this time may be? Targeting specialised ...


17

Quandl is an index of datasets. It includes open datasets avaliable for free as well as premium databases which are only available for a cost. It not only pulls them into one place for easier access, but provides an API for each dataset and packages for them to be pulled into the analysis tool of choice.


16

Community wiki to collect data sets available to download/seed on bit torrent DNS Census, the DNS registration dataset snapshot taken in 2013 (compressed ~15GB and uncompressed 157GB). The DNS Census 2013 is an attempt to provide a public dataset of registered domains and DNS records. It was inspired by the Internet Census 2012 which showed that ...


14

As far as I know, the most complete dataset index is IOGDS: International Open Government Dataset Search. It contains info about >1 million datasets. Check it out at http://logd.tw.rpi.edu/node/9903


14

As of February 2019, you'll find a list of 2600+ open data portals around the world on OpenDataSoft's website. It lists as of now 2600+ portals ranging from the US (obviously) to Afghanistan. The story behind it: Working for a SaaS company in need of loads of structured data, our team started to compile a list of all open data portals around the world as ...


13

If you already have a nicely formatted CSV then why not publish that? You can publish in both formats if you really want to do the RDFa too.


13

I'd recommend looking to see how that specific scientific discipline handles their data. Some common methods include: Discipline Registry -- you continue hosting the data elsewhere, but notify the service of the availability of the data, and characteristics so that other people can tell if it's of value to them. (eg, Anthropological, Heliophysics) ...


12

The legal text of CC-BY is quite complex, and has more terms than simple attribution, such as not implying the author endorses your work. Although such terms such as this may be useful, they make the license incompatible with other licenses (including the GPL http://www.gnu.org/licenses/license-list.html#ccby). I'm not sure whether this includes any common ...


12

A major part of the problem has to do with interpretation gray areas and where these give rise to concerns, warranted or not, regarding license compatibility or other issues. For example the question of compatibility between the CC-BY and the GNU GPL is a relatively complicated one and it boils down to "well, how do you read these two licenses?" And ...


11

Actually, with the data portals everyone is talking about, people forgot about Google. We Google for most things in life, why not data? Does no-one remember the early days of the web, when we had portals that neatly catalogued web pages. It made sense to have people manually curating lists of pages, carefully categorizing them. Portals were cool. Much ...


11

You have a few options for real time (or "near real time", which is when you have a delay between the collection & time to serve it, or for those that sample at a lower cadence) There are a lot of considerations when dealing with 'real time' data: Who is the intended audience? (and do they already have standards for serving this type of data?) Is the ...


11

Take a look at what OpenStreetMap does. There's a page describing the nature of tile server disk usage. If you go up to zoom level 18 worldwide, you're talking about 91,625,968,981 tiles, which would take around 54000GB of disk space, but would mostly never be viewed. So I'm not sure if it would ever be a sensible approach, but having said that, I heard ...


10

You can find a list of GIS file formats on Wikipedia. Here is a decent overview of open source GIS servers from the gis.se site, these are the servers people who use open data will most likely be using, so target the formats that those servers use. I would consider some kind of open vector/raster format (I like geoJSON for personal projects because it works ...


10

The other answers so far are all terrific. I'll reiterate one point, and make a new one: The openness of an API is always important, but when complete, quality bulk data is available some of these access issues become a lot more tolerable. An API is not a substitute for bulk data. The federal government has become very API focused, and many of them have ...


10

Example: I only know of the company Cloudmade because they provide (a now outdated, but fine at the time) download portal of ready-to-use shapefiles derived from OSM data. Though this service might have helped its competitors, they probably earned much more in terms of visibility, which might spawn e.g. development contracts for custom-made solutions. This ...


10

Ideally, like @Andrew - OpenGeoCode mentions, you would release it in multiple formats. I would really suggest you look into organizing it into a e-book written in Markdown and hosted on Github. There are several advantages to this such as a built-in change log, being able to let people to (publicly) fork your document(s) and share their changes with the ...


9

Our open-date site has links to about 600 useful datasets from US and Canadian government and a handful from the UN. The US federal government datasets are sorted by department. http://www.opengeocode.org/cude1.1/index.php We [opengeocode.org] maintain a catalog of open data portals across the world. We've categorized them under the categories: Data ...


9

GTFS is what everyone is using. Other vendors, and even open platforms that consume transit data, consume GTFS. Open Trip Planner is an example. (As an aside: was there a particular need that had that you feel GTFS doesn't address? Perhaps we could give you a more focused answer if you clarified.)


9

I think it would be foolish to try and replace the NAICS system. NAICS is the federal government's categorization system, and in my experience, it is also in use at the municipal level in the United States. Here's a longer description: NAICS was developed under the auspices of the Office of Management and Budget (OMB), and adopted in 1997 to replace the ...


9

The W3C offers a collection of tools that convert from CSV to RDF. However, there is no explicit mention of RDFa in any of these CSV converters. Personally, I'd give the RDF Refine plugin for OpenRefine a try. Once you have your open data in an RDF format, you could use the RDF Translator to turn it into RDFa.


9

Well my comment received a number of up votes which I take as a signal of quality and I am posting the links here so they are more visible to future visitors: opendata.socrata.com - you can upload a number of different file types here, create visualizations, link to them, and take advantage of a very mature set of APIs for data consumption and publishing ...


8

The Guardian data website has a variety and insights in to UK, European and World data. The site also includes data visualisationa and application you can use. Very enjoyable. http://www.guardian.co.uk/data


8

Regarding this question, I think this post by Denny Vrandečić (the project leader of Wikimedia's Wikidata) is well worth reading: https://plus.google.com/104177144420404771615/posts/cvGay9eDSSK Denny knows very well what he is talking about. I'll just quote the first sentences as a teaser: tl;dr - If you publish data, attach the CC0 license to it, but ...


8

There are some great resources here. I would like to add http://knoema.com to this list. They are possibly the most comprehensive and constantly updating resource. All their data is free to use and links back to source. Plus the data exploration tools and search engine is pretty useful. Their dataset explorer is here: http://knoema.com/data I found that ...


8

databib.org re3data.org recently merged: Registry of Research Data Repositories i.e. RE3 "The consolidated registry contains information for more than 1,130 data repositories that are accessed by over 5,000 unique visitors each month. On average, 10 new repositories are added every week." "A new REST-API is currently being beta tested that provides the ...


8

Here are the three options listed on opendatacommons.org, the licenses FAQ also has a bunch of really good information. Public Domain Dedication and License (PDDL) — “The PDDL places the data(base) in the public domain (waiving all rights)” Attribution License (ODC-By) — “Attribution for data/databases” Open Database License (ODC-ODbL) — “Attribution Share-...


8

Transmodel is a not very widely used format for schedule data (alternative to GTFS). For real time data (alternative to GTFS-realtime): SIRI is an XML protocol used most heavily in Europe. You'll want to consider what formats developers are most aware of and any possible performance issues. TRANSMODEL has been adopted as the European experimental ...


Only top voted, non community-wiki answers of a minimum length are eligible