Community wiki to collect data sets available to download/seed on bit torrent
DNS Census, the DNS registration dataset snapshot taken in 2013 (compressed ~15GB and uncompressed 157GB).
The DNS Census 2013 is an attempt to provide a public dataset of registered domains and DNS records. It was inspired by the Internet Census 2012 which showed that ...
As of February 2019, you'll find a list of 2600+ open data portals around the world on OpenDataSoft's website. It lists as of now 2600+ portals ranging from the US (obviously) to Afghanistan.
The story behind it:
Working for a SaaS company in need of loads of structured data, our team started to compile a list of all open data portals around the world as ...
Ideally, like @Andrew - OpenGeoCode mentions, you would release it in multiple formats.
I would really suggest you look into organizing it into a e-book written in Markdown and hosted on Github. There are several advantages to this such as a built-in change log, being able to let people to (publicly) fork your document(s) and share their changes with the ...
Our open-date site has links to about 600 useful datasets from US and Canadian government and a handful from the UN. The US federal government datasets are sorted by department.
We [opengeocode.org] maintain a catalog of open data portals across the world. We've categorized them under the categories:
Well my comment received a number of up votes which I take as a signal of quality and I am posting the links here so they are more visible to future visitors:
opendata.socrata.com - you can upload a number of different file types here, create visualizations, link to them, and take advantage of a very mature set of APIs for data consumption and publishing
Transmodel is a not very widely used format for schedule data (alternative to GTFS).
For real time data (alternative to GTFS-realtime): SIRI is an XML protocol used most heavily in Europe.
You'll want to consider what formats developers are most aware of and any possible performance issues.
TRANSMODEL has been adopted as the European experimental ...
There are some great resources here. I would like to add http://knoema.com to this list.
They are possibly the most comprehensive and constantly updating resource. All their data is free to use and links back to source. Plus the data exploration tools and search engine is pretty useful.
Their dataset explorer is here: http://knoema.com/data
I found that ...
recently merged: Registry of Research Data Repositories i.e. RE3
"The consolidated registry contains information for more than 1,130 data repositories that are accessed by over 5,000 unique visitors each month. On average, 10 new repositories are added every week."
"A new REST-API is currently being beta tested that provides the ...
The basics for FOIA are:
be very clear in your request
Remember that while there are government officials who are hostile to records requests, not all of them are. There are many people in government who want to do the right thing.
There are a few websites designed to help you file FOIA requests. In the US, Muck Rock has been ...
Another system of interest for your list would be the OKFN Global Open Data Index:
It provides national and sub-national catalog listings and categorization for a variety of data types, and an assessment of how open each country is. Similar to DataPortals.org, and run by the same people.
How to stay up to date with UK government data releases:
Office for National Statistics release calendar
Parliamentary releases mailing list
Planning alerts mailing list
The Office for National Statistics release calendar is excellent because it allows you to see weeks in advance what data is going to be ...
GitHub has all the things, seriously, it rules. OKFN hosts open data there. So do I. As do most Code for America brigades. But you can also get a free datahub.io account, at least for an organization and you can upload there.
You should also check out your area. Here in Virginia, there is an OpenVA data portal, and anyone can upload related data after ...
Quick answer with what comes to my mind. They might be several others.
Since you lower the barrier to access the data (no more scraping), you will see more people using it and new usage of your data will appear
You will also receive feedback regarding your data quality (since there is more eyes looking at it under different angles)
This can be a ...
I'm sorry to hear you had a frustrating time uploading to datahub.io and that the site wasn't clear enough about the size limit. Another datahub-like platform that we have worked with in the past is data.world. Their size limit is 500MB.
My experience, making maps from quite a few government datasets:
For point data, CSV is the best, with "lat" and "lon" columns. Very easy to work with in a wide range of tools, including text editors, spreadsheets, etc. The only downside I've come across so far is that GDAL sometimes requires you to make a .vrt companion file. (EDIT: Another downside is ...
r/datasets Reddit sub is pretty helpful/has some rather interesting sets.
Data Elixir is a weekly newsletter of curated data science news and resources from around the web. Free for data lovers.
Data Is Plural - A weekly newsletter of useful/colorful datasets
Kaggle's Datasets are pretty extensive, and supported by a vibrant data science community. ...
The other suggestions on here are great, and I would echo the recommendation to consider Github if you want people to be able to easily edit your product for creating their own versions. This will also help you track the edits that have been made so you can learn more about what people do with your manuals.
If you want to learn more about the definitions ...
The format should fit the data structure not the need of the comsumer (or the one who uses the data).
If the data is semantically stored in a json structure you can easily export the data in different formats.
So my answer is: json
The first step as you said is to define which types of entities need to be removed in the public version of the data set. One example of such definition in the United States is given in the Health Insurance Portability and Accountability Act (HIPAA) rules, which target medical texts.
According to HIPAA rules, the following information should be removed from ...
The UCI Machine Learning Repository provides databases mostly aimed at machine learning researchers.
Amazon Public Data Sets lists all public data sets that can be seamlessly integrated into AWS cloud-based application (currently 55 data sets are available).
French government data: http://www.data.gouv.fr/
Also, this Quora thread lists quite a few others.
a free, powerful data management platform from the Open Knowledge Foundation, based on the CKAN data management system.
Datasets can be added to as part of the organization OpenData StackExchange.
Additionally, participation in the "organization" as a follower, member or admin is encouraged!
Discussion on meta site
It was ...
Data.gov has open sourced it's code, which combines WordPress for the front end with CKAN for the open data catalog. The code is available via Github--for commenting, downloading, or submitting modifications.
This code will continue to evolve with user-driven updates to the functionality needed.
This will be part of the U.S. ongoing contribution to the ...
Apart from MapBox mentioned by Harry I'd also recommend having a look at CartoDB which let's you harness the power of PosGIS without the hassle of maintaining your own server (at a price).
Alternative solution is to get your own (virtual) server running and equip it with a 'geostack' of map server with database and then start building applications using map ...
One reason to open your data is that the people who may be using the data may not be your competitors; by putting it out there for other people to use, there may be people who are able to find interesting uses for your data that you might not have considered.
In the case of science data, the ultimate goal should not be to make a profit, but to make a ...
We have recently created the OpenData StackExchange organization on datahub.io.
Welcome to the Datahub, the free, powerful data management platform from the Open Knowledge Foundation, based on the CKAN data management system.
The Datahub provides free access to many of CKAN's core features, letting you search for data, register published datasets, create ...