Community wiki to collect data sets available to download/seed on bit torrent
DNS Census, the DNS registration dataset snapshot taken in 2013 (compressed ~15GB and uncompressed 157GB).
The DNS Census 2013 is an attempt to provide a public dataset of registered domains and DNS records. It was inspired by the Internet Census 2012 which showed that ...
As of February 2019, you'll find a list of 2600+ open data portals around the world on OpenDataSoft's website. It lists as of now 2600+ portals ranging from the US (obviously) to Afghanistan.
The story behind it:
Working for a SaaS company in need of loads of structured data, our team started to compile a list of all open data portals around the world as ...
Take a look at what OpenStreetMap does. There's a page describing the nature of tile server disk usage. If you go up to zoom level 18 worldwide, you're talking about 91,625,968,981 tiles, which would take around 54000GB of disk space, but would mostly never be viewed.
So I'm not sure if it would ever be a sensible approach, but having said that, I heard ...
Example: I only know of the company Cloudmade because they provide (a now outdated, but fine at the time) download portal of ready-to-use shapefiles derived from OSM data. Though this service might have helped its competitors, they probably earned much more in terms of visibility, which might spawn e.g. development contracts for custom-made solutions.
Ideally, like @Andrew - OpenGeoCode mentions, you would release it in multiple formats.
I would really suggest you look into organizing it into a e-book written in Markdown and hosted on Github. There are several advantages to this such as a built-in change log, being able to let people to (publicly) fork your document(s) and share their changes with the ...
Our open-date site has links to about 600 useful datasets from US and Canadian government and a handful from the UN. The US federal government datasets are sorted by department.
We [opengeocode.org] maintain a catalog of open data portals across the world. We've categorized them under the categories:
Well my comment received a number of up votes which I take as a signal of quality and I am posting the links here so they are more visible to future visitors:
opendata.socrata.com - you can upload a number of different file types here, create visualizations, link to them, and take advantage of a very mature set of APIs for data consumption and publishing
Transmodel is a not very widely used format for schedule data (alternative to GTFS).
For real time data (alternative to GTFS-realtime): SIRI is an XML protocol used most heavily in Europe.
You'll want to consider what formats developers are most aware of and any possible performance issues.
TRANSMODEL has been adopted as the European experimental ...
There are some great resources here. I would like to add http://knoema.com to this list.
They are possibly the most comprehensive and constantly updating resource. All their data is free to use and links back to source. Plus the data exploration tools and search engine is pretty useful.
Their dataset explorer is here: http://knoema.com/data
I found that ...
Regarding this question, I think this post by Denny Vrandečić (the project leader of Wikimedia's Wikidata) is well worth reading: https://plus.google.com/104177144420404771615/posts/cvGay9eDSSK
Denny knows very well what he is talking about. I'll just quote the first sentences as a teaser:
tl;dr - If you publish data, attach the CC0 license to it, but ...
recently merged: Registry of Research Data Repositories i.e. RE3
"The consolidated registry contains information for more than 1,130 data repositories that are accessed by over 5,000 unique visitors each month. On average, 10 new repositories are added every week."
"A new REST-API is currently being beta tested that provides the ...
The basics for FOIA are:
be very clear in your request
Remember that while there are government officials who are hostile to records requests, not all of them are. There are many people in government who want to do the right thing.
There are a few websites designed to help you file FOIA requests. In the US, Muck Rock has been ...
Another system of interest for your list would be the OKFN Global Open Data Index:
It provides national and sub-national catalog listings and categorization for a variety of data types, and an assessment of how open each country is. Similar to DataPortals.org, and run by the same people.
How to stay up to date with UK government data releases:
Office for National Statistics release calendar
Parliamentary releases mailing list
Planning alerts mailing list
The Office for National Statistics release calendar is excellent because it allows you to see weeks in advance what data is going to be ...
GitHub has all the things, seriously, it rules. OKFN hosts open data there. So do I. As do most Code for America brigades. But you can also get a free datahub.io account, at least for an organization and you can upload there.
You should also check out your area. Here in Virginia, there is an OpenVA data portal, and anyone can upload related data after ...
Quick answer with what comes to my mind. They might be several others.
Since you lower the barrier to access the data (no more scraping), you will see more people using it and new usage of your data will appear
You will also receive feedback regarding your data quality (since there is more eyes looking at it under different angles)
This can be a ...
I'm sorry to hear you had a frustrating time uploading to datahub.io and that the site wasn't clear enough about the size limit. Another datahub-like platform that we have worked with in the past is data.world. Their size limit is 500MB.
r/datasets Reddit sub is pretty helpful/has some rather interesting sets.
Data Elixir is a weekly newsletter of curated data science news and resources from around the web. Free for data lovers.
Data Is Plural - A weekly newsletter of useful/colorful datasets
Kaggle's Datasets are pretty extensive, and supported by a vibrant data science community. ...
My experience, making maps from quite a few government datasets:
For point data, CSV is the best, with "lat" and "lon" columns. Very easy to work with in a wide range of tools, including text editors, spreadsheets, etc. The only downside I've come across so far is that GDAL sometimes requires you to make a .vrt companion file. (EDIT: Another downside is ...
The other suggestions on here are great, and I would echo the recommendation to consider Github if you want people to be able to easily edit your product for creating their own versions. This will also help you track the edits that have been made so you can learn more about what people do with your manuals.
If you want to learn more about the definitions ...
The format should fit the data structure not the need of the comsumer (or the one who uses the data).
If the data is semantically stored in a json structure you can easily export the data in different formats.
So my answer is: json
The first step as you said is to define which types of entities need to be removed in the public version of the data set. One example of such definition in the United States is given in the Health Insurance Portability and Accountability Act (HIPAA) rules, which target medical texts.
According to HIPAA rules, the following information should be removed from ...
'Finding' data is a really hot topic. (Would love to up-vote but am too new).
It is something I am asked a lot about at work. We call the process 'Data Landscaping'.
In terms of sites http://www.programmableweb.com/ can be useful - but often comes a bit light. We would then start with data marketplaces and data providers (http://www.quandl.com/ for ...
The UCI Machine Learning Repository provides databases mostly aimed at machine learning researchers.
Amazon Public Data Sets lists all public data sets that can be seamlessly integrated into AWS cloud-based application (currently 55 data sets are available).
French government data: http://www.data.gouv.fr/
Also, this Quora thread lists quite a few others.
a free, powerful data management platform from the Open Knowledge Foundation, based on the CKAN data management system.
Datasets can be added to as part of the organization OpenData StackExchange.
Additionally, participation in the "organization" as a follower, member or admin is encouraged!
Discussion on meta site
It was ...