2022 Developer Survey is open! Take survey.
9 votes
Accepted

Searching for Open Data Dataset That is No Longer Online

The Memento Web and the Wayback Machine are two possible solutions: Wayback Machine The Wayback Machine by the Internet Archive is your best friend for all things that were once online, and even ...
user avatar
8 votes

A Python guide for open data file formats

Once you've decided on a data format, you will need to decide the following additional issues before publishing your open dataset: Data Layout Name/Data type (Vocabulary) of data fields. Language and ...
user avatar
7 votes
Accepted

Is it okay to download all datasets from a government open data portal?

Bad manners have nothing to do with it. As a taxpayer, it's your data, and they're making it available for you to download. Your country also benefits from having backups of their dataset floating ...
user avatar
  • 776
6 votes

What are the most useful formats in which to release geospatial data?

My experience, making maps from quite a few government datasets: For point data, CSV is the best, with "lat" and "lon" columns. Very easy to work with in a wide range of tools, including text editors,...
user avatar
6 votes

What type of software is used by the Open Data community?

Languages: R, Python, JavaScript, HTML, CSS Formats: CSV, ODS, JSON (.json, .geojson, .topojson), SHP, HTML Tools: GDAL - Geospatial Data Abstraction Library - a translator library for raster ...
user avatar
  • 11.8k
6 votes

Hackathons and opendata

One resource I have sent many people to (and they always find value in it) is Socrata's* "hackathon in a box" guide: http://hackathon-in-a-box.org/guide/ The Open Knowledge Foundation also has a ...
user avatar
6 votes
Accepted

How to collect hand-writting data

The first step is splitting the image into character arrays. To do that, check out the answers in this question: Separate image of text into component character images. In particular, the ImageMagick ...
user avatar
  • 17.1k
5 votes

Which namespace should be used for an open vocabulary published by a private organization?

I would use purl.org yes. That's weird that they're not answering. They answered pretty quickly when I tried myself. There's also https://w3id.org/ if you're looking for stable and persistent ...
user avatar
5 votes

A web API user's guide for free and open data

I found this website (http://www.datasciencetoolkit.org/). There is an API collection in there with great tools, especially to "clean" and prepare your open datasets for analysis and visualization. A ...
user avatar
  • 4,716
5 votes

Let's suppose I have potentially interesting data. How to distribute?

UPDATE: datahub.io is: a free, powerful data management platform from the Open Knowledge Foundation, based on the CKAN data management system. Datasets can be added to as part of the ...
user avatar
  • 17.1k
5 votes

Metadata standards and best practices for data dictionaries for CSV files/data

The National Information Exchange Model (NIEM) is an XML-based system for defining "data in motion" (i.e., an on-the-wire format.) What distinguishes NIEM is 2 things: it is for standardizing the ...
user avatar
5 votes

Extending North American Industry Classification System (NAICS)

I don't know of any existing standards (de facto or not) for extending NAICS, but what I would do is start with the NAICS Index File, which ties over 19,000 industry names to the standard 1,000 or so ...
user avatar
  • 431
4 votes

Metadata standards and best practices for data dictionaries for CSV files/data

I have recently started using the Linked CSV proposed standard for generating CSV files from plural data sources. Below is a vocabulary definition for the columns/data types I am using. Perhaps others ...
user avatar
4 votes
Accepted

Scraper for Openstreetmap: all south-american schools to mysql-db

Here is some information on how to get closer to what you might be looking for. One well-organized source for information like this is a site called GeoNames.org. GeoNames has a number of APIs as ...
user avatar
4 votes

A web API user's guide for free and open data

Microformats are far more implemented than other "technologies", and plenty of big companies, pushing out a lot of data, implement them. There's no great advertisement for them, so you have to figure ...
user avatar
  • 11.8k
4 votes

successful open data advocacy website

Public advocacy is effective when it shows decision makers the examples they'd like. But it's not always clear what they'd like. The best bet is to demonstrate them good examples widely used across ...
user avatar
4 votes

Cases where open data has been removed?

Here's an example from the NIH in 2008. Basically, the NIH learned that in some cases a clever algorithm could identify medical patients from two sets of open medical data. This is an early case in ...
user avatar
  • 41
4 votes

Cases where open data has been removed?

One you'll probably remember - Wikimedia released (CC0) search logs in 2012, realised they were insufficiently anonymised, and took them down the same day: https://blog.wikimedia.org/2012/09/19/what-...
user avatar
  • 693
4 votes

How should US SSN be anonimized?

Since SSN has only 9 digits, changing hash function will not suffice because attacker can simply apply the function to all 10^9 SSN's and match the result against the database. One option is to use a ...
user avatar
3 votes

Tool to extract the main concepts/topics from web pages

Thanks Tasos for your very helpful DBpedia Spotlight Python code sample above. However, converting it from Python 2 to Python 3+ wasn't entirely straightforward so thought I'd post the revised version....
user avatar
  • 31
3 votes

What are good examples of open data dashboards?

The World Bank: Data Open Government (US) Open Data Communities (UK) San Francisco Data
user avatar
  • 211
3 votes

What are good examples of open data dashboards?

You already listed a bunch. There's also Philadelphia's Open Data Pipeline Whitehouse Scorecard
user avatar
  • 5,087
3 votes
Accepted

Datasets for smaller towns or villages

I was a Town Commissioner for 6 years of a small town (population ~640, but we're also the County Seat, so our daytime population is over 3000; exact number dependent upon how many people have jury ...
user avatar
  • 4,485
3 votes

What would be the cleanest / easiest method of collecting rental property data?

HUD offers a lot of rent related information (by County). For example, the 50th percentile estimates contain the medium rent per studio, 1, 2 and 3 bedrooms per county. http://www.huduser.org/portal/...
user avatar
3 votes
Accepted

Fatality Analysis Reporting System

First you will need to define the schema of your final database and determine how you will map each file to your schema. Then I recommand using ETL tool (Extract Transform and Load) such as Talend ...
user avatar
  • 1,535
3 votes

Let's suppose I have potentially interesting data. How to distribute?

Academic Torrents is a platform to share research releated dataset and paper through a peer to peer network. Academci Torrents host a list of tracer so other can search for the torrent. Data are ...
user avatar
  • 1,535
3 votes

What are the most useful formats in which to release geospatial data?

Geospatial Data Standardization No more KML, KMZ, SHP files! I will make the somewhat bold and possibly controversial statement that geospatial data should always be available in two single ...
user avatar
3 votes

Metadata standards and best practices for data dictionaries for CSV files/data

It looks like Jeni Tennison took her Linked CSV spec to the W3C, and with some help from Gregg Kellogg and Ivan Herman it evolved into the Model for Tabular Data and Metadata on the Web and the ...
user avatar
3 votes

Metadata standards and best practices for data dictionaries for CSV files/data

The Data Documentation Initiative provides the DDI metadata standard for describing data dictionaries. http://www.ddialliance.org/ https://en.wikipedia.org/wiki/Data_Documentation_Initiative The DDI ...
user avatar
  • 131
3 votes

Hackathons and opendata

The School of Data has an active community of volunteers that you can engage for either free or very reasonable pricing I presume. Their website has great material (open to use and distribute) made ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible