9
votes
Accepted
Searching for Open Data Dataset That is No Longer Online
The Memento Web and the Wayback Machine are two possible solutions:
Wayback Machine
The Wayback Machine by the Internet Archive is your best friend for all things that were once online, and even ...
- 857
6
votes
What type of software is used by the Open Data community?
Languages:
R, Python, JavaScript, HTML, CSS
Formats:
CSV, ODS, JSON (.json, .geojson, .topojson), SHP, HTML
Tools:
GDAL - Geospatial Data Abstraction Library - a translator library for raster ...
- 11.8k
6
votes
What are the most useful formats in which to release geospatial data?
My experience, making maps from quite a few government datasets:
For point data, CSV is the best, with "lat" and "lon" columns. Very easy to work with in a wide range of tools, including text editors,...
- 850
6
votes
Accepted
How to collect hand-writting data
The first step is splitting the image into character arrays. To do that, check out the answers in this question: Separate image of text into component character images. In particular, the ImageMagick ...
- 17.5k
4
votes
successful open data advocacy website
Public advocacy is effective when it shows decision makers the examples they'd like. But it's not always clear what they'd like. The best bet is to demonstrate them good examples widely used across ...
- 3,641
4
votes
How should US SSN be anonimized?
Since SSN has only 9 digits, changing hash function will not suffice because attacker can simply apply the function to all 10^9 SSN's and match the result against the database.
One option is to use a ...
- 141
4
votes
Cases where open data has been removed?
One you'll probably remember - Wikimedia released (CC0) search logs in 2012, realised they were insufficiently anonymised, and took them down the same day:
https://blog.wikimedia.org/2012/09/19/what-...
- 793
4
votes
Cases where open data has been removed?
Here's an example from the NIH in 2008. Basically, the NIH learned that in some cases a clever algorithm could identify medical patients from two sets of open medical data. This is an early case in ...
- 41
3
votes
Tool to extract the main concepts/topics from web pages
Thanks Tasos for your very helpful DBpedia Spotlight Python code sample above. However, converting it from Python 2 to Python 3+ wasn't entirely straightforward so thought I'd post the revised version....
- 31
3
votes
Cases where open data has been removed?
Not sure if this fits your requirement, but the Enron Email Dataset comes to mind. There are multiple locations where you can learn about the dataset, i.e. here: https://www.cs.cmu.edu/~./enron/
The ...
- 131
3
votes
What are the most useful formats in which to release geospatial data?
Geospatial Data Standardization
No more KML, KMZ, SHP files!
I will make the somewhat bold and possibly controversial statement that geospatial data should always be available in two single ...
- 579
3
votes
Cases where open data has been removed?
I can think of one instance that once open data was subsequently removed from public use in future iterations. Unfortunately I don't remember the specifics, but it involved the outcome of trials for ...
- 2,959
3
votes
Metadata standards and best practices for data dictionaries for CSV files/data
It looks like Jeni Tennison took her Linked CSV spec to the W3C, and with some help from Gregg Kellogg and Ivan Herman it evolved into the Model for Tabular Data and Metadata on the Web and the ...
3
votes
Metadata standards and best practices for data dictionaries for CSV files/data
The Data Documentation Initiative provides the DDI metadata standard for describing data dictionaries.
http://www.ddialliance.org/
https://en.wikipedia.org/wiki/Data_Documentation_Initiative
The DDI ...
- 131
3
votes
methods to build your own data set from public domain data sources
First, you "stand on the shoulders of giants." Very little research or data collection is done entirely from 'scratch' (where 'scratch' corresponds to your first example). You build off of previous ...
- 629
2
votes
How to publish real-time open data
I can help. OpenSensors is a real-time service for publishing Open Data, partnered with the Open Data Institute, we are trying to make real-time Open Data access and publishing easy as possible ...
2
votes
successful open data advocacy website
We're working along very similar lines, [rest omitted upon request, converting to robot mode]
Example (click link) of converting public data locked up in PDF to spreadsheet form and doing it ...
- 121
2
votes
What are best practices for where to answer agency specific Open Data Policy implementation questions?
The Open Data Stack Exchange is a fine place for free asking of questions. FAQs are usually edited and distilled, so there's really no conflict. USAID can harvest common questions from here and post ...
- 5,488
2
votes
How should US SSN be anonimized?
This is an expansion to this answer, so please give the votes to that one.
Another option is to generate a private permutation of 10^9 element set using "true" randomness
To anonymize the data ...
- 17.5k
2
votes
Cases where open data has been removed?
Social Media APIs are prime examples here; literally most of them have done this, with the exception of flickr.
Details differ for each, but essentially these companies make API consumption easy/...
- 11.8k
2
votes
Is it possible to request an api key for each one of my users, using their registered email?
Not possible, not respectful of API provider, and borderline unethical:
Not possible:
I bet anything that is a violation of USDA API's terms of use. [EDIT] Since they recommended it, my bet was ...
- 11.8k
2
votes
Accepted
How strictly are "universal design" standards enforced in open data?
u.s. government doesn't even enforce section 508. universal design is just another idea to them. i have yet to see anyone doing it or enforcing it.
- 11.8k
1
vote
To whom does "author" refer when using schema.org's "MusicAlbum" schema?
Yes, that makes perfect sense. I'm seeing JSON-LD? I have an example I've been working on using microdata that looks something like:
<div itemscope itemtype="http://schema.org/Review">
<...
- 131
1
vote
To whom does "author" refer when using schema.org's "MusicAlbum" schema?
It is correct to think that the reviewer would be the Author according to the Review schema. The MusicAlbum schema specifies that the inLanguage value regards the language in which the music was ...
- 258
1
vote
How should US SSN be anonimized?
Since the SSNs you received are already hashed (poorly), why not just replace the existing hashes with random unique strings? That way there's absolutely no way to attack the encryption, since the ...
- 11
1
vote
Cases where open data has been removed?
Here's a relatively well known one:
AOL issued an apology yesterday for posting on a public Web site 20
million keyword searches conducted by hundreds of thousands of its
subscribers from March ...
1
vote
What type of software is used by the Open Data community?
On top of all being listed I will add SQL.
Please see the references links on utilization:
Example: https://spark.apache.org/sql/
http://www-01.ibm.com/software/data/infosphere/hadoop/big-sql.html
...
- 425
1
vote
What are best practices for where to answer agency specific Open Data Policy implementation questions?
Do what is easiest, I think. Getting authority to let people post questions on usaid.gov is problematic at best. Here it is a little more informal.
- 113
1
vote
Accepted
What are best practices for where to answer agency specific Open Data Policy implementation questions?
this is a "depends" question, but i'll lean to always having the answers on your site. you are the authority. you are releasing the data. you are in control.
bringing them here could help alert people ...
- 11.8k
1
vote
What would be the cleanest / easiest method of collecting rental property data?
Zillow has data on bedrooms, square footage and property tax estimates. I think rent is too malleable for you to easily track it on the basis of individual addresses, but you could estimate it based ...
- 355
Only top scored, non community-wiki answers of a minimum length are eligible