56

Here is a guide for each file format from the Open data handbook and a suggestion with a Python library to use. JSON is a simple file format that is very easy for any programming language to read. Its simplicity means that it is generally easier for computers to process than others, such as XML. Working with JSON in Python is almost the same such as working ...


18

APIs are often offered by websites so that developers can use the web-based data for apps, without having the uncertainty and difficulty of scraping the HTML. But it's not necessary to use the data to build apps, and this means that APIs can be a great source of data for research and analysis. Just to name a few types of API data: weather forecasts, ...


13

I surprised no-one mentioned HDF5 yet. This is a documented format with freely available C libraries for using it. It has traditionally been used for high-performance persistence of numerical scientific data. HDF5 files are self-describing and very easy to work with using python (using either PyTables or H5Py). More importantly, HDF5 is platform agnostic and ...


9

I ended up creating my own system combining a bunch of APIs. Here is what I did: I pulled the homepage text from each website and cleaned up all the html, javascript and styles I pushed this text out to various APIs to process and stored the results. I used alchemyapi.com, textrazor.com, opencalais.com. Those APIs have a lot of options but mainly I focused ...


9

The Memento Web and the Wayback Machine are two possible solutions: Wayback Machine The Wayback Machine by the Internet Archive is your best friend for all things that were once online, and even some things that still are, if you want to compare changes. From Wikipedia: The Wayback Machine is a digital archive of the World Wide Web and other ...


8

Once you've decided on a data format, you will need to decide the following additional issues before publishing your open dataset: Data Layout Name/Data type (Vocabulary) of data fields. Language and Language Script. Data Layout Tabular Data Format (TDF) - http://dataprotocols.org/tabular-data-package/ The key features of this format are the following: •...


8

You can use DBpedia Spotlight to extract semantic annotations from DBpedia. Here is the code for Python. You will need these libraries: BeautifulSoup urllib2 urllib json The example is only for one link, but you can create a script to itterate through your url list. from bs4 import BeautifulSoup import urllib2 import json import urllib link = "http://...


7

As said in the question, the format used depends mainly on which project/domain the data comes from. But most of the time the data is useless without metadata information, and when the dataset starts to get big you need a way to extract the metadata from it automatically. In astronomy, where most of the data has been open for decades, the International ...


7

Bad manners have nothing to do with it. As a taxpayer, it's your data, and they're making it available for you to download. Your country also benefits from having backups of their dataset floating around out there. You could take the public service angle of it to the next level by uploading a copy of the full archive to the Internet Archive using their S3-...


6

My experience, making maps from quite a few government datasets: For point data, CSV is the best, with "lat" and "lon" columns. Very easy to work with in a wide range of tools, including text editors, spreadsheets, etc. The only downside I've come across so far is that GDAL sometimes requires you to make a .vrt companion file. (EDIT: Another downside is ...


6

You cannot find "THE" best answer for this question. It always depends of what you want to do with the data. Statistic Analysis If you want to make a statistic analysis, you can use python like the question you mentioned. Otherwise, weka, spreadsheet (like microsoft excel), R or Rapidminer are a few tools I have worked with them. Geodata If you have ...


6

One resource I have sent many people to (and they always find value in it) is Socrata's* "hackathon in a box" guide: http://hackathon-in-a-box.org/guide/ The Open Knowledge Foundation also has a guide which I have not read as closely at http://blog.okfn.org/2012/10/26/hackathons-the-how-to-guide/. And finally, please at least skim ChallengePost's blog post ...


6

The first step is splitting the image into character arrays. To do that, check out the answers in this question: Separate image of text into component character images. In particular, the ImageMagick answer from 2015. (If you can determine how the input is given, then collect the characters as separate images.) To convert the image into a 2D array, you can ...


5

Languages: R, Python, JavaScript, HTML, CSS Formats: CSV, ODS, JSON (.json, .geojson, .topojson), SHP, HTML Tools: GDAL - Geospatial Data Abstraction Library - a translator library for raster and vector geospatial data formats that is released under an X/MIT style Open Source license by the Open Source Geospatial Foundation. As a library, it presents ...


5

UPDATE: datahub.io is: a free, powerful data management platform from the Open Knowledge Foundation, based on the CKAN data management system. Datasets can be added to as part of the organization OpenData StackExchange. Additionally, participation in the "organization" as a follower, member or admin is encouraged! Discussion on meta site It was ...


5

The National Information Exchange Model (NIEM) is an XML-based system for defining "data in motion" (i.e., an on-the-wire format.) What distinguishes NIEM is 2 things: it is for standardizing the semantics of the exchange, not just the syntax; and it as much a process model as it is a technical model. That is, it (the NIEM organization) has developed a ...


5

As I can understand from this blog post they split the database in groups of interest and they update products in base of these groups. Without Ι know more than this blog post, several companies provide the option to contact with them and arrange a different paid package for their API. I don't know if this is the case for Amazon, but why not?


5

I found this website (http://www.datasciencetoolkit.org/). There is an API collection in there with great tools, especially to "clean" and prepare your open datasets for analysis and visualization. A list of the tools: Street Address to Coordinates Street Address to Location calculates the latitude/longitude coordinates for a postal address. ...


5

I would use purl.org yes. That's weird that they're not answering. They answered pretty quickly when I tried myself. There's also https://w3id.org/ if you're looking for stable and persistent identifiers. The best however could also simply be to publish the vocabulary yourself. One of the issues with using URIs for the vocabulary of your data is that there'...


5

I don't know of any existing standards (de facto or not) for extending NAICS, but what I would do is start with the NAICS Index File, which ties over 19,000 industry names to the standard 1,000 or so NAICS codes. Take the list of index entries for each NAICS, give each one a 2-digit sequential ID, and tack this onto the standard NAICS to make it 8-digit (...


4

I have recently started using the Linked CSV proposed standard for generating CSV files from plural data sources. Below is a vocabulary definition for the columns/data types I am using. Perhaps others will find this useful/interesting approach: http://www.opengeocode.org/cude1.1/LinkedCSV-Vocab.php Update: the above link throws a 404, however it is ...


4

For scrubbing big data, OpenRefine (formerly a Google project) is a must. You easily filter and style your data with a host of tools built in.


4

The question is somehow vague as pointed out before. In terms of data analysis, the following are very popular: Statistical software R and the fantastic IDE RStudio. GitHub for collaborating and sharing code. I would also familiarise myself with the following: Scraperwiki for extracting data from the web. Open Refine is a great tool for working with messy ...


4

I would say that geojson is in the top of my personal list. JSON is really easy to use it with a programming language, especially with Python that I am familiar with. Also, it is easy for conversions. If someone has another system and he has to use the data in another format, he could convert it to that format from JSON pretty easy. Also, shapefiles are ...


4

What you're trying to do seems to fall under "topic modeling" or "clustering." Looking at the DiRT wiki's list of clustering tools, perhaps this topic-modeling tool would be a good place to start.


4

It depends what data you want to gather. Twitter is a major source of data that is API-accessible. They have a REST API and a streaming API. There are also a lot of wrappers to make it easier to use those API's from your language of choice. For Twitter and other social media websites, I would suggest looking at the book "Mining the Social Web" But this ...


4

Microformats are far more implemented than other "technologies", and plenty of big companies, pushing out a lot of data, implement them. There's no great advertisement for them, so you have to figure it out, although there are plug-ins that will tell you when a microformat can be mined/consumed/interacted with. I would assume you could do the same with ...


4

Here is some information on how to get closer to what you might be looking for. One well-organized source for information like this is a site called GeoNames.org. GeoNames has a number of APIs as well as downloads ("dumps") which are discussed on their export page. For your use case, an export seems to make sense. Each country's POIs (points of interest) ...


4

Public advocacy is effective when it shows decision makers the examples they'd like. But it's not always clear what they'd like. The best bet is to demonstrate them good examples widely used across other locations. Here are some of them: GitHub and Government The most comprehensive collection of public software projects already implemented in the world. ...


4

Here's an example from the NIH in 2008. Basically, the NIH learned that in some cases a clever algorithm could identify medical patients from two sets of open medical data. This is an early case in which attempted anonymization turned out to be insufficient. http://articles.latimes.com/2008/aug/29/local/me-dna29 FYI, here's my blog post about it at the time....


Only top voted, non community-wiki answers of a minimum length are eligible