Here is a guide for each file format from the Open data handbook and a suggestion with a Python library to use.
JSON is a simple file format that is very easy for any programming language to read. Its simplicity means that it is generally easier for computers to process than others, such as XML. Working with JSON in Python is almost the same such as working ...
Suppose that I have some sort of specialized data, perhaps that I've collected myself or been a part of the collection. And suppose that nothing prevents me from handing this data out to people. In what method should I go about distributing/storing this data so that others will be able to find it and use it, whenever this time may be?
Targeting specialised ...
I'd suggest using JSON Table Schema: http://www.dataprotocols.org/en/latest/json-table-schema.html
Here's a rough outline:
# fields is an ordered list of field descriptors
# one for each field (column) in the data
# a field-descriptor
"id": "field unique name / id",
APIs are often offered by websites so that developers can use the web-based data for apps, without having the uncertainty and difficulty of scraping the HTML. But it's not necessary to use the data to build apps, and this means that APIs can be a great source of data for research and analysis. Just to name a few types of API data: weather forecasts, ...
I have had great luck with https://github.com/jazzido/tabula
Once the PDF is loaded into the system, it takes manual selection of the table to get the data, but I really prefer it over rolling my own computer vision system, as I've found tabula to be highly accurate, and I can't say the same of a 100% automated system.
I'd recommend looking to see how that specific scientific discipline handles their data.
Some common methods include:
Discipline Registry -- you continue hosting the data elsewhere, but notify the service of the availability of the data, and characteristics so that other people can tell if it's of value to them. (eg, Anthropological, Heliophysics)
I surprised no-one mentioned HDF5 yet. This is a documented format with freely available C libraries for using it. It has traditionally been used for high-performance persistence of numerical scientific data. HDF5 files are self-describing and very easy to work with using python (using either PyTables or H5Py). More importantly, HDF5 is platform agnostic and ...
You have a few options for real time (or "near real time", which is when you have a delay between the collection & time to serve it, or for those that sample at a lower cadence)
There are a lot of considerations when dealing with 'real time' data:
Who is the intended audience? (and do they already have standards for serving this type of data?)
Is the ...
There's discussion of exactly this in this question on School of Data Q&A site.
Among other items mentioned there are (all free/open source):
http://coolwanglu.github.io/pdf2htmlEX/ - open-source, looks good but I've not tested for tabular data
http://tabula.nerdpower.org/ - open-source, designed specifically for tabular data but looks a bit of a pain ...
You can find a list of GIS file formats on Wikipedia. Here is a decent overview of open source GIS servers from the gis.se site, these are the servers people who use open data will most likely be using, so target the formats that those servers use. I would consider some kind of open vector/raster format (I like geoJSON for personal projects because it works ...
The other answers so far are all terrific. I'll reiterate one point, and make a new one:
The openness of an API is always important, but when complete, quality bulk data is available some of these access issues become a lot more tolerable. An API is not a substitute for bulk data. The federal government has become very API focused, and many of them have ...
The Memento Web and the Wayback Machine are two possible solutions:
The Wayback Machine by the Internet Archive is your best friend for all things that were once online, and even some things that still are, if you want to compare changes.
The Wayback Machine is a digital archive of the World Wide Web and other ...
Once you've decided on a data format, you will need to decide the following additional issues before publishing your open dataset:
Name/Data type (Vocabulary) of data fields.
Language and Language Script.
Tabular Data Format (TDF) - http://dataprotocols.org/tabular-data-package/
The key features of this format are the following:
I ended up creating my own system combining a bunch of APIs. Here is what I did:
I pushed this text out to various APIs to process and stored the results. I used alchemyapi.com, textrazor.com, opencalais.com. Those APIs have a lot of options but mainly I focused ...
While SKOS certainly might be the best way to represent this information it does require more effort than I believe most will be willing to provide. Couldn't we start with a simple, practical, form for providing a data dictionary with another CSV file with 4 mandatory columns. DatasetName, FieldName, FieldValue or Code, ValueDefinition and an optional ...
Sounds like you could be interested in the Sensor Observation Service (SOS) standard published by the OGC. It's an open standard which describes a service to publish sensor readings and meta data.
I haven't been using it for a while now but I had an SOS server from 52North running successfully for a while.
As said in the question, the format used depends mainly on which project/domain the data comes from. But most of the time the data is useless without metadata information, and when the dataset starts to get big you need a way to extract the metadata from it automatically.
In astronomy, where most of the data has been open for decades, the International ...
You can use DBpedia Spotlight to extract semantic annotations from DBpedia. Here is the code for Python.
You will need these libraries:
The example is only for one link, but you can create a script to itterate through your url list.
from bs4 import BeautifulSoup
link = "http://...
Bad manners have nothing to do with it. As a taxpayer, it's your data, and they're making it available for you to download. Your country also benefits from having backups of their dataset floating around out there.
You could take the public service angle of it to the next level by uploading a copy of the full archive to the Internet Archive using their S3-...
I've actually had decent luck using pdftotext (the poppler version) with the -layout flag (which tries to preserve columns, etc.), then applying regexes on the resulting text. Works much better for generated PDFs than OCRed ones, though.
My experience, making maps from quite a few government datasets:
For point data, CSV is the best, with "lat" and "lon" columns. Very easy to work with in a wide range of tools, including text editors, spreadsheets, etc. The only downside I've come across so far is that GDAL sometimes requires you to make a .vrt companion file. (EDIT: Another downside is ...
You don't identify what kind of areas you're looking for advice on. But I'll highlight several that I think are particularly relevant for Government sources:
Where feasible, data should not just be made available via an API, but also available for download. This supports other kinds of uses. I think this is important as one goal of Open Government Data is ...
You might want to consider maintaining your cleanups as a set of operations or diffs which get applied to the source data. This would help isolate you from changes to the source and allow you to reapply them to a new dataset.
OpenRefine maintains a history of operations, but you could do something similar with a set of version controlled scripts in your ...
Great question. I agree that permanent archiving of blog posts and other digital content is an important challenge in open data. It might be helpful to break this down into parts:
Having a persistent address at which potential users/machines can reference your content is crucial to good archiving, and most of the issues you list ...
To build on some of these answers, the important distinction to remember is that an API is a service, not the data itself. This will be a custom application that you will build, that will have methods for getting at the data in ways that you find useful (at least initially), and that will use the resources (bandwidth, computing power, etc..) that you are ...
Following the discussion in the comment section, I suggest that you have a look at OpenRefine. For a 4,000 rows dataset (two set of 2000 rows each) Refine allow a mix of manual and script cleaning (using fuzzy match). Here is the steps I will follow (based on what I understood) to clean this dataset:
Prepare your data
In a separate tool, merge the two set ...
You cannot find "THE" best answer for this question. It always depends of what you want to do with the data.
If you want to make a statistic analysis, you can use python like the question you mentioned. Otherwise, weka, spreadsheet (like microsoft excel), R or Rapidminer are a few tools I have worked with them.
If you have ...
One resource I have sent many people to (and they always find value in it) is Socrata's* "hackathon in a box" guide: http://hackathon-in-a-box.org/guide/
The Open Knowledge Foundation also has a guide which I have not read as closely at http://blog.okfn.org/2012/10/26/hackathons-the-how-to-guide/.
And finally, please at least skim ChallengePost's blog post ...
The first step is splitting the image into character arrays. To do that, check out the answers in this question: Separate image of text into component character images. In particular, the ImageMagick answer from 2015.
(If you can determine how the input is given, then collect the characters as separate images.)
To convert the image into a 2D array, you can ...