7

I am a newcomer to the world of Open Data and I would like to know what type of tools/software one could be confronted with. Not for a specific task, but mostly in general. For example:

  1. Which tools are often used regardless of the task ?
  2. Which tools are a must ?
  3. Is it important to consider the choice of the OS? If so, what are the consequences of working on Mac OSx, Windows or Linux?

I was hoping for an answer like this one if possible.

  • 5
    Your question is really vague. Are you talking about collecting data, releasing data, seearching dataset, cleaning data, interpreting dataset, visaluazing them? – magdmartin Nov 12 '13 at 21:27
  • @magdmartin I wouldn't say vague, I prefer to say it is a very broad question. I really don't know much about Open Data, so I'm just looking for an advice for the first steps in the domain because I'm sincerely interested in becoming a "data scientist". Why not a list of tools needed for the different tasks you suggested? If it is not much to ask of course. – Felipe Aguirre Nov 12 '13 at 23:00
  • 1
    If it helps anyone I've put together a crash course script for newcomers to R – geotheory Nov 20 '13 at 13:25
6

You cannot find "THE" best answer for this question. It always depends of what you want to do with the data.

Statistic Analysis If you want to make a statistic analysis, you can use python like the question you mentioned. Otherwise, weka, spreadsheet (like microsoft excel), R or Rapidminer are a few tools I have worked with them.

Geodata If you have geodata and the only thing you want to do is to visualize them in a map, you can use cartoDB.

Visualization A pretty cool tool is the open spending. You can create awesome graphs most of the time for financial data.

Clean data Many times, you will find data that needs cleaning such as removing unwanted parts or fix mispelling labels etc. For this kind of work, Google refine or Open Refine as it is called now is perfect.

I cannot remember something else now. If I refresh my memory with anything else, I will update the post.

  • Thanks @Anastasios. Can you provide me with some hints about OS compatibility? – Felipe Aguirre Nov 12 '13 at 23:01
  • Well. Except the Statistic analysis, all the other tools are online. I know that python and libre office for excel are compatible with linux. Unfortunately I don't know for weka, R and Rapidminer – Tasos Nov 12 '13 at 23:23
  • R is available for all common platforms as well. Rstudio also has a server edition for if you don't want to run it on your own computer (though then you need to get a server to run it on). – kasterma Nov 13 '13 at 10:53
  • Thank you! I haven't used it in another OS. That's why I didn't know it :) – Tasos Nov 13 '13 at 10:56
5

Languages:
R, Python, JavaScript, HTML, CSS

Formats:
CSV, ODS, JSON (.json, .geojson, .topojson), SHP, HTML

Tools:
GDAL - Geospatial Data Abstraction Library - a translator library for raster and vector geospatial data formats that is released under an X/MIT style Open Source license by the Open Source Geospatial Foundation. As a library, it presents a single raster abstract data model and vector abstract data model to the calling application for all supported formats. It also comes with a variety of useful commandline utilities for data translation and processing.
http://www.gdal.org/index.html
ogr2ogr - converts simple features data between file formats
http://www.gdal.org/ogr2ogr.html
Open Source Geospatial Foundation - OSGeo was created to support the collaborative development of open source geospatial software, and promote its widespread use.
http://www.osgeo.org/
QGIS - is a cross-platform free and open-source desktop geographic information system (GIS) application that provides data viewing, editing, and analysis capabilities.
http://www.qgis.org/
Mapbox - mapping platform - repos on github
https://www.mapbox.com/

Tabula is excellent for PDF scraping, however its a resource hog:
http://tabula.technology/
Scraperwiki is mentioned in another response, but sadly they've moved on to charging, however you can set up your own version from their sourcecode:
https://github.com/scraperwiki
Google Drive is great for easy storage, sharing/collaborating, and converting to/from other formats.
http://drive.google.com/
I can't emphasis how great drive is for alot of the work I do. You can even use it for scraping HTML tables into Google Spreadsheets, utilizing the importHTML function; read more here:
https://eagereyes.org/data/scrape-tables-using-google-docs
Web Scraping with Google Docs:
https://docs.google.com/presentation/d/1HrCFCMh5XxF4JSma_A5-QpAvlyFhpeBzWkZT7CvEsS4/edit#slide=id.p
geojson.io is amazing for a number of things:
easily convert kml into geojson, topojson, csv, and shp
github/gist integrations allows for easy storage
view maps instanteously upon upload
http://geojson.io/
mapshaper also provides easy format conversion from shapefiles
http://mapshaper.org/
ogr2ogr web client - convert gis data formats
http://ogre.adc4gis.com/
leaflet.js - An Open-Source JavaScript Library for Mobile-Friendly Interactive Maps
http://leafletjs.com/
heatmap.js - Dynamic Heatmaps for the Web
http://www.patrick-wied.at/static/heatmapjs/
datahub.io for data storage/sharing
http://datahub.io/
ckan for running your own data portal
http://ckan.org/ Open Data Kit - (ODK) is an open-source suite of tools that helps organizations author, field, and manage mobile data collection solutions.
https://opendatakit.org/ data.gov - tools they use, repos on github
https://www.data.gov/developers/open-source
Open Data Tools - Tools to Explore, Publish and Share Public Datasets
http://opendata-tools.org/
Nokogiri - web scraping library (ruby)
http://www.nokogiri.org/
Mapping Cheatsheet - this list has some tools already listed, but in greater detail, as well as more tools, plus information in regards to choosing that apply to your particular data:
https://github.com/tmcw/mapmakers-cheatsheet

US Census Github Tools
https://github.com/uscensusbureau

Open Data Research Portal
http://open-data.fokus.fraunhofer.de/en/platform/

Guides:
Open Data Guide
http://opendata.guide/
Project Open Data - Open Data Policy — Managing Information as an Asset
https://project-open-data.cio.gov/

4

The question is somehow vague as pointed out before. In terms of data analysis, the following are very popular:

  1. Statistical software R and the fantastic IDE RStudio.

  2. GitHub for collaborating and sharing code.

I would also familiarise myself with the following:

  1. Scraperwiki for extracting data from the web.

  2. Open Refine is a great tool for working with messy data.

  • White spaces in the links if someone with more rep wants to edit that. – Ulrich Nov 19 '13 at 16:26
4

For scrubbing big data, OpenRefine (formerly a Google project) is a must. You easily filter and style your data with a host of tools built in.

3

While studying bioinformatics one of our professors told us to get comfortable with linux. There's some open-source bioinformatics software that's linux based.

On the other hand a lot of data task don't need such tool.

On the other hand plenty of tools are hosted on servers and linux servers are cheaper than windows servers. Knowing how to communicate with a linux server without having an UI is a useful skill.

R is widely used statistics package. Learning to work with it is useful but you can also do most task within another framework.

1

On top of all being listed I will add SQL. Please see the references links on utilization: Example: https://spark.apache.org/sql/ http://www-01.ibm.com/software/data/infosphere/hadoop/big-sql.html https://amplab.cs.berkeley.edu/benchmark/

Python,Java/Scala,R and SQL are by default a must to be mastered.

Mac/Linux over Windows.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.