I am a newcomer to the world of Open Data and I would like to know what type of tools/software one could be confronted with. Not for a specific task, but mostly in general. For example:
I was hoping for an answer like this one if possible.
You cannot find "THE" best answer for this question. It always depends of what you want to do with the data.
Statistic Analysis If you want to make a statistic analysis, you can use python like the question you mentioned. Otherwise, weka, spreadsheet (like microsoft excel), R or Rapidminer are a few tools I have worked with them.
Geodata If you have geodata and the only thing you want to do is to visualize them in a map, you can use cartoDB.
Visualization A pretty cool tool is the open spending. You can create awesome graphs most of the time for financial data.
Clean data Many times, you will find data that needs cleaning such as removing unwanted parts or fix mispelling labels etc. For this kind of work, Google refine or Open Refine as it is called now is perfect.
I cannot remember something else now. If I refresh my memory with anything else, I will update the post.
Languages:
R, Python, JavaScript, HTML, CSS
Formats:
CSV, ODS, JSON (.json, .geojson, .topojson), SHP, HTML
Tools:
GDAL - Geospatial Data Abstraction Library - a translator library for raster and vector geospatial data formats that is released under an X/MIT style Open Source license by the Open Source Geospatial Foundation. As a library, it presents a single raster abstract data model and vector abstract data model to the calling application for all supported formats. It also comes with a variety of useful commandline utilities for data translation and processing.
http://www.gdal.org/index.html
ogr2ogr - converts simple features data between file formats
http://www.gdal.org/ogr2ogr.html
Open Source Geospatial Foundation - OSGeo was created to support the collaborative development of open source geospatial software, and promote its widespread use.
http://www.osgeo.org/
QGIS - is a cross-platform free and open-source desktop geographic information system (GIS) application that provides data viewing, editing, and analysis capabilities.
http://www.qgis.org/
Mapbox - mapping platform - repos on github
https://www.mapbox.com/
Tabula is excellent for PDF scraping, however its a resource hog:
http://tabula.technology/
Scraperwiki is mentioned in another response, but sadly they've moved on to charging, however you can set up your own version from their sourcecode:
https://github.com/scraperwiki
Google Drive is great for easy storage, sharing/collaborating, and converting to/from other formats.
http://drive.google.com/
I can't emphasis how great drive is for alot of the work I do. You can even use it for scraping HTML tables into Google Spreadsheets, utilizing the importHTML function; read more here:
https://eagereyes.org/data/scrape-tables-using-google-docs
Web Scraping with Google Docs:
https://docs.google.com/presentation/d/1HrCFCMh5XxF4JSma_A5-QpAvlyFhpeBzWkZT7CvEsS4/edit#slide=id.p
geojson.io is amazing for a number of things:
easily convert kml into geojson, topojson, csv, and shp
github/gist integrations allows for easy storage
view maps instanteously upon upload
http://geojson.io/
mapshaper also provides easy format conversion from shapefiles
http://mapshaper.org/
ogr2ogr web client - convert gis data formats
http://ogre.adc4gis.com/
leaflet.js - An Open-Source JavaScript Library for Mobile-Friendly Interactive Maps
http://leafletjs.com/
heatmap.js - Dynamic Heatmaps for the Web
http://www.patrick-wied.at/static/heatmapjs/
datahub.io for data storage/sharing
http://datahub.io/
ckan for running your own data portal
http://ckan.org/
Open Data Kit - (ODK) is an open-source suite of tools that helps organizations author, field, and manage mobile data collection solutions.
https://opendatakit.org/
data.gov - tools they use, repos on github
https://www.data.gov/developers/open-source
Open Data Tools - Tools to Explore, Publish and Share Public Datasets
http://opendata-tools.org/
Nokogiri - web scraping library (ruby)
http://www.nokogiri.org/
Mapping Cheatsheet - this list has some tools already listed, but in greater detail, as well as more tools, plus information in regards to choosing that apply to your particular data:
https://github.com/tmcw/mapmakers-cheatsheet
US Census Github Tools
https://github.com/uscensusbureau
Open Data Research Portal
http://open-data.fokus.fraunhofer.de/en/platform/
Guides:
Open Data Guide
http://opendata.guide/
Project Open Data - Open Data Policy — Managing Information as an Asset
https://project-open-data.cio.gov/
The question is somehow vague as pointed out before. In terms of data analysis, the following are very popular:
I would also familiarise myself with the following:
Scraperwiki for extracting data from the web.
Open Refine is a great tool for working with messy data.
For scrubbing big data, OpenRefine (formerly a Google project) is a must. You easily filter and style your data with a host of tools built in.
While studying bioinformatics one of our professors told us to get comfortable with linux. There's some open-source bioinformatics software that's linux based.
On the other hand a lot of data task don't need such tool.
On the other hand plenty of tools are hosted on servers and linux servers are cheaper than windows servers. Knowing how to communicate with a linux server without having an UI is a useful skill.
R is widely used statistics package. Learning to work with it is useful but you can also do most task within another framework.
On top of all being listed I will add SQL. Please see the references links on utilization: Example: https://spark.apache.org/sql/ http://www-01.ibm.com/software/data/infosphere/hadoop/big-sql.html https://amplab.cs.berkeley.edu/benchmark/
Python,Java/Scala,R and SQL are by default a must to be mastered.
Mac/Linux over Windows.
