16

I am quite new to open data. I see a lot of people looking for machine-readable data, but to start, I am wondering which tool I can use to start playing with some dataset.

I am particularly looking for a tool that would provide basic text results (I do not look for a classy UI to release the result, even though it would be a plus) from my dataset, when running query on it. I am mostly expecting a DB-like tool, even though the first criterion would be the simplicity of the tool, and a quick setup.

What tool would you recommend me to start?

Improve this question
3
  • 1
    Are you asking "What's a great tool for me to use to understand how to work with data?" or are you asking "what's a good user interface I can use to allow for people to play with my datasets?"
    – Clay Johnson
    Commented May 8, 2013 at 20:14
  • For me. My goal is to acquire and analyse data, not release data. At least, I don't expect the user interface to be refined. I update my post accordingly.
    – Vince
    Commented May 8, 2013 at 20:16
  • 'dataset' had different meaning across research communities. You may be asking about a series of granules (and thus you're dealing with homogeneous records of some form, be they datum, records, or images) ... but in some communities 'dataset' is given to collections of records, and so a 'dataset' is something like "Full resolution, Level 1, 171 Angstrom SOHO/EIT images" ... and a managing a collection of those has much different issues.
    – Joe
    Commented May 12, 2013 at 11:36

9 Answers 9

Reset to default
14

I found OpenRefine really useful for people new to Open Data. It offer a nice and clean spreadsheet like interface with the power of querying by field (like SQL).

Improve this answer
7

learn a general programming language first

I would recommend focusing your efforts on learning a general-purpose scripting language suitable for working with data, such as python or R, before fiddling with dedicated tools for very specific purposes. Such languages have a wealth of resources for interacting with APIs, SQL-like databases, generating visualizations and performing statistical analyses. They also have large and active communities of users who can help point you in the right direction for just about any task.

(High level scripting languages are generally easier to learn and faster to write than low-level languages such as C or fortran.)

Once you have such a universal swiss-army knife in your toolkit, you can branch out to whatever specific tools work best for the data you are dealing with at the time. Nearly everything on the list above falls into one of these more specific problem categories and discusses a resource or tool that can be manipulated through a generic programming language.

Improve this answer
6

There are a lot of tools on the market that try and fill this gap-- Socrata, Google Fusion Tables, and the forthcoming "Enigma.io"

All of them will take some form of commitment to learn how to understand and use. So why not spend that time learning and understand the tool that gives you the most flexibility to do with what you want. In that case, it's probably MySQL or Postgres, both of which are widespread (can run on any platform), tested (have been used for years), and flexible (are open source). Plus, unlike any of the web-based applications I've named, you don't have to worry about the vendor that creates the tool mothballing it. MySQL and Postgres are largely permanent pieces of infrastructure. Once installed on your machine, it'd be hard for Oracle to come and take your SQL away.

They're also easy to learn. I'd suggest grabbing the MySQL binaries for the platform of your choice, then a tool like Sequel Pro for the Mac, that can help you create a table and import your favorite CSV file. Then take some basic tutorials. This one is a good place to start.

Your ROI is likely going to be higher in learning the basics how to grab data and insert it into MySQL or Postgres than it is mastering the others.

Improve this answer
3

I would suggest learning SQL using SQLite, which has a browser plugin for Firefox. Here's a tutorial by Troy Thibodeaux of the Associated Press that I use with students that I teach. I think that learning how to turn human questions into SQL is a good skill for interpreting and analyzing data.

Improve this answer
2

The most basic tool that you can use to manage data sets would be any spreadsheet application. There are limitations such as data manipulation functionality, and some applications have size restriction.

The next step would be a database application such as Microsoft Access. After that, you would go towards a managed relational database system such as MySQL (opensource) or SQL Server. Larger data sets are typically stored in data warehouses run by RDBMS (Relational Database Management Systems).

Improve this answer
2

I recently learned about CKAN. It's an open source data management system that visualizes data. You can play around with it before committing to downloading it by visiting their live demo page.

Improve this answer
2

I think you might also want to look at Neo4j, an open-source graph database.

A new book, Graph Databases by Ian Robinson, Jim Webber, Emil Eifrem is being freely distributed in its raw, unedited form (http://graphdatabases.com/). It explains why the NOSQL databases can provide capabilities, connections and insights that the usual relational database managers cannot.

Basically, I believe you're not going to be able to discern anything from Big Data unless you use tools that prevent your getting hammered by the very bigness of the data.

Improve this answer
2

Another tool that can be used to manage and publish datasets is Junar.

There are City Governments like Palo Alto and other institutions using this to manage datasets, publish data, allow data to be used in different ways, allow for data to be accessed via API.

Improve this answer
1

When dealing big data like that of common crawl(40 tb), I would use a Hadoop MapReduce system, using hdfs for your data storage.

Programs like Hive give you SQL-like abilities over the data. Hadoop is very scalable for a system that will contain lots of data.

One can start off slow with Hadoop using the CDH4 vm image.

http://blog.cloudera.com/blog/2012/08/hadoop-on-your-pc-clouderas-cdh4-virtual-machine/

Improve this answer
1
  • 1
    :) thanks for the suggestion, but that's not really the start I expect. I suppose Hadoop MapReduce needs quite some configuration, and I am not sure this size of data allows to "play" with data without advanced skills. But yeah, probably a good tool for the long term.
    – Vince
    Commented May 8, 2013 at 20:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.