OpenRefine can be used to parse semi-structured data into a table like structure, where it can be operated on in a manner similar to a spread sheet and exported.
The site features a tutorial on converting a list on wikipedia into a table which may be a good starting point.
The operations involved can also be exported incase you need to perform the same ...
Python is the tool I would use.
from itertools import izip_longest
def grouper(iterable, n, fillvalue=None):
'''Collect data into fixed-length chunks or blocks
grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx'''
args = [iter(iterable)] * n
return izip_longest(fillvalue=fillvalue, *args)
with open("raw_data.txt", 'r') as f:
The problem with the second example you posted, is there is almost no structure to it. It doesn't even have a consistent ordering of rows for each record.
I think this is where Perl shines, so I went ahead and wrote up a prototype. ( took me about an hour )
I don't think there exists a tool that would be any simpler, that would get as decent a result. ( ...
PyBossa is an open platform for crowd-sourcing. You write a bit of HTML/JS that is the microtask. They have examples including PDF transcribing. Features include user registration, user credits, statistics.
These blog posts by master Windows programmer Charles Petzold contain a few tables by country, and list a few other sources, notably the FBI.
The analyses are brilliant - they look effortless and to-the-point.
Assumption: you are interested in United States data. This is all I really know about, sorry.
Unfortunately, I don't think you are going to find raw/disaggregated data at a national level. You probably already know about the aggregated FBI UCR datasets.
I think what you are looking for is what the public safety ecosystem calls "Incident" from a system ...
The best example I have heard of is Real-time traffic monitoring using mobile
phone data (PDF).
The idea is to derive road traffic velocity from the position data that the mobile phones within cars "generate" when moving from one base station to the next. The frequency of these base station handshakes approximates the travel velocity of the car. Practical ...
pyparsing is really a powerful tool. As the name implies, it's a module for Python. It's like zooming out one step from regexp. There are many examples at http://pyparsing.wikispaces.com/Examples
Here's another example which also makes use of regexp, in case you also want to use that for more exact pattern matching.
I've always done work like this in Perl -- my basic methodology goes something like this. (note, this is for dealing with multi-GB files ... it can be simplified if you can load the whole thing in memory)
I've denoted helper routines with &, although you probably don't want to use that old perl-4 calling style as it'll force it to ignore function ...
It entirely depends on how unstructured your data is. You'll be making a little "mini-domain language" to put the data into structured form.
The more unstructured it is, the more you going to have to get in the minds of everyone who made the records. Shoot for the most common and put the last 20% in the "complete-by-hand" file.
Python, Sed/awk, Perl are ...
The Los Angeles Times did some very interesting work using natural language processing (NLP) to analyze their archival recipes and convert them into structured data.
An interview with the team ran on OpenNews Source, and Anthony Pesce wrote more (including some code samples) on the LAT Data Desk blog.
Mobile (cell) phone data is more frequently being used for research purposes in a variety of fields and in very novel ways. Three examples to add to that provided by @ojdo include:
Geographic analysis of social divisions & interactions in France
Movement of malaria carriers in Kenya
Optimisation of bus routes in Ivory Coast
Regarding crowd-sourced ...
The Zooniverse team https://www.zooniverse.org/ has a bunch of great projects using crowd sourcing for research by reading in data or extracting measurements and category information from non-machine-readable records. Examples are extracting weather data from historic ships logs, digitized copies of botanical records, astronomical and biological images, ...
I think you are asking about what to do with headers in tabular data, if so, here are my thoughts:
Headers should be one column/one row and only one column/one row. If you have to label across rows, simply add the name of the label to the rows you want to apply it to.
This does lead to extremely long headers, so also another thing to apply here is some ...
I re-read the numbered list, and I don't know if it would qualify -- but there have been examples of people mining what people are talking about (twitter) and looking for (search queries) to extract information that you might not expect:
Air quality : http://www.forbes.com/2010/11/05/air-quality-research-technology-twitter.html
Disease outbreaks : http://...
I have personally found that using Perl and Spreadsheet::ParseExcel was relatively simple and useful for extracting data from Excel sheets.
Another approach that may work is to upload into Google and use the Google APIs to extract the data.
4.5gb twitter data. 41.7 million user profiles, 1.47 billion social relations, 4,262 trending topics, and 106 million tweets.
download link bellow:
also you can check site SNAP from Stanford University
Amazon Mechanical Turk is pretty well setup for these kinds of tasks. Of course, you have to pay workers to complete the tasks, but there are plenty of open-source clients to access the service. You might especially want to look at boto for Python and MTurkR for R (note: I am the developer of this package).
If the data is in tables within pdfs, I've created a script in R for splitting the tables into cells and using OCR. You could use the cell images the script creates and use pybossa or mechanical turk to crowdsource each cell and merge them all back into a table.
It does seem like a scripting language is the proper approach. I'd read line by line. Starting a new record at the top of the file or after an empty line. Use some pattern recognition and you'll be parsing away fairly quickly.