23

OpenRefine can be used to parse semi-structured data into a table like structure, where it can be operated on in a manner similar to a spread sheet and exported. The site features a tutorial on converting a list on wikipedia into a table which may be a good starting point. The operations involved can also be exported incase you need to perform the same ...


4

The problem with the second example you posted, is there is almost no structure to it. It doesn't even have a consistent ordering of rows for each record. I think this is where Perl shines, so I went ahead and wrote up a prototype. ( took me about an hour ) I don't think there exists a tool that would be any simpler, that would get as decent a result. ( ...


4

These blog posts by master Windows programmer Charles Petzold contain a few tables by country, and list a few other sources, notably the FBI. The analyses are brilliant - they look effortless and to-the-point. http://charlespetzold.com/blog/2015/07/De-Obfuscating-the-Statistics-of-Mass-Shootings.html http://charlespetzold.com/blog/2015/10/More-Bogus-Gun-...


3

Assumption: you are interested in United States data. This is all I really know about, sorry. Unfortunately, I don't think you are going to find raw/disaggregated data at a national level. You probably already know about the aggregated FBI UCR datasets. I think what you are looking for is what the public safety ecosystem calls "Incident" from a system ...


3

pyparsing is really a powerful tool. As the name implies, it's a module for Python. It's like zooming out one step from regexp. There are many examples at http://pyparsing.wikispaces.com/Examples Here's another example which also makes use of regexp, in case you also want to use that for more exact pattern matching.


3

I've always done work like this in Perl -- my basic methodology goes something like this. (note, this is for dealing with multi-GB files ... it can be simplified if you can load the whole thing in memory) I've denoted helper routines with &, although you probably don't want to use that old perl-4 calling style as it'll force it to ignore function ...


3

The best example I have heard of is Real-time traffic monitoring using mobile phone data (PDF). The idea is to derive road traffic velocity from the position data that the mobile phones within cars "generate" when moving from one base station to the next. The frequency of these base station handshakes approximates the travel velocity of the car. Practical ...


3

Take a look at UTAH: https://github.com/sonalake/utah-parser It's a good tool for handling files like this.


2

The Los Angeles Times did some very interesting work using natural language processing (NLP) to analyze their archival recipes and convert them into structured data. An interview with the team ran on OpenNews Source, and Anthony Pesce wrote more (including some code samples) on the LAT Data Desk blog.


2

This link is the Bureau of Justice Statistics start page on fire-arm related crimes: http://www.bjs.gov/content/guns.cfm


2

Mobile (cell) phone data is more frequently being used for research purposes in a variety of fields and in very novel ways. Three examples to add to that provided by @ojdo include: Geographic analysis of social divisions & interactions in France Movement of malaria carriers in Kenya Optimisation of bus routes in Ivory Coast Regarding crowd-sourced ...


2

The Zooniverse team https://www.zooniverse.org/ has a bunch of great projects using crowd sourcing for research by reading in data or extracting measurements and category information from non-machine-readable records. Examples are extracting weather data from historic ships logs, digitized copies of botanical records, astronomical and biological images, ...


1

I think you are asking about what to do with headers in tabular data, if so, here are my thoughts: Headers should be one column/one row and only one column/one row. If you have to label across rows, simply add the name of the label to the rows you want to apply it to. This does lead to extremely long headers, so also another thing to apply here is some ...


1

I re-read the numbered list, and I don't know if it would qualify -- but there have been examples of people mining what people are talking about (twitter) and looking for (search queries) to extract information that you might not expect: Air quality : http://www.forbes.com/2010/11/05/air-quality-research-technology-twitter.html Disease outbreaks : http://...


1

I have personally found that using Perl and Spreadsheet::ParseExcel was relatively simple and useful for extracting data from Excel sheets. Another approach that may work is to upload into Google and use the Google APIs to extract the data.


1

4.5gb twitter data. 41.7 million user profiles, 1.47 billion social relations, 4,262 trending topics, and 106 million tweets. http://an.kaist.ac.kr/traces/WWW2010.html download link bellow: http://an.kaist.ac.kr/~haewoon/release/twitter_social_graph/twitter_rv.zip also you can check site SNAP from Stanford University https://snap.stanford.edu/data/#web or ...


1

Amazon Mechanical Turk is pretty well setup for these kinds of tasks. Of course, you have to pay workers to complete the tasks, but there are plenty of open-source clients to access the service. You might especially want to look at boto for Python and MTurkR for R (note: I am the developer of this package).


Only top voted, non community-wiki answers of a minimum length are eligible