I'm a programmer and I use OpenRefine all the time. Some of the advantages it has over breaking out Python or some other language include:
results of transformation expressions are previewed interactively with live data
quick, interactive, filter facets which allow for easy browsing of instances/rows which match a variety of filters
exploratory analysis of ...
OpenRefine can be used to parse semi-structured data into a table like structure, where it can be operated on in a manner similar to a spread sheet and exported.
The site features a tutorial on converting a list on wikipedia into a table which may be a good starting point.
The operations involved can also be exported incase you need to perform the same ...
Python is the tool I would use.
from itertools import izip_longest
def grouper(iterable, n, fillvalue=None):
'''Collect data into fixed-length chunks or blocks
grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx'''
args = [iter(iterable)] * n
return izip_longest(fillvalue=fillvalue, *args)
with open("raw_data.txt", 'r') as f:
I think the best benefit from OpenRefine is its GUI. You can always do everything OpenRefine offers you with Python, Java, etc. but for non-programmers it helps them to perform basic (and sometimes not so basic) operations with data without having to learn how to code.
If you're comfortable with R, probably not a whole lot. OpenRefine's sweet spot is for facilitating data management for non-programmers, and packaging together a bunch of common data-munging tasks behind nice point-and-click interfaces. Sunlight (my employer) has used it a fair bit for this purpose, if researchers need to be able to clean up some data ...
NOAA provides weather data. You can see the general information and visualization at http://www.weather.gov/ Specific data products are found at http://www.ncdc.noaa.gov/most-popular-data When you click on a dataset you are interested, there is technical documentation and material to guide you in the use of the data. For example, local climatological data ...
If you're doing this interactively, most browsers will format tables as TSV when they're selected and cut. Pasting this into the clipboard dialog of Refine's project creation dialog will allow you to import the data as TSV.
If you've got a bunch to do or need to do this repeatedly, I'd use Google Spreadsheet's importHtml(url,"table",N) function which will ...
The problem with the second example you posted, is there is almost no structure to it. It doesn't even have a consistent ordering of rows for each record.
I think this is where Perl shines, so I went ahead and wrote up a prototype. ( took me about an hour )
I don't think there exists a tool that would be any simpler, that would get as decent a result. ( ...
Open GIS-ready cloud data are available at various levels of detail, both in time and space.
It is quite tempting to work with satellite photos of cloud coverage; unless you are a professional, don't do that - there are a bunch of hidden snags you have to know about.
For current data on (points) airfields and airports of the world, your best bet is METAR, ...
pyparsing is really a powerful tool. As the name implies, it's a module for Python. It's like zooming out one step from regexp. There are many examples at http://pyparsing.wikispaces.com/Examples
Here's another example which also makes use of regexp, in case you also want to use that for more exact pattern matching.
I've always done work like this in Perl -- my basic methodology goes something like this. (note, this is for dealing with multi-GB files ... it can be simplified if you can load the whole thing in memory)
I've denoted helper routines with &, although you probably don't want to use that old perl-4 calling style as it'll force it to ignore function ...
As is often the case when handling HTML tables you will most likely need to save the table into an intermediate format that OpenRefine can accept. This is straightforward for a simple table, and is a reasonable solution for a small amount of data.
As an example, consider the simple HTML table on this webpage. First, select the table and copy to the ...
There is a comparison of different weather forecast services here:
Maybe this helps to select the best service for your requirements.
The Los Angeles Times did some very interesting work using natural language processing (NLP) to analyze their archival recipes and convert them into structured data.
An interview with the team ran on OpenNews Source, and Anthony Pesce wrote more (including some code samples) on the LAT Data Desk blog.
JSON is a simple text based data format than can be converted to arrays or any structured data to access it in ...
Besides the NCDC data for forecasts, you can also get pretty detailed meterological information for airports (US and international) in METAR format:
... but to make use of it, you need to be able to decode the station codes to find airports near the location of interest.
It entirely depends on how unstructured your data is. You'll be making a little "mini-domain language" to put the data into structured form.
The more unstructured it is, the more you going to have to get in the minds of everyone who made the records. Shoot for the most common and put the last 20% in the "complete-by-hand" file.
Python, Sed/awk, Perl are ...
Here is a Gist with some log sources.
It's still not 10 million lines, but one of the larger ones:
(400+ MB filesize, about 2 million lines, from 2015)
You can also consider the google trick
I wrote Wik2dict a decade ago to turn MW database dumps into the dict format. It's python code. Could help you figuring out some things.
The best way to convert this xml into mysql is by using mediawiki's xml import functionality.
If you don't have a good reason to import the dump into mysql it's better to avoid it as it's extremely slow with such a large ...
I found some hints for the schema here https://meta.wikimedia.org/wiki/Help:Export#Export_format
To read manually the XML, try using a viewer like:
head -n [numberoflines] dump.xml (gnu/linux terminal)
I'm a python fan, so that is the path I normally take.
If the CVs are in PDF format, then I use pdftotext to convert them into .txt files (without formatting).
Once in .txt files, to find email addresses, I usually split the lines into individual strings, and then look for strings that contain the '@' character. Since that may include twitter names or ...
It does seem like a scripting language is the proper approach. I'd read line by line. Starting a new record at the top of the file or after an empty line. Use some pattern recognition and you'll be parsing away fairly quickly.