8 votes

A Python guide for open data file formats

Once you've decided on a data format, you will need to decide the following additional issues before publishing your open dataset: Data Layout Name/Data type (Vocabulary) of data fields. Language and ...
user avatar
8 votes

How can I work with a 4GB csv file?

On Windows, there is also a software called Delimit ("Open data files up to 2 billion rows and 2 million columns large!") http://delimitware.com For instance it can split, sort and extract only some ...
user avatar
  • 181
8 votes
Accepted

Where can I download photoperiod (daylength) data?

Turns out there is a function daylength in the geosphere package in R that calculates day length for any latitude and date.
user avatar
  • 311
7 votes
Accepted

R packages with open data in them

Here is a human-friendly list of all 731 datasets in R, as well as CSV download links.
user avatar
7 votes
Accepted

R package for geographic regions

The best place to find classes of R packages is with a task view: http://cran.r-project.org/web/views/ Within task views, the Spatial view is going to have a large number of options: http://cran.r-...
user avatar
6 votes

How can I work with a 4GB csv file?

This answer is not really useful for non-programmers, but if could manage some programming in perl, the Parse::CSV module is especially designed for this task. From the doc: It provides a flexible ...
user avatar
  • 235
6 votes

Hackathons and opendata

One resource I have sent many people to (and they always find value in it) is Socrata's* "hackathon in a box" guide: http://hackathon-in-a-box.org/guide/ The Open Knowledge Foundation also has a ...
user avatar
5 votes

How can I work with a 4GB csv file?

Well, in short, Talend Open Studio for Data Integration is an ETL. It can be used for many use cases, including data migration, files processing, etc. You can easily build jobs using a visual editor ...
user avatar
  • 171
5 votes

How can I work with a 4GB csv file?

If you're on Windows, I can't sing the praises of LogParser high enough. It allows you to query files in a wide variety of formats (mostly log formats as that's what it was meant for, but XML and CSV ...
user avatar
5 votes

R package for geographic regions

On the software side : an alternative way of getting your data in R would be to find the package that imports spatial datasets. For example rgdal (https://gis.stackexchange.com/questions/19064/how-to-...
user avatar
  • 253
5 votes

Scraping columns from a PDF

The pdftotext library (man page) that comes standard with most linux distros and can be installed on Windows contains a -layout flag that preserves table structure. pdftotext -layout input.pdf output....
user avatar
  • 17.2k
5 votes

How to scrape rugby union data from espn.co.uk

You can use python, R or other code language as you prefer: Example for python: https://stackoverflow.com/questions/6325216/parse-html-table-to-python-list Example for R: https://stackoverflow.com/...
user avatar
  • 275
5 votes

ASCII Character Frequency Analysis

You can pretty easily create character (or word) frequency data based on a few lines of code. Here is python 2.7 code that collects counts of each character # -*- coding: utf-8 -*- import string ...
user avatar
  • 17.2k
5 votes

Seasonal Sales Data for time Series Analysis

The US Census Bureau provides a good data source: U.S. Census Bureau's Monthly & Annual Retail Trade I work at Quandl and here are some databases you might want to check out: US Census Bureau ...
user avatar
  • 1,026
4 votes

How can I work with a 4GB csv file?

I am a big fan of tad which crunches these kind of files easily.
user avatar
  • 141
4 votes

How can I work with a 4GB csv file?

I recently had to parse the 6GB NPPES file and here is how I did it: $ wget http://download.cms.gov/nppes/NPPES_Data_Dissemination_July_2017.zip $ unzip NPPES_Data_Dissemination_July_2017.zip $ split ...
user avatar
4 votes

List of data sources at the state, county and zip / zcta level

FRED, from the Federal Reserve Bank of St. Louis offers various economic time-series data on various geographic levels. I personally use these on a county level, where you can find data on: population,...
user avatar
4 votes

Is there any dataset for problems common people are facing to build apps against?

Look at the Unanswered Questions at Software Recommendations. Many are open data app ideas waiting for an implementer. Also, travel is something that most people do from time to time, and it often ...
user avatar
4 votes
Accepted

Is there any dataset for problems common people are facing to build apps against?

It's generally difficult to build applications to scratch someone else's itch without having a vested interest in it (such as getting paid). You're typically better off trying to find something that ...
user avatar
  • 4,485
4 votes
Accepted

Data showing what JavaScript operations are usable in what browser

The caniuse.com project includes compatabilities for many browsers versus CSS, HTML, JS API, SVG, and Other categories. "Can I use" provides up-to-date browser support tables for support of front-...
user avatar
  • 17.2k
4 votes

Open datasets for data mining R

R itself has a datasets package. Check out the R Datasets Package Description: Base R datasets Details: This package contains a variety of datasets. For a complete list, use library(help = "...
user avatar
  • 17.2k
4 votes

Scraping columns from a PDF

Within the constraints of using the bot tools that Open Corporates provides, I'm not sure what to suggest. However, since people will no doubt get here based on more general searches, it's worth ...
user avatar
  • 5,488
4 votes

Why is my scraper returning inconsistent results and timing out?

You may get better answers on CodeReview, but just glancing at your code I have one suggestion that is to split the codes into a scraping and parsing steps. This will help with reproducibility and ...
user avatar
  • 17.2k
4 votes

Why is my scraper returning inconsistent results and timing out?

Some tips for scrapers: log: don't stop at "sometimes I get data and sometimes I don't" -- when you get unexpected results in the response, log it so you can learn what happened. Specifically when ...
user avatar
  • 5,488
4 votes
Accepted

Global Precipitation Measurement (GPM) data download in R

I created a function to download data IMERG HDF5 and convert them to raster GeoTIFF. You can see the code in https://github.com/AybarCL/GPM-data-management
user avatar
4 votes

Historical Twitter Data

Unfortunately, the search index for searching tweets only goes back 7 days. Link to the documentation here, it can be found under 'until'. You might be able to collect tweets from individual users' ...
user avatar
4 votes
Accepted

Machine readable list of monetary currencies, including symbol

There are at least three github repos Currency-List - many formats - "List of all currencies with names and ISO 4217 codes in all languages and all data formats." JSON (gist) - (I'll use this one for ...
user avatar
  • 17.2k
4 votes
Accepted

Data on hurricanes?

NOAA's Hurricane Center has current gis as well as archived hurricane data
user avatar
  • 11.8k
4 votes

Data on hurricanes?

Wunderground has a quite extensive hurricane archive (since 1851), unfortunately they are a commercial organisation, so you'll probably have to pay for API integration with their historic data. A ...
user avatar
  • 1,443

Only top scored, non community-wiki answers of a minimum length are eligible