8
votes
A Python guide for open data file formats
Once you've decided on a data format, you will need to decide the following additional issues before publishing your open dataset:
Data Layout
Name/Data type (Vocabulary) of data fields.
Language and ...
- 8,637
8
votes
How can I work with a 4GB csv file?
On Windows, there is also a software called Delimit ("Open data files up to 2 billion rows and 2 million columns large!") http://delimitware.com For instance it can split, sort and extract only some ...
- 181
8
votes
Accepted
Where can I download photoperiod (daylength) data?
Turns out there is a function daylength in the geosphere package in R that calculates day length for any latitude and date.
- 311
7
votes
Accepted
R packages with open data in them
Here is a human-friendly list of all 731 datasets in R, as well as CSV download links.
- 471
7
votes
Accepted
R package for geographic regions
The best place to find classes of R packages is with a task view:
http://cran.r-project.org/web/views/
Within task views, the Spatial view is going to have a large number of options:
http://cran.r-...
- 1,530
6
votes
How can I work with a 4GB csv file?
This answer is not really useful for non-programmers, but if could manage some programming in perl, the Parse::CSV module is especially designed for this task.
From the doc:
It provides a flexible ...
- 235
6
votes
Hackathons and opendata
One resource I have sent many people to (and they always find value in it) is Socrata's* "hackathon in a box" guide: http://hackathon-in-a-box.org/guide/
The Open Knowledge Foundation also has a ...
- 5,153
5
votes
How can I work with a 4GB csv file?
Well, in short, Talend Open Studio for Data Integration is an ETL.
It can be used for many use cases, including data migration, files processing, etc.
You can easily build jobs using a visual editor ...
- 171
5
votes
How can I work with a 4GB csv file?
If you're on Windows, I can't sing the praises of LogParser high enough. It allows you to query files in a wide variety of formats (mostly log formats as that's what it was meant for, but XML and CSV ...
- 151
5
votes
R package for geographic regions
On the software side : an alternative way of getting your data in R would be to find the package that imports spatial datasets. For example rgdal (https://gis.stackexchange.com/questions/19064/how-to-...
- 253
5
votes
Scraping columns from a PDF
The pdftotext library (man page) that comes standard with most linux distros and can be installed on Windows contains a -layout flag that preserves table structure.
pdftotext -layout input.pdf output....
- 17.2k
5
votes
How to scrape rugby union data from espn.co.uk
You can use python, R or other code language as you prefer:
Example for python: https://stackoverflow.com/questions/6325216/parse-html-table-to-python-list
Example for R: https://stackoverflow.com/...
- 275
5
votes
ASCII Character Frequency Analysis
You can pretty easily create character (or word) frequency data based on a few lines of code. Here is python 2.7 code that collects counts of each character
# -*- coding: utf-8 -*-
import string ...
- 17.2k
5
votes
Seasonal Sales Data for time Series Analysis
The US Census Bureau provides a good data source:
U.S. Census Bureau's Monthly & Annual Retail Trade
I work at Quandl and here are some databases you might want to check out:
US Census Bureau ...
- 1,026
4
votes
How can I work with a 4GB csv file?
I am a big fan of tad which crunches these kind of files easily.
- 141
4
votes
How can I work with a 4GB csv file?
I recently had to parse the 6GB NPPES file and here is how I did it:
$ wget http://download.cms.gov/nppes/NPPES_Data_Dissemination_July_2017.zip
$ unzip NPPES_Data_Dissemination_July_2017.zip
$ split ...
- 141
4
votes
List of data sources at the state, county and zip / zcta level
FRED, from the Federal Reserve Bank of St. Louis offers various economic time-series data on various geographic levels. I personally use these on a county level, where you can find data on: population,...
4
votes
Is there any dataset for problems common people are facing to build apps against?
Look at the Unanswered Questions at Software Recommendations.
Many are open data app ideas waiting for an implementer.
Also, travel is something that most people do from time to time, and it often ...
- 8,003
4
votes
Accepted
Is there any dataset for problems common people are facing to build apps against?
It's generally difficult to build applications to scratch someone else's itch without having a vested interest in it (such as getting paid).
You're typically better off trying to find something that ...
- 4,485
4
votes
Accepted
Data showing what JavaScript operations are usable in what browser
The caniuse.com project includes compatabilities for many browsers versus CSS, HTML, JS API, SVG, and Other categories.
"Can I use" provides up-to-date browser support tables for support of front-...
- 17.2k
4
votes
Open datasets for data mining R
R itself has a datasets package. Check out the R Datasets Package
Description: Base R datasets
Details: This package contains a variety of datasets. For a complete list, use library(help = "...
- 17.2k
4
votes
Scraping columns from a PDF
Within the constraints of using the bot tools that Open Corporates provides, I'm not sure what to suggest.
However, since people will no doubt get here based on more general searches, it's worth ...
- 5,488
4
votes
Why is my scraper returning inconsistent results and timing out?
You may get better answers on CodeReview, but just glancing at your code I have one suggestion that is to split the codes into a scraping and parsing steps. This will help with reproducibility and ...
- 17.2k
4
votes
Why is my scraper returning inconsistent results and timing out?
Some tips for scrapers:
log: don't stop at "sometimes I get data and sometimes I don't" -- when you get unexpected results in the response, log it so you can learn what happened. Specifically when ...
- 5,488
4
votes
Accepted
Global Precipitation Measurement (GPM) data download in R
I created a function to download data IMERG HDF5 and convert them to raster GeoTIFF. You can see the code in https://github.com/AybarCL/GPM-data-management
4
votes
Historical Twitter Data
Unfortunately, the search index for searching tweets only goes back 7 days. Link to the documentation here, it can be found under 'until'.
You might be able to collect tweets from individual users' ...
- 301
4
votes
Accepted
Machine readable list of monetary currencies, including symbol
There are at least three github repos
Currency-List - many formats - "List of all currencies with names and ISO 4217 codes in all languages and all data formats."
JSON (gist) - (I'll use this one for ...
- 17.2k
4
votes
Accepted
Data on hurricanes?
NOAA's Hurricane Center has current gis as well as archived hurricane data
- 11.8k
4
votes
Data on hurricanes?
Wunderground has a quite extensive hurricane archive (since 1851), unfortunately they are a commercial organisation, so you'll probably have to pay for API integration with their historic data. A ...
- 1,443
Only top scored, non community-wiki answers of a minimum length are eligible