Questions tagged [programming]

For open data related questions about computer programming, programming languages (such as Python, R, or Java), and application development.

Filter by
Sorted by
Tagged with
86
votes
20answers
148k views

How can I work with a 4GB csv file?

What's the best way to access a 4GB csv file? I would like get a 'cut' of this open data set: Full Replacement Monthly NPI File, available here http://nppes.viva-it.com/NPI_Files.html Specifically, ...
49
votes
6answers
17k views

A Python guide for open data file formats

If you are an open data researcher you will need to handle a lot of different file formats from datasets. Sadly, most of the time, you don’t have the opportunity to choose which file format is the ...
35
votes
11answers
29k views

Good tools to parse repetitive unstructured data

I'm looking to parse a large number of lines of repetitive but unstructured data. This is a task that happens at least once every project, in my experience, so I'm looking for a tool to transform ...
16
votes
4answers
2k views

List of data sources at the state, county and zip / zcta level

I am trying to compile a list data sources - excluding the US Census - that cover the continental US at the state, county and ZIP/ZCTA level. Any help would be appreciated. Additional details: I ...
13
votes
3answers
3k views

What does OpenRefine offer that other data-parsing tools don't?

I see OpenRefine mentioned a lot here, but I don't see it doing much that R and others can't. What capabilities does it offer that I'm not seeing in the promo page that R or other data packages ...
13
votes
2answers
3k views

How many software developers are there in the world, per country?

This question has been asked (and closed) in various forms across this network, ie: Stack Overflow: How many software developers are there in the world? Programmers: Where can I find ...
13
votes
1answer
209 views

Where can I find massive and high dimensional survival datasets

I am working on developing some high-dimensional survival analysis methods with R, but I do not know where to find such high-dimensional survival datasets. Could anyone tell me where to find such ...
12
votes
3answers
2k views

Where can I download photoperiod (daylength) data?

I'm looking to download photoperiod (day length) data for a variety of locations in Europe on a variety of dates. For each date and locations, I need to the number of hours of daylight. Ideally, the ...
11
votes
4answers
6k views

A web API user's guide for free and open data

There are many web-based Application Programming Interfaces (API) that allow data-hungry people to get their hands on all sorts of data. It's a great way to collect data as-needed, instead of looking ...
10
votes
2answers
2k views

Catalog.data.gov using CKAN API with python requests package

I'm not sure if this is a requests issue, a CKAN API issue, or a catalog.data.gov issue. I can do this GET query: import requests import json r = requests.get('http://catalog.data.gov/api/search/...
8
votes
3answers
371 views

What is a good editor for linked data?

I mainly serialize linked data through the Java Jena library. One of the features I really like is where you create a java class from an OWL file, which in eclipse enable autocomplete, when coding an ...
8
votes
2answers
1k views

Scraping columns from a PDF

We are trying to capture rows from PDF tables & output them as lines of JSON - can anyone suggest a good technique for separating columns? Ideally using the tools linked below. Specifically this ...
7
votes
4answers
186 views

R packages with open data in them

I love and play a lot with the Baseball data package for R (Lahman), and I was wonder if there is anything like it but for more oriented to social sciences. Either surveys or admin data (PSID, HRS, ...
7
votes
2answers
156 views

Data on hurricanes?

As a programmer and meteorology enthusiast, I'd love to be able to get latitude/longitude of tropical storms/hurricanes (specifically in the Atlantic). Does anyone know of websites that can provide ...
7
votes
2answers
255 views

Why is my scraper returning inconsistent results and timing out?

I've been building a scraper in python (using mechanize) to get a list of banks off of: https://connectonline.asic.gov.au/RegistrySearch/faces/landing/ProfessionalRegisters.jspx The scraper is: ...
7
votes
2answers
447 views

Historical Twitter Data

I'm trying to get historical twitter data starting from 2010 with specific keywords related to S&P500 companies. I will probably end up using their tickers as keywords once I finalize my list of ...
7
votes
1answer
1k views

What is a good Python CKAN Tutorial?

Where can I find some resources to help me learn how to write Python to CKAN (in this instance data.gov.uk) repositories? I've had some signs of life from bits of code that I've found, but would ...
6
votes
2answers
169 views

R package for geographic regions

Someone told me about a R package with the definition of objects some regions and nations boundaries. I'm trying to find the border of Veneto, an italian region. What's the name of the package? Do you ...
6
votes
3answers
461 views

Coding homework dataset

I would like to compare 25+ different coding (same programming language, if possible Java but other languages are fine too) implementations of the same homework. Where can I find such homework ...
6
votes
1answer
3k views

ASCII Character Frequency Analysis

I've been looking for a table of the frequencies of all 128 ASCII characters, not just letters. The best table I've been able to find is this one, but that one only includes printable ASCII characters,...
6
votes
1answer
131 views

Does this violate copyright of music/lyrics/sound?

So, we are beginning to create a Karaoke app. And we have few queries in the copyrights aspect. We would like to have these cleared before we get any further with our app's development. The decided ...
5
votes
2answers
93 views

Hackathons and opendata

My organisation is hoping to run a hackathon event in a few months time. We're a public organisation and we have lots of data and ideas how this might run but we need someone to direct and basically ...
5
votes
2answers
64 views

Popularity of programming libraries

There are so many libraries, for instance for C# networking or Java encryption. I am looking for data about the "popularity" of published libraries. Any measure of "popularity" is fine, for instance ...
5
votes
1answer
98 views

Machine readable list of monetary currencies, including symbol

Does someone know a machine readable currency list, that includes the symbol? For example Aruba Guilder,AWG,ƒ,ƒ Australia Dollar,AUD,$,$ I've found various sources, but they either lack the symbol (...
5
votes
1answer
250 views

Open datasets for data mining R

Are there any (big) datasets for datamining in R considering that I want to build a sort of expert system application with practical usage? The domains may be: health data, lifestyle etc
5
votes
4answers
5k views

Looking for open dataset containing data for disease and symptoms

I am doing a data mining project on "health prediction system". So, Is there any open dataset containing data for disease and symptoms.
4
votes
5answers
1k views

Stemming of long English text

I need to perform stemming in my application, but mostly for a small subset of stems. I.e. to check if a given word is of a certain stem out of a small list of stem words. Maybe a hundred stem words ...
4
votes
1answer
558 views

Global Precipitation Measurement (GPM) data download in R

I would like to know if resources exist for downloading GPM data in R. I'm aware of the Hydrological Data Discovery Tools ("hddtools") package, which can be used for accessing Tropical Rainfall ...
4
votes
1answer
58 views

Site that provides up to date list of programming languages/frameworks to open source repositories using it?

When you are learning how to program in a certain language or with a particular framework, it is very useful to have an actual real-world example of that project being used somewhere in the open ...
4
votes
1answer
72 views

Where is the real beginner's guide

I am trying to learn something about how APIs work. I was told that DOL has the best API and supporting documentation in the federal government. So here I am, on page 1 of the beginner's guide, and ...
4
votes
2answers
117 views

Sierra Leone Ethnic Groups - Shapefile

I am looking for open data on the Sierra Leone Ethnic groups. Preferably in Shapefile format or any other format that can be used on a map. So far I created a map with the districts and their ...
4
votes
1answer
102 views

Querying OSHA Data Returning Unexpected Results

I'm running the ruby sample provided by DOL to access DOLs OSHA API. The relevant ruby code is: request.call_api 'Compliance/OSHA/full', :select => 'activity_nr,estab_name,...
4
votes
1answer
207 views

PHP script automatically convert content in open data formats

Is there a PHP script that can take posts from my website and convert them to CSV, RDF, etc.? Can someone help? Thank you.
4
votes
1answer
158 views

Load NSF Research Award Abstracts in Python or R

I want to work on NSF Research Award Abstracts 1990-2003 Data Set in R. Every record of the data is stored in a text file and text files are grouped in folders (See data files here) Is there any easy ...
4
votes
1answer
80 views

Lobby meetings of European Commission - Integrity Watch

The staff of the European Commission is obliged to keep a register of their meetings with lobbyists. The site Integrity Watch administered by Transparency International keeps track of these meetings ...
4
votes
1answer
282 views

Where can software developers interested in open government data be reached?

I am looking for most recommended places outside of the open government mailing lists to invite software developers interested in open government data to participate in Poplus.org re-usable civic ...
4
votes
1answer
87 views

geolocalization of tweets

Do you know a technique to find spatial indicators using the package TwitteR in R? For instance if I have to find the word "dog" using the package twitteR, but I miss the coordinates, there is the ...
4
votes
0answers
37 views

Images with symmetry parameters

I wrote a program that searches for aesthetic pictures. One of my parameters is how much a picture is symmetrical. This picture is taken from 33Pol (from Iran-Isfahan). As can be seen, the water ...
4
votes
0answers
59 views

Solved Programming Challenges Dataset

I'm looking for a dataset of solved programming challenges, like the ones on http://codeeval.com. The dataset should have the natural language description of the problem and N solutions for each ...
4
votes
0answers
63 views

Reference data set for benchmarking string comparison algorithms

Recently, I have implemented an extension to the classical Levenshtein-Damerau and Gotoh (aka "affine gaps") string matching algorithms. To evaluate this modification, however, I need a large (in ...
3
votes
3answers
112 views

Where can one find developers interested in a PDF data extraction hackathon?

I'm helping with a sponsored online hackathon for liberating tabular data out of PDF files. Where would be the best places to find developers interested in participating?
3
votes
2answers
481 views

How to scrape rugby union data from espn.co.uk

This is the site I want to gather data from but I'm really not sure where to start. http://www.espn.co.uk/rugby/playerstats?gameId=271423&league=271937
3
votes
3answers
2k views

Historical Mobile App Rankings/Downloads and Prices

I am a PhD student in Economics who is starting a new research project on the mobile app market. For this project, I would ideally need historical data on the rankings (or downloads), prices, and ...
3
votes
3answers
67 views

Looking for Arrests Data at a state level across years

I just find out that R comes with a data about arrests in the US on its base package USArrests. This is a interesting data but only for the year 1973. Anybody knows if there is another package with a ...
3
votes
2answers
2k views

Get Historic Temperature Data by GPS and Timestamp Using R

I have a dataset: ID Time Lat Long a 2016-06-01 10:35:26 41.79925 -71.33052 a 2016-06-01 10:35:57 41.80174 -71.33307 a 2016-06-01 10:37:30 41....
3
votes
2answers
86 views

List of programming languages

I am looking for a large list of as many programming languages possible. The largest resource I have found so far was http://java.dzone.com/articles/big-list-256-programming , but even that is missing ...
3
votes
2answers
119 views

Introduction to Statistical Thought - med.1000 dataset

Does anyone know where the med.1000 dataset used in the Introduction to Statistical Thought book (by Lavine, PDF) comes from? I just can't seem to find it with Google! It contains measurements of ...
3
votes
1answer
875 views

Where can I get a sample dataset for A/B split testing?

I'm working on A/B split testing now. Where can I get a sample dataset for A/B split testing?
3
votes
1answer
77 views

where can I find geocoded sentiments?

I am a student of economics and I am conducting my master thesis on cultural industries. Trying to understand what is the feeling of the people about different themes, I'am using TwitteR package in R ...
3
votes
3answers
849 views

Is there a way to extract the brand names of drugs available in USA?

RxNorm is a terminology that has a good content. It can be downloaded (after signing a user agreement) from NLM website. But it requires dealing with databases (Oracle or MySQL). Can I parse the file ...