All Questions

138
votes
26answers
24k views

A database of open databases?

While there are many open databases available, is there a database or the project that would contain the information of such databases? In other words, is there an open meta-database of open ...
123
votes
14answers
201k views

Is there a global database of all products with EAN 13 barcodes?

EAN 13 is an international system. Is there an API or database that contains all items that have these barcodes? Like all food, goods you can buy in a regular convenience store. Is there a global open ...
85
votes
19answers
10k views

Is there a Git for data?

I know you can put CSV or XML in Git already, but there are issues for those wishing to collaborate on a dataset for creation, cleaning it up, pull requests, etc. Are there alternative version control ...
82
votes
20answers
138k views

How can I work with a 4GB csv file?

What's the best way to access a 4GB csv file? I would like get a 'cut' of this open data set: Full Replacement Monthly NPI File, available here http://nppes.viva-it.com/NPI_Files.html Specifically, ...
55
votes
6answers
59k views

Open API for nutritional information and/or food barcodes?

I've seen many applications (mobile and web) that use a database of nutritional information and barcodes to track daily food consumption. Smartphones have the ability to scan barcodes, and many mobile ...
48
votes
6answers
16k views

A Python guide for open data file formats

If you are an open data researcher you will need to handle a lot of different file formats from datasets. Sadly, most of the time, you don’t have the opportunity to choose which file format is the ...
41
votes
9answers
1k views

Benefits of using CC0 over CC-BY for data

I've heard a few times that for data one should use rather CC0 (Creative Commons - Public Domain Dedication) license than CC-BY (Creative Commons - Attribution). What is the reason for that? (As ...
37
votes
8answers
844 views

Should I approach an agency unofficially before FOIAing them?

Are there drawbacks to FOIAing (Freedom of Information Act, either federally or equivalent state/local laws) an agency? For example, can it lead to an adversarial relationship?
37
votes
6answers
863 views

Let's suppose I have potentially interesting data. How to distribute?

This is a very simple question: Suppose that I have some sort of specialized data, perhaps that I've collected myself or been a part of the collection. And suppose that nothing prevents me from ...
35
votes
11answers
27k views

Good tools to parse repetitive unstructured data

I'm looking to parse a large number of lines of repetitive but unstructured data. This is a task that happens at least once every project, in my experience, so I'm looking for a tool to transform ...
35
votes
7answers
6k views

Are there any good libraries available for doing normalization of company names?

I've run into a number of use cases where I need to normalize company names in a database before running automated and manual matching. We've usually ended up writing a specific script with endless ...
34
votes
11answers
8k views

Multinational list of popular first names and surnames?

Is there a database containing the list of the most popular first names and surnames (with occurrence count, or at least sorted by popularity) for many nations/countries? I need such data for the ...
31
votes
12answers
2k views

Is there a list of all US Government agencies and sub agencies and is it available via API?

Specifically I'm looking for: Agency canonical names Agency abbreviature/acronym Hierarchy of agencies e.g. Census Bureau is part of Department of Commerce Agency logos Website/social media accounts ...
31
votes
10answers
15k views

Downloadable archive of weather conditions for Europe?

Is there a downloadable archive of weather conditions in Europe? Also, the temperature, humidity, precipitation by location? So the database would enable, for example, to draw the chart with ...
27
votes
12answers
611 views

What are the most common ways that users find out about new data sets?

I am interested in how typical open data users, such as journalists, researchers, companies, developers, and others, find out about new open data sets today. For example, how do sources like search ...
27
votes
6answers
48k views

Cryptocurrency historical prices

I'm looking for cryptocurrency historical data, including prices and market cap (either from exchanges or average price) of the main cryptocurrencies, namely: Bitcoin, Ripple, Litecoin, Ethereum, Dash....
26
votes
8answers
16k views

Open API for SEC data?

Is there any free API for programmatically grabbing SEC filing data, such as company financials or insider trading? It seems ironic that the EDGAR search gives you information in a nice tabular form, ...
25
votes
12answers
3k views

Metadata standards and best practices for data dictionaries for CSV files/data

We publish most of our machine-readable open data in CSV format. What are best practices and/or standards to publish data dictionaries (e.g. definitions of columns in CSV files including human-...
25
votes
15answers
42k views

Are there any open datasets for soccer statistics?

I'm mainly interested in soccer related statistics. There are quite a few different API's relating to soccer, but most of them are commercial and far, far out of my price range. I've looked at DBpedia,...
25
votes
8answers
2k views

Tools for merging similar datasets continuously

How do I merge similar datasets, not just once but on an ongoing basis? This seems like a reasonably common problem so I'd be interested if there is a tool out there aimed at this problem, as a guided ...
25
votes
4answers
29k views

A list of cities of each country

Is there a free csv, xml or in some other format database of all (or least top 20-50 biggest) cities for each country in the world?
25
votes
7answers
2k views

Extracting tables from multiple PDFs

What's the best practice of extracting tables from a large number of PDF, which may be formatted differently? For example, I have a series of PDFs like this one, and I would like to extract the ...
24
votes
3answers
434 views

Are there any regular Open Data conferences?

I'm interested in any Open Data conferences which are held on a regular basis (e.g., yearly). Are there any such conferences?
24
votes
7answers
542 views

Evidence for the economic impact of open data?

Open Data is discussed in many contexts, ranging from transparency, government accountability, and their economical benefits. Are there up-to-date studies and publications analyzing the real and ...
24
votes
7answers
6k views

How can I download the complete Wikidata database?

Wikidata is a new Wikimedia project: It centralizes access to and management of structured data, such as interwiki references and statistical information. This data would be of enormous interest ...
24
votes
5answers
3k views

How to normalize the data when mapping crime reports?

Most maps using count of reports of crime end up being maps of the population density of cities. An attractive response would be to turn these counts into a rate, but a rate per what? Most commonly, ...
23
votes
7answers
26k views

Twitter open datasets

Where can I get Twitter datasets available for analysis? I found two: The May 2011 Calufa Twitter Scrape Cheng-Caverlee-Lee September 2009 - January 2010 Twitter Scrape
22
votes
11answers
4k views

Graph visualization/analysis tool?

Since the Harvard Business Review claimed that Data Scientist is The Sexiest Job of the 21st Century, graph analysis and big data are becoming very popular topics at my university. I'm familiar with ...
22
votes
10answers
17k views

Free database or API of all North American businesses

I'm looking for a free/open source downloadable database/API of all the businesses in North America. The sort of data that I'm looking for includes: name, address, industry, email address, website, ......
21
votes
2answers
228 views

Data for word hierarchies

I'm looking for a dataset which will give me is-a and has-a type relations between senses (and give senses from words). WordNet seems to be the most popular model, but I'm trying to evaluate other ...
21
votes
2answers
8k views

Bulk download Sci-Hub papers

I wonder whether it is possible to bulk download all papers stored in Sci-Hub. I am aware of the questions: Is there a more user-friendly way to download multiple articles from arXiv? Bulk download ...
20
votes
9answers
5k views

Is there an open API for world terrain data?

I'm looking for open-source or publicly-available terrain data, for instance heightmaps or DEMs -- ideally GeoTIFFS by way of an easy-to-use API. For instance: you pass a pair of lat/longs and maybe a ...
20
votes
9answers
3k views

Any uses of JSON-LD?

The formal semantic web, now linked data, using RDF is largely flailing in most regions outside the UK. The main problem is the complexity in implementation compared with the tools typically used by ...
20
votes
10answers
5k views

Wikipedia table to JSON (or other machine-readable format)

I'm interested in the data listed in the tables listed here. Is there a good way to extract the information (as JSON, preferably) without resorting to HTML parsing?
20
votes
6answers
18k views

Any CDR (call data record) dataset?

I would like to do some analytics on CDR data, preferably mobile, but anything will do. Ideally I would like to have between 100MB to 10GB of data. I have been trying to look for this type of data ...
20
votes
3answers
9k views

Historical Weather Forecasts

Does anybody know if there is a place I can find historical data of weather forecasts by forecaster? I'm mainly interested in forecasts of high temperature and low temperature, but precipitation ...
19
votes
8answers
631 views

How can I trust the authenticity of an open data source?

If some site claims to offer a definitive data source, how can I trust that it is not purposely biased? Even excluding elaborate conspiracies, how could I know that it isn't just a big hoax? It ...
19
votes
6answers
20k views

Any Open Data Sets for the (Football) World Cup (in Brazil 2014)?

Are there any public data sets for the World Cup in Brazil 2014? Didn't find anything for download on the official FIFA site (besides a single-page PDF booklet for the match schedule). Ideally the ...
19
votes
5answers
2k views

Is there a complete list of all US tax-exempt nonprofits in machine-readable format?

I've seen they can be purchased from Guidestar, but wondered if there was an freely available open data source.
19
votes
7answers
6k views

Database with monthly climate/weather data by country

I am searching for a online database for climate data (especially temperature) for non-european countries on a monthly basis. 1990-2014 would be nice. Thank you!
19
votes
6answers
401 views

Open Data Scorecards?

How can one evaluate a government entity's progress in making their data open? I've read the 8 Principles of Open Government Data, but I'd like to know if there are tools to specifically grade an ...
19
votes
6answers
2k views

De-identified patient data

Are there any medical or clinical data sources that have de-identified patient data (name, date of birth, sex) as well as basic physical parameters such as weight, or height? I'm specifically ...
19
votes
3answers
2k views

What does CKAN stand for and what does it do?

I have seen this acronym a lot on this website, but I don't know what it means! The main header of CKAN.org claims, "the world’s leading open-source data portal platform". What is a data portal ...
19
votes
7answers
539 views

Keeping track of updates to open data published as CSV

I looked through several related questions here (e.g. Tools for merging similar datasets continuously) but nothing seemed to quite answer my question. So, apologies in advance if I missed something. ...
19
votes
3answers
201 views

Is there an official/complete list of all US government agency “developers' pages”?

The FCC's developers page happens to have a sidebar linking to many other US government agencies' developers pages, APIs, etc. It's a handy list, but I assume the FCC hosts it only out of the goodness ...
18
votes
13answers
8k views

2014 Ebola outbreak dataset

I'm looking for a dataset on the 2014 Ebola outbreak with as many following fields as possible: For each individual contaminated with ebola: Estimated datetime and location of detection Datetime and ...
18
votes
4answers
9k views

Downloading the Panama Papers

I wonder where the Panama Papers can be downloaded. I have looked around, I couldn't find them. Where can I find them? What is the legality, and sensibility of requesting and downloading leaked ...
18
votes
4answers
1k views

Sources of weather data

Barry, could you abuse this site's "answer your own question" feature to create a community wiki answer for sources of weather data (both current and historical), since it gets asked so often?
18
votes
4answers
6k views

Source of ABA Numbers for US banks

I'm looking an open data source of American Banking Association Route Numbers for US banks. The ABA number is the routing number that is found on a US bank check.
18
votes
3answers
5k views

List of public holidays by countries?

When working across country border it makes things easier to know in advance if people in a specific country are going to be on public holiday a certain day. While some calendars application offer to ...

15 30 50 per page