Questions tagged [data-format]

For questions regarding data formats and file formats, such as comparison of different data formats or recommendations of a file format.

Filter by
Sorted by
Tagged with
85
votes
20answers
146k views

How can I work with a 4GB csv file?

What's the best way to access a 4GB csv file? I would like get a 'cut' of this open data set: Full Replacement Monthly NPI File, available here http://nppes.viva-it.com/NPI_Files.html Specifically, ...
48
votes
6answers
17k views

A Python guide for open data file formats

If you are an open data researcher you will need to handle a lot of different file formats from datasets. Sadly, most of the time, you don’t have the opportunity to choose which file format is the ...
14
votes
3answers
443 views

Do you know any open/standard resume format?

I would like to know if there is any kind open/standard resume or Curriculum Viate format which allows interchanging of resumes between systems. Any hint?
11
votes
3answers
268 views

What are the most useful formats in which to release geospatial data?

In the past, I have released data as zipped Shapefiles (for the professional GIS crowd) and KML/KMZ (viewable by anyone using Google Earth). Are there other formats that would be especially helpful?
11
votes
5answers
487 views

Standards for self-documenting text files?

One of the advantages of most scientific file formats is that you have a block of attached metadata that tells a reader how to read / interpret the file (what the fields are, for the most part -- how ...
11
votes
2answers
124 views

A standard format for English vocabulary

Is there a standard format for English vocabulary lists and some data sources in that format? (For example a standard XML) For example SAT vocabulary or Basic English words, including definition, ...
9
votes
3answers
752 views

Alternative to GTFS (General Transit Feed Specification)?

I recently heard of GTFS as a specification for the times and geographic information regarding transit networks. It is initially used by Google to integrate them into Google Maps but is now made to be ...
8
votes
12answers
3k views

Best way to convert Excel Files to Open Data Formats

We have plenty of data buried in a lot of Excel files (.xls, .xlsx). The data comprises mainly statistics (first row comtains colum headers, first column contains row headers, rest of data are decimal ...
8
votes
5answers
3k views

What's the best way to host map tiles?

I'm interested in generating some open-source maps, with possibly world-wide coverage. I know that depending on the number of zoom levels I include, the number of tiles can be very large, taking up a ...
8
votes
5answers
206 views

What are the standards for data in terms of Open APIs?

Update: It appears there have been two directions in terms of answering this question. Data interchange (ex. HTTP, FTP, P2P), and data formatting (ex. JSON, XML, CSV). My original question was about ...
7
votes
5answers
434 views

Which format (CSV, JSON, Atom, RSS?) should events data be published in?

I'm developing recommendations for local councils publishing event listings. Compared to other kinds of data, events data seems very likely to be used by web and mobile apps (as opposed to downloaded ...
7
votes
2answers
100 views

Open Data formats used

I would like to know if there is any kind of official study with the OD formats used to publish datasets worldwide with respective percentage.
7
votes
3answers
132 views

Is there an open data format for screen/play scripts?

I'd like to write an app that would utilize dialog from a play script, but I don't want to reinvent the wheel. Is there an open format for encapsulating characters, locations, dialog, scenes, acts, ...
7
votes
1answer
185 views

What's a good file format for sharing 3D models of cities?

Some councils have 3D models of the buildings which they're considering sharing. What would be a good format to encourage them to use, if possible? I had thought 3DS would be the standard, but ...
7
votes
1answer
87 views

Data management practices

Software has code repositories and package managers, so installation and maintenance are done in one line (or click). What are practices for managing data, instead? Apart from manually organized ...
7
votes
1answer
87 views

Protocol/software to publish weather information online

I've got a temperature and humidity sensor that captures the data outside the house and whose data I collect every 5 minutes. Is there a standard protocol/format that I can use to publish those data ...
7
votes
1answer
104 views

Format for exchanging open data catalogs

There are a number of catalogs of open data portals out there. Like many well intended efforts, many of them go stale for long periods of time or altogether. One way to combat this problem is to have ...
6
votes
6answers
113 views

Suggested formats for open data documents

I work for a software company and am thinking about publishing our training manuals as open (creative commons) licensed documents, so that others can use them, edit them, and derive new products/...
6
votes
2answers
278 views

Tales of woe from fixed-format (non-delimited) ASCII data distribution

It seems that many/most government surveys in the health sector are distributed in a fixed-format ASCII, often with SAS code provided to create variables out of the mess (e.g. NHAMCS values and ...
6
votes
2answers
121 views

Open data quality frameworks

I'm looking for some kind of lightweight framework for assessing the quality of datasets, as a publisher. The most widely known one is Tim Berners-Lee's "5 star open data", but it's actually not very ...
6
votes
1answer
68 views

Plans and documentation to tidy RAW data in Maude?

MAUDE is the Manufacturer and User Facility Device Experience database from the FDA. The MAUDE database houses medical device reports submitted to the FDA by mandatory reporters 1 (manufacturers, ...
5
votes
2answers
147 views

Optimal Data Format(s) for Open GIS Data Repositories?

What format(s) would/should be provided/included with a gis dataset repository? I know that there is no correct answer, i'd like as many different answers/opinions as possible to make the best ...
5
votes
2answers
47 views

Formats of texts from wikisource?

In what formats can I retrieve texts from wikisource? The obvious format is the HTML they deliver for reading, but the HTML source is very layout oriented and messy. Is there a downloadable source ...
5
votes
1answer
197 views

IRS Codes in machine readable format

Are there any sources of machine readable IRS codes (e.g. state, country etc.)? For example, on the 1099-B instructions, the 1f code on this form. However I would like avoid having to scrape the pdf ...
5
votes
1answer
57 views

Government standards, guidelines, and practices for collecting data

By standards, guidelines, and practices for collecting data I mean the rules that facilitate: Practical value: Collecting data with a specific goal in mind Data integrity: Minimizing omissions and ...
5
votes
1answer
46 views

Open database of filename extensions

Filename extensions like ".doc" or ".licx" are usually associated with a type of file, and are sometimes strongly associated with one or several pieces of software. Is there an open database of these?...
4
votes
2answers
644 views

Is there a tool to convert geoRSS into geoJSON?

I have a geoRSS file that I would like to convert to geoJSON to use it with Leaflet. I found MyGeodata Converter (mentioned on What are the most useful formats in which to release geospatial data?), ...
4
votes
2answers
1k views

Are there IMDb datasets with IMDb ID numbers and actor birthdates?

I want datasets that have the name of the movie and its link on the site or the tt-number. E.g. for Jurassic Park (1993) it would be in the format: > NAME ID LINK >...
4
votes
1answer
80 views

File formats by scientific community?

I'm at a workshop on data publication, and there's talk about trying to do an audit of scienfitic file formats by related communities. Is anyone aware of any existing surveys that we might be able to ...
4
votes
1answer
207 views

PHP script automatically convert content in open data formats

Is there a PHP script that can take posts from my website and convert them to CSV, RDF, etc.? Can someone help? Thank you.
4
votes
1answer
38 views

How can I convert xy line-plots to textual data values?

I want to reverse engineer a simple XY-line graph to scalar data values. An example of such a plot is linked below, water temperature as a function of datetime. As far as I know, the raw data (...
4
votes
0answers
118 views

How to classify EEG and ECG signals

I'm a student and a beginner on Physionet. What is the best method to classify EEG and ECG signals: KNN or SVM? And how can I do it with matlab? Should I convert the .dat files to .mat or just extract ...
4
votes
0answers
175 views

Example of credit card usage data

I want to understand the data saved when a person uses his or her credit card in a shop or online. Is there a common data set format for this? If so, what does it contain? Furthermore, can anyone ...
3
votes
2answers
648 views

Applying filters to headers in a huge CSV file

I am working with a CSV file that is approximately 500 MB. When I open this file in Excel, the wrong data is displayed under the column headers. I suspect this may be due to the large size of the CSV ...
3
votes
1answer
35 views

What data formats that can be published on the first level (1 star) of open data?

I wonder what are the possible data formats that can be published for 1-star level of open data. The only example I have found is with the PDF format, I wonder if there are other possible formats to ...
3
votes
2answers
81 views

How to extract data from web files? [closed]

Looking for some brief information on how to extract data from files on the web? If there are any tutorials or pieces that you know of, it would be great help.
3
votes
1answer
86 views

Spend transaction data formats

What is the best format that we cam reasonably get government bodies to publish their spend transaction data in? In an ideal world we would ask for Linked Data, where every supplier was correctly ...
3
votes
2answers
217 views

Elevation data by latitude and longitude for the Grand Canyon

I am looking for elevation data organized by latitude and longitude for the Grand Canyon. I have found USGS elevation data (http://nationalmap.gov/elevation.html), which is in the form of an ESRI ...
3
votes
1answer
49 views

First and last date in YYYY-MM-DD format of ISO 8061

Which is the first date of ISO 8061 format? Is it by Georgian Calander (1582-1-15) or 0001-1-1 ??? Which is the last date? I guess it must be 9999-12-31. Am I right??? The citations given in Wiki ...
3
votes
1answer
55 views

Common but out-of-the-ordinary file formats

I am wondering what some file formats are that are out of the scope of "normal" file formats you typically find on the web. The "normal" file formats I'm not considering in this post are: Image ...
3
votes
0answers
173 views

What are the limitations of the Wikidata data model?

Wikidata data model is as follows: The following diagram is a more summarized version of it (source): All terms are defined here. I wonder what the limitations Wikidata data model are.
2
votes
1answer
118 views

90ies Windows .exe database on CD: Extract underlying data

I would like to extract data from a register stored on a CD from the 90ies. The Windows executable on the CD allows one to browse the database with keywords and by specific variables. However, there ...
2
votes
1answer
2k views

A dataset with ~10 million rows

I am looking for a dataset with 10 millions of rows to analyze it. Actually to rework it into more usable format and come up with some interesting metrics for it. So there are two requirements: 1) ~...
2
votes
2answers
44 views

Any libraries for SEC Forms, such as a 10Q, as an Interface?

I'd like to process some SEC data, such as a 10Q, but don't want to build a parser for the data myself. Are there any existing libraries that can parse or represent these types of forms?
2
votes
1answer
1k views

Corruption Perception Index (CPI) by Transparency International

I need the indeces in analyzable format, i.e. csv, xls, from the year 2005 to 2015. The results are in HTML table/PDF format on the TI website. Can anyone link a source or write a script to scrape the ...
2
votes
1answer
187 views

Working with Sample Surveys, Statistical Packages, and Fixed Format ASCII Data (.dct ,.do, .sas, .sps) and .dat

US Department of Education sent me National Household Education Surveys Program data via cd-rom and I'm looking for tips/software/best-practices here for converting/viewing/manipulating this data. ...
2
votes
1answer
35 views

Seeking source of curated/cleansed data for 2016 Canadian Census?

I am working with open data sources and the 2016 Canadian Census is of great interest to me. The Canadian Statistics website provides boundary files, which work great, and some aggregated census data, ...
2
votes
1answer
187 views

Finder API list ACA Approved Health Insurance API JSON?

Does Healthcare.gov or Finder API list ACA Approved Health Insurance in API JSON format?
2
votes
1answer
32 views

What is your level of external data?

I'm working on an internship of Big Data, I've build a predictive model. To enrich my model, I'm searching for the open data / external (paid) data. But I could only find the data that is in type of ...
2
votes
1answer
94 views

HIOS Plan Year 2016 XSD Schema

Can someone provide a link to the new (plan year 2016) XSD schema for HIOS admin/plan/rate/etc. data exchange? There have been changes to the XML format for this year and we need to update our ETL ...