Questions tagged [best-practice]

Best-practice questions generally involve a short guide or tutorial related to an outstanding example of the use or implementation of Open Data principles or practices.

Filter by
Sorted by
Tagged with
10
votes
4answers
6k views

A web API user's guide for free and open data

There are many web-based Application Programming Interfaces (API) that allow data-hungry people to get their hands on all sorts of data. It's a great way to collect data as-needed, instead of looking ...
2
votes
1answer
45 views

Patient Workflow Steps in MIMIC-III

I'm trying to build a workflow model for subjects in the MIMIC-III data set. there's clearly no shortage of activities in here but if I were trying to track the major steps in the process (treatment ...
2
votes
0answers
37 views

How avoid duplication in a data lifecycle?

I will put an example. I have a 1 TB CSV file, I need to conserve the raw file and sanitize then. After the sanitization near 10% of entries has changed. Both files need to be stored for years. So,...
11
votes
3answers
267 views

What are the most useful formats in which to release geospatial data?

In the past, I have released data as zipped Shapefiles (for the professional GIS crowd) and KML/KMZ (viewable by anyone using Google Earth). Are there other formats that would be especially helpful?
2
votes
1answer
27 views

reference to share on best practices [closed]

I'd like to find a best practices document to share with the developer of a internal DB interface. Specifically I'd like to show that the approach that has been used for 15+ years results in output ...
15
votes
4answers
719 views

How to publish real-time open data

I'd want to publish some "real-time" data, i.e. my FV panel energy production, on a server. Which is the best way of cataloguing the data and make it available? Energy is just an example, say that I ...
6
votes
4answers
2k views

Tool to extract the main concepts/topics from web pages

I need to find the main themes or topics for each of 50,000 website urls. For example, http://austinzencenter.org/ should return topics such as zen, monastic, austin, retreat, buddha, etc. I'm ...
3
votes
1answer
39 views

How strictly are “universal design” standards enforced in open data?

"Universal design"is a U.S. government standard that forces items, in this case metadata, to be accessible to people with disabilities. Apparently universal design standards were promulgated some ...
48
votes
6answers
16k views

A Python guide for open data file formats

If you are an open data researcher you will need to handle a lot of different file formats from datasets. Sadly, most of the time, you don’t have the opportunity to choose which file format is the ...
3
votes
0answers
32 views

Examples and experiences of text and datamining use of open data?

I am currently working on a research project that looks at the barriers and best practices when it comes to text and datamining. www.FutureTDM.eu Do you have any experience with this in relation to ...
3
votes
0answers
63 views

How does open data differ in quality depending on the source?

Let's talk about government data, e.g. data.gov, first. Putting aside "outliers," are most government data sets guided by e.g. "mandates" by the government? If not, how does the fact that say weather, ...
7
votes
4answers
235 views

Displaying Trust and Data Provenance?

What would be the most useful way to display data provenance on an open data site? Overarching statements on the site seem to be limited in that amongst many datasets, at least some will likely have ...
3
votes
1answer
84 views

Is it possible to request an api key for each one of my users, using their registered email?

I am using the USDA food database in an app, however I will not being able to use it in the live version because of the API request limitations. One workaround would be to give each registered user ...
9
votes
2answers
1k views

Anyone have a good way of comparing two large and unstructured lists (~2k entries each) for commonalities between them?

Let's say, two governments have separate lists of reports, and I'm trying to figure out the most common elements between them. Think of one government administrator reporting # of sick days per year ...
1
vote
1answer
37 views

Should the testing of my learning algorithm be restricted only on standard datasets or can I use any dataset to publish my results?

I've trained a speech recognition model using the LibriSpeech database which isn't as widely used as other datasets like TIMIT or MNIST. Does that factor in any way or can I publish the results ...
25
votes
7answers
3k views

Extracting tables from multiple PDFs

What's the best practice of extracting tables from a large number of PDF, which may be formatted differently? For example, I have a series of PDFs like this one, and I would like to extract the ...
4
votes
1answer
86 views

methods to build your own data set from public domain data sources

Data sets can originate from government sources, corporations, or individuals. When an individual collects data, the exercise of collecting data falls into at least two categories: The individual ...
6
votes
1answer
81 views

How to collect hand-writting data

I'm working on machine learning project that requires I collect hand-writing data of alphabets from an uncommon language (I'm 100% there is no available data out there). My question is how should I ...
1
vote
2answers
36 views

To whom does “author” refer when using schema.org's “MusicAlbum” schema?

I am trying to encode reviews of music albums. If I understand the "MusicAlbum" schema correctly, the =reviewer= is the "author" of the "review" content? Can anyone point me to examples of well-...
25
votes
12answers
3k views

Metadata standards and best practices for data dictionaries for CSV files/data

We publish most of our machine-readable open data in CSV format. What are best practices and/or standards to publish data dictionaries (e.g. definitions of columns in CSV files including human-...
5
votes
3answers
245 views

How should US SSN be anonimized?

I have just gotten access to a US government dataset. It is not open, but could eventually be made open. The dataset includes hashed US SSNs. It looks like they used some general hashing function that ...
8
votes
6answers
312 views

Cases where open data has been removed?

I'm looking for cases where data (including software) that once was open is not anymore, or was not for periods of several months or more. I can imagine several scenarios for this, including ...
7
votes
6answers
317 views

What type of software is used by the Open Data community?

I am a newcomer to the world of Open Data and I would like to know what type of tools/software one could be confronted with. Not for a specific task, but mostly in general. For example: Which tools ...
7
votes
2answers
178 views

successful open data advocacy website

I'm part of a local advocacy group working on getting local government to open up more data and adopt open data policies. We're also trying to raise awareness about the role open data plays in our ...
5
votes
2answers
147 views

Optimal Data Format(s) for Open GIS Data Repositories?

What format(s) would/should be provided/included with a gis dataset repository? I know that there is no correct answer, i'd like as many different answers/opinions as possible to make the best ...
5
votes
3answers
552 views

What are good examples of open data dashboards?

What are good examples of open data dashboards? This can either be: A dashboard that uses open data, for example the London City Dashboard (but better...) A collection of metrics about the open ...
37
votes
6answers
905 views

Let's suppose I have potentially interesting data. How to distribute?

This is a very simple question: Suppose that I have some sort of specialized data, perhaps that I've collected myself or been a part of the collection. And suppose that nothing prevents me from ...
7
votes
2answers
124 views

What would be the cleanest / easiest method of collecting rental property data?

What would be the easiest method of collecting data about rental properties. I'm looking for per-unit information including: Monthly Rent Are utilities included? Monthly Electric bill Heating fuel ...
5
votes
1answer
54 views

Involvement of central statistics bureau with national open data portal

Although this is a very subjective question as each national government has quite a different setup. However, in general, are there any patterns or working models involving statistics bureaus having a ...
5
votes
2answers
93 views

Hackathons and opendata

My organisation is hoping to run a hackathon event in a few months time. We're a public organisation and we have lots of data and ideas how this might run but we need someone to direct and basically ...
6
votes
1answer
118 views

Is there any documentation or case study of scaling up an Open Data API?

Forecasting open data demand is quite challenging. A new public dataset means more downloads, a new app results in way more requests. It would be interesting to discuss the throughput of open data ...
7
votes
1answer
2k views

Techniques for Pulling Prices for a Large Number of Amazon Items

Lately I have been playing around with the Amazon Product Advertising API, and it made me very curious about how Amazon-price checking sites like Keepa and camelcamelcamel actually work. For a ...
14
votes
3answers
246 views

Preservation of blog posts, articles and essays

I am curious if there is there any good way to guarantee that my articles (posted on personal sites and other blogs) will last? Is there a reliable repository for articles, post and essays? Or are ...
7
votes
3answers
64 views

What are best practices for where to answer agency specific Open Data Policy implementation questions?

USAID recently launched a Frequently Asked Questions page to handle Q&A concerning their policy ADS 579. These Q&As are directed at both internal and external audiences. I could see posting ...
8
votes
1answer
77 views

Which namespace should be used for an open vocabulary published by a private organization?

If you want to contribute a Vocabulary to the LOD community but work for a (commercial) company, the question arises within which (DNS) domain you define the vocabulary/ ontology namespace URI/ IRI? ...
4
votes
2answers
93 views

Are there good examples of requiring open data in RFPs?

The recent U.S. Executive Order on open and machine-readable data requires the U.S. government to begin developing ways to integrate open data into procurement and grant-making. Are there good ...
12
votes
1answer
292 views

Searching for Open Data Dataset That is No Longer Online

Many questions are posted here in search of a specific dataset that is either no longer online, or has died from linkrot, or a combination of the two. What is the best way to get around this? Or ...
5
votes
1answer
174 views

Scraper for Openstreetmap: all south-american schools to mysql-db

I have seen the python scraper described here. Well this is very interesting. I am not a programmer and therefore some of the techniques and ideas of scrapers are too complicated for me. But it ...
5
votes
2answers
205 views

Source of open trend data

I'm currently querying Twitter to access trend data. I'm looking for an open alternative to source trend information. The main question is: What's trending today? I'd love to move away from using ...
4
votes
1answer
91 views

Extending North American Industry Classification System (NAICS)

If I want to integrate with the NAICS system but want to "zoom in" on certain categories, and to add categories for various illicit or underground activities. Is there a standard way of extending the ...
7
votes
2answers
118 views

Is it okay to download all datasets from a government open data portal?

I'm thinking of making a copy of all my country's open data portal datasets. I will be rate limiting my downloads. Is that okay or is it bad manners to get a full data dump? I am asking of the ethical/...
1
vote
0answers
67 views

How do small companies access user transaction data?

I am not sure if this is relevant in this forum but I have always wonder how companies like Retention Science, Factual and several other smaller companies seem to be able to get transaction data or ...
4
votes
2answers
154 views

Open data build project examples

This post provides a good explanation why and how the data collection process should be part of the implementation of an open data project rather than just a once-executed-and-then-forgotten step. ...
19
votes
7answers
547 views

Keeping track of updates to open data published as CSV

I looked through several related questions here (e.g. Tools for merging similar datasets continuously) but nothing seemed to quite answer my question. So, apologies in advance if I missed something. ...
1
vote
1answer
84 views

Datasets for smaller towns or villages

While the state level government should produce open datasets for many categories like company registers, land registers and so on... what are here some datasets that are specific to smaller towns or ...
1
vote
1answer
82 views

Fatality Analysis Reporting System

The National Highway Traffic Safety Administration has a database of fatal crashes called Fatality Analysis Reporting System (FARS) http://www.nhtsa.gov/FARS One can download each years data going ...
7
votes
3answers
217 views

What are examples of people sharing personal data as open data?

Personal data usually has to be anonymised, e.g. aggregated, to be published as open data. However, some individuals are willing to share some of their personal data for example to contribute to ...
4
votes
1answer
129 views

Captcha in a website with open data

I don't have an example to showcase, but I have a general question. If a website provides its data open and free, but to download them you need to fill a captcha, can I use an automatic way to auto ...
5
votes
2answers
721 views

Load data from HTML tables into OpenRefine?

One answer to the question Wikipedia table to JSON (or other machine-readable format) suggested to use OpenRefine for handling HTML tables. All I could find regarding OpenRefine and HTML tables were ...
4
votes
2answers
134 views

Examples of metadata for non-uniform collections

Are there examples of metadata — lists of what a site offers, that users search to help them decide what to download — for non-uniform collections like UCI ml or openml.org ? Metadata is ...