Questions tagged [best-practice]
Best-practice questions generally involve a short guide or tutorial related to an outstanding example of the use or implementation of Open Data principles or practices.
57 questions
1
vote
0
answers
19
views
How can I sufficiently rigorously create a list from news reports?
For a research paper I'm currently working on, I need to compile a dataset based on various news reports. Obviously, there's no way I can be sure that my list is exhaustive, but how can I compile data ...
- 141
1
vote
0
answers
20
views
Preserving user-privacy whilst enabling strangers to perform data analysis on some form of private data
Context
This question is about the intent to analyse taskwarrior data of people to improve automatic schedulers of tasks for individual taskwarrior users. This intend is accompanied with two desires. ...
- 141
1
vote
1
answer
73
views
Patient Workflow Steps in MIMIC-III
I'm trying to build a workflow model for subjects in the MIMIC-III data set. there's clearly no shortage of activities in here but if I were trying to track the major steps in the process (treatment ...
- 11
2
votes
0
answers
48
views
How avoid duplication in a data lifecycle?
I will put an example.
I have a 1 TB CSV file, I need to conserve the raw file and sanitize then.
After the sanitization near 10% of entries has changed.
Both files need to be stored for years.
So,...
3
votes
1
answer
32
views
reference to share on best practices [closed]
I'd like to find a best practices document to share with the developer of a internal DB interface. Specifically I'd like to show that the approach that has been used for 15+ years results in output ...
- 31
3
votes
1
answer
49
views
How strictly are "universal design" standards enforced in open data?
"Universal design"is a U.S. government standard that forces items, in this case metadata, to be accessible to people with disabilities.
Apparently universal design standards were promulgated some ...
- 541
3
votes
0
answers
36
views
Examples and experiences of text and datamining use of open data?
I am currently working on a research project that looks at the barriers and best practices when it comes to text and datamining.
www.FutureTDM.eu
Do you have any experience with this in relation to ...
- 31
3
votes
0
answers
73
views
How does open data differ in quality depending on the source?
Let's talk about government data, e.g. data.gov, first. Putting aside "outliers," are most government data sets guided by e.g. "mandates" by the government? If not, how does the fact that say weather, ...
- 541
3
votes
1
answer
97
views
Is it possible to request an api key for each one of my users, using their registered email?
I am using the USDA food database in an app, however I will not being able to use it in the live version because of the API request limitations. One workaround would be to give each registered user ...
- 31
1
vote
1
answer
42
views
Should the testing of my learning algorithm be restricted only on standard datasets or can I use any dataset to publish my results?
I've trained a speech recognition model using the LibriSpeech database which isn't as widely used as other datasets like TIMIT or MNIST. Does that factor in any way or can I publish the results ...
- 11
4
votes
1
answer
93
views
methods to build your own data set from public domain data sources
Data sets can originate from government sources, corporations, or individuals.
When an individual collects data, the exercise of collecting data falls into at least two categories:
The individual ...
- 151
6
votes
1
answer
93
views
How to collect hand-writing data
I'm working on machine learning project that requires I collect hand-writing data of alphabets from an uncommon language (I'm 100% there is no available data out there). My question is how should I ...
- 235
1
vote
2
answers
53
views
To whom does "author" refer when using schema.org's "MusicAlbum" schema?
I am trying to encode reviews of music albums. If I understand the "MusicAlbum" schema correctly, the =reviewer= is the "author" of the "review" content? Can anyone point me to examples of well-...
- 131
5
votes
3
answers
270
views
How should US SSN be anonimized?
I have just gotten access to a US government dataset. It is not open, but could eventually be made open. The dataset includes hashed US SSNs. It looks like they used some general hashing function that ...
- 231
8
votes
6
answers
327
views
Cases where open data has been removed?
I'm looking for cases where data (including software) that once was open is not anymore, or was not for periods of several months or more.
I can imagine several scenarios for this, including
...
- 389
7
votes
3
answers
72
views
What are best practices for where to answer agency specific Open Data Policy implementation questions?
USAID recently launched a Frequently Asked Questions page to handle Q&A concerning their policy ADS 579.
These Q&As are directed at both internal and external audiences.
I could see posting ...
- 175
13
votes
1
answer
314
views
Searching for Open Data Dataset That is No Longer Online
Many questions are posted here in search of a specific dataset that is either no longer online, or has died from linkrot, or a combination of the two. What is the best way to get around this? Or ...
- 11.9k
7
votes
2
answers
196
views
successful open data advocacy website
I'm part of a local advocacy group working on getting local government to open up more data and adopt open data policies. We're also trying to raise awareness about the role open data plays in our ...
- 266
7
votes
2
answers
125
views
Is it okay to download all datasets from a government open data portal?
I'm thinking of making a copy of all my country's open data portal datasets. I will be rate limiting my downloads. Is that okay or is it bad manners to get a full data dump? I am asking of the ethical/...
- 319
2
votes
0
answers
80
views
How do small companies access user transaction data?
I am not sure if this is relevant in this forum but I have always wonder how companies like Retention Science, Factual and several other smaller companies seem to be able to get transaction data or ...
- 21
5
votes
2
answers
96
views
Hackathons and opendata
My organisation is hoping to run a hackathon event in a few months time. We're a public organisation and we have lots of data and ideas how this might run but we need someone to direct and basically ...
- 151
4
votes
1
answer
110
views
Extending North American Industry Classification System (NAICS)
If I want to integrate with the NAICS system but want to "zoom in" on certain categories, and to add categories for various illicit or underground activities. Is there a standard way of extending the ...
- 161
4
votes
2
answers
165
views
Open data build project examples
This post provides a good explanation why and how the data collection process should be part of the implementation of an open data project rather than just a once-executed-and-then-forgotten step.
...
- 151
1
vote
1
answer
95
views
Datasets for smaller towns or villages
While the state level government should produce open datasets for many categories like company registers, land registers and so on... what are here some datasets that are specific to smaller towns or ...
- 235
5
votes
1
answer
216
views
Scraper for Openstreetmap: all south-american schools to mysql-db
I have seen the python scraper described here.
Well this is very interesting. I am not a programmer and therefore some of the techniques and ideas of scrapers are too complicated for me.
But it ...
- 361
7
votes
2
answers
144
views
What would be the cleanest / easiest method of collecting rental property data?
What would be the easiest method of collecting data about rental properties. I'm looking for per-unit information including:
Monthly Rent
Are utilities included?
Monthly Electric bill
Heating fuel
...
- 171
1
vote
1
answer
106
views
Fatality Analysis Reporting System
The National Highway Traffic Safety Administration has a database of fatal crashes called Fatality Analysis Reporting System (FARS) http://www.nhtsa.gov/FARS One can download each years data going ...
- 113
8
votes
1
answer
89
views
Which namespace should be used for an open vocabulary published by a private organization?
If you want to contribute a Vocabulary to the LOD community but work for a (commercial) company, the question arises within which (DNS) domain you define the vocabulary/ ontology namespace URI/ IRI?
...
- 409
5
votes
3
answers
3k
views
What are good examples of open data dashboards?
What are good examples of open data dashboards?
This can either be:
A dashboard that uses open data, for example the London City Dashboard (but better...)
A collection of metrics about the open ...
- 1,458
13
votes
4
answers
6k
views
A web API user's guide for free and open data
There are many web-based Application Programming Interfaces (API) that allow data-hungry people to get their hands on all sorts of data. It's a great way to collect data as-needed, instead of looking ...
- 17.7k
9
votes
4
answers
3k
views
Tool to extract the main concepts/topics from web pages
I need to find the main themes or topics for each of 50,000 website urls.
For example, http://austinzencenter.org/ should return topics such as zen, monastic, austin, retreat, buddha, etc.
I'm ...
- 289
7
votes
1
answer
3k
views
Techniques for Pulling Prices for a Large Number of Amazon Items
Lately I have been playing around with the Amazon Product Advertising API, and it made me very curious about how Amazon-price checking sites like Keepa and camelcamelcamel actually work.
For a ...
- 173
6
votes
1
answer
128
views
Is there any documentation or case study of scaling up an Open Data API?
Forecasting open data demand is quite challenging. A new public dataset means more downloads, a new app results in way more requests.
It would be interesting to discuss the throughput of open data ...
- 221
5
votes
1
answer
62
views
Involvement of central statistics bureau with national open data portal
Although this is a very subjective question as each national government has quite a different setup. However, in general, are there any patterns or working models involving statistics bureaus having a ...
- 311
7
votes
3
answers
228
views
What are examples of people sharing personal data as open data?
Personal data usually has to be anonymised, e.g. aggregated, to be published as open data.
However, some individuals are willing to share some of their personal data for example to contribute to ...
- 1,458
4
votes
1
answer
133
views
Captcha in a website with open data
I don't have an example to showcase, but I have a general question. If a website provides its data open and free, but to download them you need to fill a captcha, can I use an automatic way to auto ...
- 315
5
votes
2
answers
157
views
Optimal Data Format(s) for Open GIS Data Repositories?
What format(s) would/should be provided/included with a gis dataset repository?
I know that there is no correct answer, i'd like as many different answers/opinions as possible to make the best ...
- 11.9k
6
votes
6
answers
344
views
What type of software is used by the Open Data community?
I am a newcomer to the world of Open Data and I would like to know what type of tools/software one could be confronted with. Not for a specific task, but mostly in general. For example:
Which tools ...
- 161
53
votes
6
answers
18k
views
A Python guide for open data file formats
If you are an open data researcher you will need to handle a lot of different file formats from datasets. Sadly, most of the time, you don’t have the opportunity to choose which file format is the ...
- 4,714
5
votes
2
answers
228
views
Source of open trend data
I'm currently querying Twitter to access trend data.
I'm looking for an open alternative to source trend information.
The main question is:
What's trending today?
I'd love to move away from using ...
- 151
4
votes
2
answers
146
views
Examples of metadata for non-uniform collections
Are there examples of
metadata —
lists of what a site offers, that users search to help them decide what to download —
for non-uniform collections like
UCI ml
or openml.org ?
Metadata is ...
- 498
5
votes
2
answers
975
views
Load data from HTML tables into OpenRefine?
One answer to the question Wikipedia table to JSON (or other machine-readable format) suggested to use OpenRefine for handling HTML tables.
All I could find regarding OpenRefine and HTML tables were ...
- 5,830
9
votes
2
answers
1k
views
Anyone have a good way of comparing two large and unstructured lists (~2k entries each) for commonalities between them?
Let's say, two governments have separate lists of reports, and I'm trying to figure out the most common elements between them. Think of one government administrator reporting # of sick days per year ...
- 427
14
votes
7
answers
1k
views
Should data APIs require registration and API keys?
When publishing a data API, there seems to be a conflict between requiring a user to sign-up for an API key and the principal of open access. What is best practice in publishing an open data API ...
- 2,371
40
votes
6
answers
1k
views
Let's suppose I have potentially interesting data. How to distribute?
This is a very simple question:
Suppose that I have some sort of specialized data, perhaps that I've collected myself or been a part of the collection. And suppose that nothing prevents me from ...
- 501
4
votes
2
answers
103
views
Are there good examples of requiring open data in RFPs?
The recent U.S. Executive Order on open and machine-readable data requires the U.S. government to begin developing ways to integrate open data into procurement and grant-making.
Are there good ...
- 345
13
votes
3
answers
267
views
Preservation of blog posts, articles and essays
I am curious if there is there any good way to guarantee that my articles (posted on personal sites and other blogs) will last? Is there a reliable repository for articles, post and essays? Or are ...
- 824
7
votes
4
answers
208
views
Best practices for huge explorable linked data directories
The main requirements for our open data directory are:
XML-based by default with ability to switch to JSON. It's important the easy way to make it human readable with just linking to XSL.
All the ...
- 171
19
votes
7
answers
1k
views
Keeping track of updates to open data published as CSV
I looked through several related questions here (e.g. Tools for merging similar datasets continuously) but nothing seemed to quite answer my question. So, apologies in advance if I missed something.
...
- 291
15
votes
4
answers
807
views
How to publish real-time open data
I'd want to publish some "real-time" data, i.e. my FV panel energy production, on a server. Which is the best way of cataloguing the data and make it available?
Energy is just an example, say that I ...
- 251