19

If some site claims to offer a definitive data source, how can I trust that it is not purposely biased? Even excluding elaborate conspiracies, how could I know that it isn't just a big hoax?

It seems unlikely that there's any central registry of these sites but it'd be nice to have some sort of list of bad sites (assuming there are any). Maybe like the lists of spam hosts.

So, what criteria could I use to determine if a source is trustable?

(In the most general sense, I'm hoping to safeguard against any purposely distorted data. In particular, I'm interested in biological data but this could apply to almost anything that's hard to verify. I should be able to trust well established sites like the FDA but what do I know about some small research group? Of course, it's probably fine but it'd be nice to have some "peer review".)

Improve this question
4
  • What kind of sites are you looking at? What kind of data? Without that kind of information it's hard to answer your question.
    – RCA
    Commented May 8, 2013 at 20:44
  • In the most general sense, I'm hoping to safeguard against any purposely distorted data. In particular, I'm interested in biological data but this could apply to almost anything that's hard to verify. I should be able to trust well established sites like the FDA but what do I know about some small research group? Of course, it's probably fine but it'd be nice to have some "peer review".
    – igelkott
    Commented May 9, 2013 at 16:25
  • 1
    Isn't that a question for Skeptics SE?
    – Deer Hunter
    Commented May 22, 2013 at 13:08
  • 1
    @DeerHunter : No, the question would be off-topic on Skeptics SE as it's not about a specific notable claim.
    – Christian
    Commented Nov 19, 2013 at 0:21

8 Answers 8

Reset to default
10

There are attempts at creating a registry of open data sites, see ckan's datahub http://datahub.io/. I'm not aware of any effort to list bad sites, but I've not come across any instances of deliberately false open data.

Improve this answer
12

Authenticity and bias are two different concepts. An authentic source would be a government website some other regulated institute, or body of knowledge.

Bias is subjective and you should use your best judgment in determining whether your data has bias. Bias may also refer to the way some data is calculated, such as in statistics. However, it depends on what type of data you are working with.

Improve this answer
4

You need to define terms here and then evaluate clearly.

  1. Authenticity. Do you mean the data is what it claims? That the sources are verified as such? This ought to be reasonably easy to verify.

  2. Bias. Ok, what do you mean here? Do you mean that the data is a representative sample? Representative in what way? Representative of what? Obviously this depends on your needs.

These need to be looked at separately. In general, you can address the first by looking at where the data came from and how trustworthy the source is. If they list contributors, you can follow up with them.

The second is much harder. You need to look at the following, once you are comfortable with the data source:

  1. What was the purpose the data was collected? The further you get from this purpose, the less likely you have a representative sample for your purposes. For example, if you are looking at prevalence of sickle cell anemia among sub-saharan Africans and that is a representative sample, it is not a representative sample of sickle cell anemia rates world-wide.

  2. How big is the sample?

  3. What conclusions can you draw from the sample?

Note that bias depends as much on what you want to do with the data as it does the data itself. Data may be entirely unbiased regarding one usage and entirely biased regarding another.

Improve this answer
4

If you're going to have to use data from "small research groups" then you don't have a ton of choice it seems but to trust the sources. Due diligence to see the perceptions of that organization- are they politically leaning in a big way or a trusted source of information? Is their research/data work generally respected and trusted?

Beyond that you have to trust and have faith- but small shops and large agencies make mistakes and also introduce bias in the entire process of data collection- often as early as in the policies that prescribe how and what to collect.

Disclaimer- we're a small social research org that publishes a lot of data- often from aggregates raw records from local government that can't be published directly- makes it hard for people to test our sources, and some people won't trust us, but most do. Track record counts.

Improve this answer
4

Cross-correlating your data! In other words: Does it "jibe" with other sources with similar data?

Otherwise, without collecting the data yourself or validating its methodology, there is no other way besides resting on the data provenance.

Improve this answer
3

I'm not sure how useful it would be to do this but if you were able to trace back to find a copy at an authoritative site, you could use an MD5 hash of the two datasets to see if they are identical. Again, I'm not sure whether this defeats the purpose since you could then just use the original source file.

(Disclaimer - I am the Sr. API Strategist for GSA)

Improve this answer
1
  • I second this. Especially for today's world, internet security and computer security are very important. Just one note, tho, MD5 is not the only hash algorithm. Sha-2 is another. There are probably hundreds and judging which one is the most secure is not easy.
    – DrZ214
    Commented May 1, 2018 at 3:53
3

For any complicated data set the decision is more complicated than whether you trust or don't trust it.

You actually have to understand the data that you are dealing with. Even authoritative biological databases contain errors.

If you are for example looking at protein data there's Swiss-Prot and Trembl. Swiss-Prot contains data that's reviewed by human while Trembl also contains data that are the output of algorithms. As a result there are more errors in Trembl. But Trembl has also more data. Which database to use depends entirely on your goals.

Improve this answer
1

There are some things that make some data sources more reliable than others.

Look for provenance, or where do the data come from?

  • They come from a governmental institution where they were originally created
  • They come from a research group that originally made the data set
  • They come from a trusted repository for Open Data (e.g., a CLARIN centre) that has been externally reviewed (e.g., having a Core Trust Seal)
  • They are published alongside a peer-reviewed publication with a reliable electronic journal

Also look for metadata or how well are the data described?

  • They come with responsible authors from a reliable institution
  • The description is consistent with the data given
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.