7

Assume that a person moves from one address to another. They update their driver's license, which means the DMV has the correct address. However the Town Clerk has the previous address. Both the DMV and The Town Clerk publicly expose their data through api's.

How should a consumer of both the DMV's and The Town Clerk's data inform The Town Clerk that their data is outdated? Are there any standards or ways to automate such requests?

Improve this question
8
  • This question seems off-topic. It would be more appropriate for a mathematical or statistical site. Relevant
    – Kermit
    May 9, 2013 at 15:34
  • @FreshPrinceOfSO I can see what you mean regarding addressing disparities, but I think the suggesting corrections/modifications would be on topic for open data.
    – Ryan Gates
    May 9, 2013 at 15:40
  • 4
    This was closed as off-topic, but now it is on vote to be brought in. I think if you add very specific example (e.g. define A, B and Z to some real entities) it will be legit, focused question.
    – Dmitry Kachaev
    May 9, 2013 at 18:08
  • 1
    I agree with @DmitryKachaev, the question is already much better, but still too general: Who are the data publishers? Governmental institutions in the US? Research projects in the EU? What systems do they use for publishing the data? These are all factors that might heavily influence the feedback process.
    – Patrick Hoefler
    May 9, 2013 at 20:43
  • 1
    @PatrickHoefler I agree too, I still don't completely understand the question even with the edits. With some context around A, B, Z I think the question is fit for reopening. May be expand your example with driver's licenses and addresses by including DMV/Town Clerk as the entities that are interacting.
    – CaesiumFarmer
    May 13, 2013 at 23:27

3 Answers 3

Reset to default
4

I will answer this in the more general term (leaving the consumer out). Invariably when data is aggregated and coalesced from plural data sources you will have conflicts in records. Just think of all the wacky occupants you receive in your mailbox or the wacky places you supposedly lived when you do a FREE background search.

These data sources need to periodically perform a validation/reconciliation process on the aggregated/coalesced dataset. Typically, the process involves:

  1. A primary table which holds the 'best values' for a record.
  2. A secondary table which uses the primary key in the first table as a foreign key, which holds the conflicting values of the first table.
  3. A process which consists of a:
    A. Validation - eliminates secondary records deemed to be invalid.
    B. Reconciliation - chooses the values that are 'best' at the moment.
  4. The process is repeated periodically (or on demand) on records that are updated or new records are added.

Note: The not-best-value records in the secondary table, until deemed invalid, are retained, so that they can be reapplied when other records are updated or added.

Improve this answer
2

I think the best approach is to start with a small random sample of conflicts and investigate them if this is possible. If you can understand the type of error (out of date data, for example) then you can make an educated decision regarding which one to trust.

If that is not possible and you can get a tiebreaker, then go with that. If this is not possible you will need to start looking at it from a statistical viewpoint. However this should be a last resort because type of error should be more of a factor than anything else.

Improve this answer
1

A tool for merging conflicts and feeding corrections back - how about git? See my similar problem here: Tools for merging similar datasets continuously

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.