7

Assume that a person moves from one address to another. They update their driver's license, which means the DMV has the correct address. However the Town Clerk has the previous address. Both the DMV and The Town Clerk publicly expose their data through api's.

How should a consumer of both the DMV's and The Town Clerk's data inform The Town Clerk that their data is outdated? Are there any standards or ways to automate such requests?

  • This question seems off-topic. It would be more appropriate for a mathematical or statistical site. Relevant – Kermit May 9 '13 at 15:34
  • @FreshPrinceOfSO I can see what you mean regarding addressing disparities, but I think the suggesting corrections/modifications would be on topic for open data. – Ryan Gates May 9 '13 at 15:40
  • 4
    This was closed as off-topic, but now it is on vote to be brought in. I think if you add very specific example (e.g. define A, B and Z to some real entities) it will be legit, focused question. – Dmitry Kachaev May 9 '13 at 18:08
  • 1
    I agree with @DmitryKachaev, the question is already much better, but still too general: Who are the data publishers? Governmental institutions in the US? Research projects in the EU? What systems do they use for publishing the data? These are all factors that might heavily influence the feedback process. – Patrick Hoefler May 9 '13 at 20:43
  • 1
    @PatrickHoefler I agree too, I still don't completely understand the question even with the edits. With some context around A, B, Z I think the question is fit for reopening. May be expand your example with driver's licenses and addresses by including DMV/Town Clerk as the entities that are interacting. – CaesiumFarmer May 13 '13 at 23:27
2

I think the best approach is to start with a small random sample of conflicts and investigate them if this is possible. If you can understand the type of error (out of date data, for example) then you can make an educated decision regarding which one to trust.

If that is not possible and you can get a tiebreaker, then go with that. If this is not possible you will need to start looking at it from a statistical viewpoint. However this should be a last resort because type of error should be more of a factor than anything else.

1

A tool for merging conflicts and feeding corrections back - how about git? See my similar problem here: Tools for merging similar datasets continuously

4

I will answer this in the more general term (leaving the consumer out). Invariably when data is aggregated and coalesced from plural data sources you will have conflicts in records. Just think of all the wacky occupants you receive in your mailbox or the wacky places you supposedly lived when you do a FREE background search.

These data sources need to periodically perform a validation/reconciliation process on the aggregated/coalesced dataset. Typically, the process involves:

  1. A primary table which holds the 'best values' for a record.
  2. A secondary table which uses the primary key in the first table as a foreign key, which holds the conflicting values of the first table.
  3. A process which consists of a:
    A. Validation - eliminates secondary records deemed to be invalid.
    B. Reconciliation - chooses the values that are 'best' at the moment.
  4. The process is repeated periodically (or on demand) on records that are updated or new records are added.

Note: The not-best-value records in the secondary table, until deemed invalid, are retained, so that they can be reapplied when other records are updated or added.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.