In openFDA's source code documentation it's said that the FAERS pipeline includes a procedure for de-duping follow-up reports. I don't find in source code where it happens.

What it means? Only last follow-up of each case is kept or what?



This used to be handled within the source code. It is now handled by processing the quarterly reports sequentially and letting Elasticsearch handle the versioning. A basic last-one-wins approach. The openfda team added a feature to the pipelines for incremental indexing that allows this to happen without too much of a performance hit.

I will have the documentation updated to reflect this change. Thanks!

| improve this answer | |
  • But nothing is done to detect and remove eventual duplicated cases with different case ID? (i.e., if two cases, with different case ID, represent the same "conceptual" case, they are not "merged"? ) – Gabriele Jun 8 '16 at 11:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.