What methods are used to prepare sensitive data for public access?

Question

I am a policy researcher looking for the best technical methods to sanitize data so it can be uploaded to data portals. I am not working with a specific dataset but looking to gain an understanding of the methods used and the tradeoffs between usability and risk of re-identification. Are there scripts, open source applications or services that can be automatically applied to large varieties of datasets?

Answer 1

The first step as you said is to define which types of entities need to be removed in the public version of the data set. One example of such definition in the United States is given in the Health Insurance Portability and Accountability Act (HIPAA) rules, which target medical texts.

According to HIPAA rules, the following information should be removed from the text:

(i) Names of patients and family members
(ii) Addresses and their components
(iii) Dates (month and day parts, unless the inclusion of the year part identities an individual to be older than 90 years old)
(iv) Explicit mention of ages over 89 years old
(v) Telephone and fax numbers
(vi) Social Security numbers
(vii) Medical record numbers
(viii) Health plan beneficiary numbers
(ix) Account numbers
(x) Certificate or license numbers
(xi) Vehicle identifiers and serial numbers
(xii) Device identifers and serial numbers
(xiii) Electronic mail addresses
(xiv) Web universal resource locators (URLs)
(xv) Internet protocol (IP) addresses
(xvi) Biometric identifiers
(xvii) Full face photographic images
(xviii) Employers
(xix) Any other unique identifying number, characteristic or code

The list would partly differ for educational data, for example. I am not aware of any cross-domain standard.

As for a tool to enforce it, some ideas: Automated de-identification of free-text medical records. It is mostly a named-entity recognition task, any NER tool could fit. In terms of models, rules, SVM and CRFs are the most widely used. More recently, neural networks have been used for that task: https://arxiv.org/abs/1606.03475