2

I downloaded all of the JSON files from openFDA (2004Q1-2015Q4) so I could run my own python scripts to mimic the count query provided by the API. My goal is to get the counts of adverse reactions with drugs, without the prescribed limit of 1000 online. I want my JSON file to have results like this query (fatigue as an example reaction):

https://api.fda.gov/drug/event.json?search=patient.reaction.reactionmeddrapt:fatigue&count=patient.drug.medicinalproduct.exact

Here is my code now (based on this example https://gist.github.com/HansNelsen/aeec93279dcd1792855d39fc37bead2e):

#!/usr/bin/env python

''' Simple example of reading all of the zip files from the openFDA download
    and doing something with them. In this case, we are building an index
    of all the medicinalproduct and drugcharacterization values and a count
    of how often each occurred.
'''

from collections import defaultdict
import glob
import simplejson as json
from os.path import basename
import re

DATA_FILES = glob.glob('./*/*.json')

reaction_drug_counts = {}

for filename in DATA_FILES:

    print(filename)

    file = open(filename)
    json_data = json.load(file)
    file.close()

    for row in json_data['results']:

        for reaction in row.get('patient', {}).get('reaction', []):

            MedDRA_term = reaction.get('reactionmeddrapt','unknown')

            if MedDRA_term not in reaction_drug_counts:

                reaction_drug_counts[MedDRA_term] = {}

            for drug in row.get('patient', {}).get('drug', []):

                characterization = int(drug.get('drugcharacterization', '0'))

                # if characterization == 1:
                # additional restriction to only collect counts of "suspect" drugs

                product = drug.get('medicinalproduct', 'unknown')

                if not product or re.search("^[\W_]+$", product):

                    continue

                current_counts = reaction_drug_counts[MedDRA_term]

                if product not in current_counts:

                    current_counts[product] = 1

                else:

                    current_counts[product] += 1


f = open("my2.json","w")
json.dump(reaction_drug_counts,f, indent = 4, sort_keys=True)

This should result in a JSON file that contains the results if I performed a count query on every adverse reaction in FAERS. The problem I'm getting, however, is that it's not matching to the original query, specifically when I look for "FATIGUE", the drug "TYSABRI" should be getting 15214 unique counts, but my script only shows 9554.

Originally, I had a restriction where it would only count "suspect" drugs (given by a drugcharacterization value of 1), and resulted in 9529 counts for "FATIGUE" and "TYSABRI". I assumed that it was because of this original restriction, but indeed, after taking it out, it still didn't match the API results (In fact, it barely increased).

Is there something I'm missing in my code? Or could it be that the API has connections between drugs (such as counting "NATALIZUMAB" reports in the "TYSABRI" count)?

EDIT: What ended up happening is that the dictionaries were case sensitive, so I added a .upper() for the terms to be saved. In addition, it was possible that a certain drug/event could be found in the same safety report, so I implemented it so that it could be added to the count only once. The results are nearly identical, and if anyone wants the updated version, just let me know.

1 Answer 1

2

I would check to make sure that you are processing all of the files. Start by printing out filename variable or len(DATA_FILES). It may be the case that you are missing some downloads.

Also, are you using .exact queries on your API search? I ask, because what you are doing in the code would be the same as counting and search with the .exact suffix.

As a side note, the API counts documents not instances of a term, so if a term is on document more than once, then that could explain the difference.

1
  • Yes! Actually I have found several errors, and I've achieved almost identical results now. What happened was my dictionary considered "Fatigue" and "FATIGUE" as two different keys, and the identical counts you talked about for the document. Thanks!
    – kdavid2
    Jul 7, 2016 at 1:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.