I downloaded all of the JSON files from openFDA (2004Q1-2015Q4) so I could run my own python scripts to mimic the count query provided by the API. My goal is to get the counts of adverse reactions with drugs, without the prescribed limit of 1000 online. I want my JSON file to have results like this query (fatigue as an example reaction):
Here is my code now (based on this example https://gist.github.com/HansNelsen/aeec93279dcd1792855d39fc37bead2e):
#!/usr/bin/env python
''' Simple example of reading all of the zip files from the openFDA download
and doing something with them. In this case, we are building an index
of all the medicinalproduct and drugcharacterization values and a count
of how often each occurred.
'''
from collections import defaultdict
import glob
import simplejson as json
from os.path import basename
import re
DATA_FILES = glob.glob('./*/*.json')
reaction_drug_counts = {}
for filename in DATA_FILES:
print(filename)
file = open(filename)
json_data = json.load(file)
file.close()
for row in json_data['results']:
for reaction in row.get('patient', {}).get('reaction', []):
MedDRA_term = reaction.get('reactionmeddrapt','unknown')
if MedDRA_term not in reaction_drug_counts:
reaction_drug_counts[MedDRA_term] = {}
for drug in row.get('patient', {}).get('drug', []):
characterization = int(drug.get('drugcharacterization', '0'))
# if characterization == 1:
# additional restriction to only collect counts of "suspect" drugs
product = drug.get('medicinalproduct', 'unknown')
if not product or re.search("^[\W_]+$", product):
continue
current_counts = reaction_drug_counts[MedDRA_term]
if product not in current_counts:
current_counts[product] = 1
else:
current_counts[product] += 1
f = open("my2.json","w")
json.dump(reaction_drug_counts,f, indent = 4, sort_keys=True)
This should result in a JSON file that contains the results if I performed a count query on every adverse reaction in FAERS. The problem I'm getting, however, is that it's not matching to the original query, specifically when I look for "FATIGUE", the drug "TYSABRI" should be getting 15214 unique counts, but my script only shows 9554.
Originally, I had a restriction where it would only count "suspect" drugs (given by a drugcharacterization value of 1), and resulted in 9529 counts for "FATIGUE" and "TYSABRI". I assumed that it was because of this original restriction, but indeed, after taking it out, it still didn't match the API results (In fact, it barely increased).
Is there something I'm missing in my code? Or could it be that the API has connections between drugs (such as counting "NATALIZUMAB" reports in the "TYSABRI" count)?
EDIT: What ended up happening is that the dictionaries were case sensitive, so I added a .upper() for the terms to be saved. In addition, it was possible that a certain drug/event could be found in the same safety report, so I implemented it so that it could be added to the count only once. The results are nearly identical, and if anyone wants the updated version, just let me know.