Setting up a private instance

Question

In general, I want to use OpenFDA to look at the overlap between FDA drug data and cosmetics ingredients, primarily using the drug event and drug label data sets. Device and food questions might arise for me, but first things first.

I think I am that rare (not primary audience?) use case where my kinds of queries exceed use limits even with an access key; for example, when I am looking for how a given UNII appears in all records some result counts far exceed 5000.

I started to process zip files, then started looking into running my own server to remove the use limits. I have a virtual machine (vagrant/VirtualBox, ubuntu, and elasticsearch) set up, and the openfda github code base (I also ran bootstrap rather on a whim/lark and can't tell to what effect, embarrassingly enough).

Processing the fda data (maybe in zip files maybe not?) and inserting the results in elasticsearch seems to be part of the purpose of the opendfda github code base (deduping etc.), so I am very much rethinking trying to do that myself. Yet I am stuck on how to proceed.

Questions: 1. Do I need/want both the front and and backend github code if I am just trying to analyze the records that come back? (Doubtless, the answer takes some form of "It depends..." but I'm not sure even on what issues the decision/answer may rely.)

2. If I am on something close to the right track with the virtualmachine, what is the best way to set up my own server with drug event and drug label data into elasticsearch....so I can get back out responses to my queries? A mid-level (not 80,000-foot view and not a 1,000-foot bird's eye view) set of general steps to my goal would be awesome, and then I could struggle through the more detailed "how" parts on my own time is an ideal answer for me -- tho arguably still very broad.

I feel I've been through youtube (nothing to see, I think), SE and the FDA site and the github code README files, but I am still not getting a lot of ideas from those documents to suggest what I'm about is ... possible (or good or right) for me, even with a little help. Maybe the human touch (or 2x4) here will help. TY.

Answer 1

I suppose it depends on how much re-use you are looking to get. Drug event, run from scratch, takes roughly 24 hours to run and needs around 80GB of space. The initial drug label run can take a long time as well (I want to say 15 hours). Also, the raw data pull for SPL from the s3 bucket takes a long time and takes up a lot of space (around 300k labels in total). Once they are up and running, the incremental reloads for each are under an hour.

The reason I mention these time estimates is that getting the downloads (once) and spinning through them each question you have doesn't take that long. Downloading is a bandwidth thing, so it varies. Looping over the zips for drug event takes about an hour.

Here is an example of looping over the drug event data to get a histogram: https://gist.github.com/HansNelsen/aeec93279dcd1792855d39fc37bead2e

The main advantage of this approach is that you never have to unzip the files completely, so you don't need to have a ton of disk space. This example is just generating a histogram based upon drug name and characterization, but you can make those any fields that you like.

For me, it would be a simple trade off between the number of hours I want to invest in the setup of the pipeline versus the number of queries I have and how often I want to refresh the answers. If I only had a few questions, I would try to answer them all in a single pass and generate a handful of aggregate files from the downloaded data.

Also, it is not obvious from the documentation, but the downloads are indexed under: https://open.fda.gov/download.json, so you can loop over the ones that you want, instead of scraping the https://open.fda.gov site.

Hope that helps.