In general, I want to use OpenFDA to look at the overlap between FDA drug data and cosmetics ingredients, primarily using the drug event and drug label data sets. Device and food questions might arise for me, but first things first.
I think I am that rare (not primary audience?) use case where my kinds of queries exceed use limits even with an access key; for example, when I am looking for how a given UNII appears in all records some result counts far exceed 5000.
I started to process zip files, then started looking into running my own server to remove the use limits. I have a virtual machine (vagrant/VirtualBox, ubuntu, and elasticsearch) set up, and the openfda github code base (I also ran bootstrap rather on a whim/lark and can't tell to what effect, embarrassingly enough).
Processing the fda data (maybe in zip files maybe not?) and inserting the results in elasticsearch seems to be part of the purpose of the opendfda github code base (deduping etc.), so I am very much rethinking trying to do that myself. Yet I am stuck on how to proceed.
Questions: 1. Do I need/want both the front and and backend github code if I am just trying to analyze the records that come back? (Doubtless, the answer takes some form of "It depends..." but I'm not sure even on what issues the decision/answer may rely.)
- If I am on something close to the right track with the virtualmachine, what is the best way to set up my own server with drug event and drug label data into elasticsearch....so I can get back out responses to my queries? A mid-level (not 80,000-foot view and not a 1,000-foot bird's eye view) set of general steps to my goal would be awesome, and then I could struggle through the more detailed "how" parts on my own time is an ideal answer for me -- tho arguably still very broad.
I feel I've been through youtube (nothing to see, I think), SE and the FDA site and the github code README files, but I am still not getting a lot of ideas from those documents to suggest what I'm about is ... possible (or good or right) for me, even with a little help. Maybe the human touch (or 2x4) here will help. TY.