I am trying to do some classification and clustering on the openFDA dataset. I started by downloading a year's worth of data and realized I run into system limits and would need to solve for that.
Then I decided to query for just one drug indication, the subset I am looking at has about 422k records. But I realized that I hit the skip limit and cannot get the data through the API.

I then decided to run the api locally so as to push through the limits, but I am unable to find clear instructions on how to do it. I have installed node, elasticsearch, and run bootstrap and npm install? I believe I need to run the pipeline, do I just run all-pipelines.sh?

If you need that much data, I'd recommend downloading JSON export files from the Downloads section of a respective endpoint and then doing your data mining on them. Running the pipelines would be more challenging in your case.

  • I was able to find a solution. I am iterating through day ranges at a time, so I am able to push through the max skip limit. I initially started with downloaded files, but they are huge and hits system limits. I have installed ijson, so that could help if I need to go back to downloaded data for other aspects. – h.das Nov 30 '16 at 17:42

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.