1

I am trying to do some classification and clustering on the openFDA dataset. I started by downloading a year's worth of data and realized I run into system limits and would need to solve for that.
Then I decided to query for just one drug indication, the subset I am looking at has about 422k records. But I realized that I hit the skip limit and cannot get the data through the API.

I then decided to run the api locally so as to push through the limits, but I am unable to find clear instructions on how to do it. I have installed node, elasticsearch, and run bootstrap and npm install? I believe I need to run the pipeline, do I just run all-pipelines.sh?

Improve this question

1 Answer 1

Reset to default
1

If you need that much data, I'd recommend downloading JSON export files from the Downloads section of a respective endpoint and then doing your data mining on them. Running the pipelines would be more challenging in your case.

Improve this answer
1
  • I was able to find a solution. I am iterating through day ranges at a time, so I am able to push through the max skip limit. I initially started with downloaded files, but they are huge and hits system limits. I have installed ijson, so that could help if I need to go back to downloaded data for other aspects.
    – h.das
    Nov 30, 2016 at 17:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.