Getting Started with the OpenFDA API

Question

I am totally lost with the OpenFDA API. I've downloaded the API from Github, run the bootstrap.sh, run python setup.py install, and I have no clue what to do now.

My specific questions:

1. Downloading data

The API Reference page mentions a way to write code to download the data automatically, and then it gives me a nice bit of "download API query" code without every telling me how to use this code from my machine (this means: to download the dataset, not as a query to retrieve database information using search or connect. This was already covered in excellent detail and I understand it quite clearly).

2. Running the API Locally

When I navigate to api/faers/ and open the README.md file, the first instruction tells me to "Get an ES instance running locally with data in the FAERS mapping format."

Um, how do I do that?

If I proceed with the other steps, I get connection refused errors on step 4.

3. Querying from my machine

Similarly to question #1, I'm confused about how to query the database from my machine and especially how this relates to the API key. Can someone give me some example code or point to a tutorial on this---I can't seem to find any.

As a final note, I am not interested in any links to sites you have made that have already done this for me; I am only asking for help with setting it up for myself.

Thank you!

Answer 1

1. Downloading data: there is an endpoint: https://api.fda.gov/download.json. The response is a list of all of the downloads. It is meant to be used in lieu of scraping the page for the links. Once you have the paths, you can use something like python requests library to grab the data.
2. Running the API locally: you will need to have node installed, run npm install from the ./api/faers directory, have elasticsearch running locally (or port forward localhost:9200 to a running instance), finally, run node api.js from within the ./api/faers directory. This presumes you have run the pipeline and loaded an elasticsearch instance. This is why I asked to see what exactly you need, because /drug/event.json and /device/event.json take awhile to run. The rest run in short order, so running them on a laptop is not a big deal.
3. Querying: The examples on the documentation site show you everything you need to know. https://open.fda.gov/drug/event/ has the example: https://api.fda.gov/drug/event.json?search=receivedate:[20040101+TO+20160601]&count=receivedate. To call this with an api key, just add that on as a parameter &api_key=xxxxxxxxxx. If you are running the api locally, just replace the https://api.fda.gov/ with http://localhost:8000/. Everything you need to know about querying the API is here: https://open.fda.gov/api/

A python example for calling the api:

import requests

URL = 'https://api.fda.gov/drug/event.json?search=receivedate:[20040101+TO+20160601]&count=receivedate'

data = requests.get(URL).json()

for result in data.get('results', []):
  print result

There are limits to what you can do with the API. I would suggest seeing if the one that is maintained by the FDA works for you, since it will save you a lot of time and money. If you need more than 5000 on a skip and limit loop, then you can either use the downloads to get all the API output as zips (question #1) or you can standup your own server and remove the limits (question #2).