I have been looking to extract all data from FDA so I can use it all.

  1. I created this code below that works but seems it is too much data it fails. This way it just gives me 25000 records.
url = 'https://api.fda.gov/drug/ndc.json?sort=dea_schedule:desc&limit={}&skip={}'

all_data_df = []

limit = 100
for skip in range(0, total, limit):
    query = url.format( limit, skip)
    print('query:', query)
    data = pd.read_json(query, orient='values', typ='series', convert_dates=False)
    data = data['results']
    all_data_df = pd.DataFrame(all_data_df.append(data))

I need all data. FDA site says to use paging here the link https://open.fda.gov/apis/paging/ This below will give me the next page that they mention but I am not sure how to do put it together. Any help is appreciated

re = requests.get('https://api.fda.gov/drug/ndc.json?sort=dea_schedule:desc&limit=100')
url =re.links['next']['url']

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.