Extracting data from OpenFDA using python to save it locally

Question

I have been looking to extract all data from FDA so I can use it all.

1. I created this code below that works but seems it is too much data it fails. This way it just gives me 25000 records.

url = 'https://api.fda.gov/drug/ndc.json?sort=dea_schedule:desc&limit={}&skip={}'

all_data_df = []

limit = 100
for skip in range(0, total, limit):
    query = url.format( limit, skip)
    print('query:', query)
    data = pd.read_json(query, orient='values', typ='series', convert_dates=False)
    data = data['results']
    all_data_df = pd.DataFrame(all_data_df.append(data))

I need all data. FDA site says to use paging here the link https://open.fda.gov/apis/paging/ This below will give me the next page that they mention but I am not sure how to do put it together. Any help is appreciated

re = requests.get('https://api.fda.gov/drug/ndc.json?sort=dea_schedule:desc&limit=100')
url =re.links['next']['url']

Answer 1

It appears your best option would be to use the downloadable JSON files: https://open.fda.gov/data/downloads/