1

I was looking through the openFDA API and have been trying to find a good way to get data from the start of 2014 to now. Unfortunately it appears the downloads only go to 01-25-16. I also ran a query on the live database giving a very wide date range and got some strange results which do not appear to match up with that of the downloaded JSON file.

The download JSON file can be found here: "http://download.open.fda.gov/2016-01-25/food/enforcement/food-enforcement-0001-of-0001.json.zip" and my query run can be found here https://api.fda.gov/food/enforcement.json?search=report_date:[20140101%20TO%2020170429]&limit=100 . The problem I am finding is that the query seems to return far less results then I am expecting if it stores the same contents as the downloadable JSON.

What I mean by that is that the first entry for the query is with a report date of "20160316" and the last entry having a report date of "20140903". Where as the download has well over a 100 entries for any given year within its bounds. Is this an issue with my query, the database, a bug or something else entirely? Any help would be much appreciated.

Improve this question
2
  • why does it appear to only go to 2016-01-25? what have you tried/done? here's one prior download.open.fda.gov/2015-12-22/food/enforcement/…
    – albert
    Apr 29, 2017 at 19:36
  • I haven't done anything to the dataset, the date provided is just the last effective date that data is being put into the JSON. So for example the file you posted only goes up to 2015-12-22, where as the 2016-01-25 file should have everything in the file you posted plus more information. My problem is that this JSON does not appear to match data with a query on the OpenFDA dataset even though these JSONs are made from the same dataset. A query on the database seems to produce fewer results for a given time span compared to the download JSON object.
    – Zachary Mailhot
    May 1, 2017 at 13:31

1 Answer 1

Reset to default
1

I'm Jack with the openFDA team. Your query looks solid, but I think you're missing one vital piece of information to solve this issue: the skip parameter. We're limited in the number of records we can display at any given time, specifically limited to 100. You've already set the limit to 100 which is good, but it'll show the same 100 results every time with your current query. In order to view more than those, you use the skip parameter to skip the number of records you indicate.

Ex. To skip those first 100 records you've already seen, you'd use https://api.fda.gov/food/enforcement.json?search=report_date:[20140101%20TO%2020170429]&limit=100&skip=100

If you wanted to see another 100 records, use https://api.fda.gov/food/enforcement.json?search=report_date:[20140101%20TO%2020170429]&limit=100&skip=200

And so on. Your query found a total of 8995 records, so obviously looking through the records this way isn't the best if you're wanting to look at every single record unless you're going to write a script to parse the results and iterate the skip parameter by 100. Using the downloaded records is an alternative, or you can narrow your search by other factors which are relevant to the data you're searching for.

I hope that helps clear things up - let me know if you have additional questions.

-Jack

Improve this answer
1
  • Out of curiosity is there a way to sort the returns by the dates they were issued? so the first 100 would all have issued dates around 20140101, I mainly wanted it ordered for a sanity check before creating a local store of them
    – Zachary Mailhot
    May 1, 2017 at 22:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.