2

I get error when attempting to use count=patient.drug.openfda.manufacturer_name.exact without including the filter by field like in search=patient.drug.openfda.pharm_class_epc:"nonsteroidal+anti-inflammatory+drug . And, even if I add a filter by field, the api only returns the top 100 manufacturers.

When I add the limit keyword to get more than 100, I get up to the first 1000 (&limit=1000&skip=0), but then refuses to support the search for the second page of 1000 (&limit=1000&skip=1000). If says that the skip term is not supported in count queries.

Improve this question
2
  • I'm running the example queries on the openFDA site and trying different limits and skips. It is working with small numbers. On larger numbers it looks like the FDA server is timing out (overloaded) - thus the cause of the error messages.
    – Andrew - OpenGeoCode
    Commented Jun 4, 2014 at 17:17
  • is there any reason you're messing with the api when you could just download the whole data set? asdfree.com/search/label/…
    – Anthony Damico
    Commented Mar 27, 2015 at 11:18

2 Answers 2

Reset to default
2

Direct count queries are supported: e.g.

https://api.fda.gov/drug/event.json?count=patient.drug.openfda.manufacturer_name.exact

works without an accompanying search.

I can confirm that it's not returning the full list of manufacturers. The 1000 entry cap is to avoid the backend server overloading, but it should only be relevant for non-count queries: count queries are supposed to return all of the results by default, which is why skip is disabled for them.

Thanks for reporting this -- I'll file an issue to figure out what's going on.

Improve this answer
1
  • Thanks. I can confirm that your query works. Obviously I was doing something wrong. However, it only returns 100 entries. When I add the limit, I can get up to 1000, but nothing more.
    – erflores
    Commented Jun 5, 2014 at 13:55
2

Thanks for your feedback on this. We have two different limits at play here:

For search queries, we support up to 100 records being returned at a time (using limit). You can then paginate through this with skip.

For count queries, we support returning up to 1000 results. You cannot paginate through count queries with skip. This is due to the nature of Elasticsearch (the backend for openFDA). Running that long tail of counts is extremely expensive, and getting records in that long tail tend not to be statistically significant anyway. The only exception for this is when you are counting a date field, in which case we will return everything we can find for you.

If you are having an issue with things being slow or crashing, we would love to see the exact queries you are running. Large returns (for instance, a limit of 1000) can return JSON so large that it can crash your browser.

Sean Herron, openFDA Team Member

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.