3

I have been working to build a download and unzipping "pipeline" going (as a noob) with OpenFDA data, and I saw with interest @h.das' closed post about zips - "Python crashing when trying to read zipped json files", but this issue to me seems more like an issue with the source file than my noob status (tho of course I'm out a limb there) or the lack of RAM (I threw a TON of RAM at this, btw).

Programmatically with python 2.7's zipfile library and on the plain ol' bash cli on a ubuntu machine as well as through chrome on a win10 laptop I get (just now 2017-03-17 9am ish) a failed file error of one kind or another with the following URL: https://download.open.fda.gov/drug/event/2011q4/drug-event-0006-of-0007.json.zip

It's easy enough to pass over with exception handling but I don't want to miss some 300+MB of data in the middle of what seems like exactly what I want to look at.

Improve this question
8
  • on os x/firefox, i can download it by itself. and open it up. its 1.5gb json file. could there be file size limitations/restrictions in what you are doing?
    – albert
    Commented Mar 17, 2017 at 13:33
  • 1
    a failed file error of one kind or another is not an error message. If you want help, you have to be specific. And the 1.503.379.837 byte JSON file in it unpacks just fine on Windows.
    – user4293
    Commented Mar 17, 2017 at 13:48
  • 1
    Thanks for the responses, and I'm happy to be as specific as I need to be. I will look into this being a network limitation or some setting somewhere -- if others are actually able to download and work with the json, then it seems unlikely it's a bad file -- so that helps...I think. As for being more specific on the error messages, I get a network error on chrome in win10 and a badzip file error type through the exception construct in the python script I wrote -- does not seem like a need here* to get into those specifics. I'll look for filesize "switches" configs from other sources. Thanks
    – ksf
    Commented Mar 17, 2017 at 13:59
  • Hi @ksf, this is Jack from the openFDA team. I'm not able to confirm any issue with the file and was able to extract it without issue. As others have mentioned, it is a large file, which may be causing some problems. Let me know if you need any more assistance dealing with the openFDA API and data.
    – Jack Finch
    Commented Mar 18, 2017 at 21:31
  • Thanks, @Jack Finch, really appreciate it--and the gang. I'm looking into the file size thing . It looks like the reported error messages (esp the python exception) were sending me in the wrong direction. I'll shout if I have something. Best ...
    – ksf
    Commented Mar 21, 2017 at 13:04

1 Answer 1

Reset to default
1

Hi this is Jack from the openFDA team. I'm not able to confirm any issue with the file and was able to extract it without issue. As others have mentioned, it is a large file, which may be causing some problems. Let me know if you need any more assistance dealing with the openFDA API and data.

--Reposted as an answer for posterity--

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.