0

I'm finding that my ability to parse the JSON using Azure Data factory is not possible after file updates on July 07. The process, which took 10 minutes previously now takes 50+ hours.

Is anyone esle experiencing an issue with a certain download file or have similar problems with Azure Data Factory?

I believe I have isolated the problem to file content itself, as all other system parameters remain constant.

Was there any issue with the new submissions / updates to the dataset? This happens on 510k to a smaller extent - as there are very large arrays in the JSON.

Is there a way of identifying the largest arrays in the full UDI endpoint?

0

I'm Jack with the openFDA team. I want to confirm that we haven't changed anything in the UDI code which might account for this, so if there's a change in the data, it'd be upstream. We're going to do an analysis of the UDI data to look for any red flags like the very large arrays you mention, discrepancies, etc. Let me know if you see any examples of very large arrays or if your experience processing the data changes. Thank you for bringing this to our attention!

  • Hello Jack, Thank you for your response. A little further detail ; generally the Azure Data factory JSON file parsing definitions work very well. What they are doing is cross/applying the results portion of each file given a collection of JSON paths and writing them into SQL. For the most part this happens without a problem, but a good example of where it has issues is in the 510k results - where combinations of registration_number / fei_number are VERY large for a given 510k. 510k therefore would parse in about 3 hours, but registration_listing (similar size) about 20 minutes. – Zach Jul 19 at 17:12
  • But overall, I'm hoping to identify a collection of anomalously large JSON objects added july, or some that don't conform to the general structure? – Zach Jul 19 at 17:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.