0
  • Currently I am downloading the entire data set as CSV from [2].
  • Ex. Inspection extracts to 5 CSVs totaling to 1.5 GB, and Violation extracts 14 files totaling 2 GB.
  • Looking for a way to only load the updated and new records since the last data set (i.e. incremental load) and not have to download the entire data set.

I compared the 2 data sets for Inspections and Violations (Ex. osha_inspection_20210222.csv vs osha_inspection_20210304.csv) based on the primary keys (combination of columns activity_nr + citation_id), and find:

  • For inspections, for majority of the mismatched records I see the column values for case_mod_date and ld_dt differ, but otherwise the rest of the record is identical. Ideally the system would re-load changed records, but I don't see any changes... So not sure why these date columns are been updated?
  • For violations, for the majority of the mismatched records the column load_dt differs for some records, but once again there is no change in the rest of the columns.

-> Ideally I would like to skip having to reload these records that are otherwise identical.

-> Apart from this, the other mismatching records have some genuine changes in the column values. I can identify and fetch these using the above date columns, via the API.

-> I noticed that there is no way to identify new records. As at times some new records are added in between existing activity_nr's. So identifying a new record is not as simple as looking for records with activity_nr >= last loaded activity_nr .

Anyone having more insight on this data set, kindly advise. Thanks


References

  1. "DOL HEALTH & SAFETY DATASETS" > "DOL OSHA ENFORCEMENT V2" > "Inspection Table"
  2. OSHA Downloads (CSV/XML/API) : Visit link -> Click on "OSHA Data"

Notes

  • Sample Python code to connect to this resource (just an FYI for other developers)
import requests
url = "https://data.dol.gov/get/inspection/limit/2"
header={"X-API-KEY" : 'XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX'} 
# Visit https://devtools.dol.gov/developer > "API Keys" to get your own
r = requests.get(url, header)
j = r.json()
print(str(j))
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.