Website in question: http://www-odi.nhtsa.dot.gov/ewr/qb/index.cfm
Automakers are required to report on a quarterly basis to NHTSA any claims brought against them that involve death, injury or property damage.
While NHTSA makes other databases open to the public in the form of large flat files (text) EWR reports are still segregated by automaker, by Report Type and by Quarter.
I'm trying to find the best tool (and tool operator) to use to scrape all this data from the website.
Currently, if you select an Manufacturer, Report Type and Report Period - then click on, in my case "light duty vehicle" you are taken to a new page where you have the option of downloading into a text file the results of that one inquiry.
With 26 automakers (listed below) and multiple types of reports and multiple quarters - doing this by hand will take an eternity.
Any thoughts on an easier, faster way to pull all of this data into a single file?
I was able, with the help of a friend, to come up with the following code that works for production data, but doesn't seem to work for Death and Injury.When I chane "prod" to "di" I get the correct "app" option value, but for some reason the script skips over the "yr_qtr" selection and loops back to the next manufacturerCode.