3

I'm a public health epidemiologist, and this question has come up in a number of contexts. I've heard tell of people using Google or Yelp for limited uses, but we'd be building state-wide datasets of certain retailers for public health questions.

As a specific example, a tobacco policy nonprofit I work with has been purchasing a business list, updated every year, at significant cost. They use their own coding to guess tobacco retailers from that generic business list (gas stations, pharmacies other than CVS, grocery stores, stores with "smoke" or "tobacco" in their names, etc.) for likely retailers. They then compute density measures, visit a sample of stores to validate them, do store surveys that inform policy, etc. But the same could apply to questions like parks, food deserts, etc.

Historically these datasets were prized and pricey possessions. Moving forward I have a half-dozen projects interested in moving to open-data solutions - not just for money reasons, but for keeping data current, living "live" on user-generated data, for instance. But specific APIs (google, yelp, etc.) have APIs that generally limit bandwidth. Has anyone had luck with specific databases / APIs more than others, or with reaching out to companies to ask for "public benefit" arrangements?

  • Do you have any programming skills if so what languages? – risail Aug 16 '16 at 17:34
  • Yes, I'm a former computer scientist, current epidemiologist. I primarily use R, which has some packages on scraping I've used in the past. I'm creeping into python, but not nearly as good. Also sas,vba,perl,etc. But it's as much an issue of legality and apis as it is programming, to me. – Mike Dolan Fliss Aug 16 '16 at 18:09
2

In most cases it's not illegal to scrape sites, or unethical (dependent on use). Getting direct access to a companies database is always tricky if not cost prohibitive. The first question is always what data do you want and where is it. Then look and see if they have an API that meets your needs and skill set, if not use scrapy and write directly to a db. There are other Python libraries written for this purpose as well such as BS4 and mechanize

  • Thanks for this. Have you had any experience with this directly? I'm also particularly curious about nonprofits requests for additional API access, which I know some companies give for beneficial use cases. – Mike Dolan Fliss Aug 22 '16 at 13:17
  • Yes I do, and there is no shortage of tutorials on how to use these tools on the web. As for requesting data from a company, no one is going to be able to offer advice on that, you are going got need to pick up the phone and contact the company you want data from. Worse they can say is No. – risail Aug 22 '16 at 13:31
  • :| Yes, I've seen tutorials. That's not what I'm here for. I wasn't criticizing you or suggesting you were inexperienced. Yes, I can pick up the phone and call. I disagree that "no one can offer advice on how to do that." I've seen some advice elsewhere, and was hoping someone on here would elaborate. Thanks for recommending a few python packages. I'll look elsewhere for other nonprofit professional data folks who have successfully gotten API extension contracts. Maybe some will come by this thread, and have something to contribute. Otherwise I'll answer my own question. Best wishes. – Mike Dolan Fliss Aug 22 '16 at 14:37
  • I think you misunderstood what I was trying to say....my bad – risail Aug 22 '16 at 14:39
  • Well, lesson learned to me - a number of the responses, including yours, have appropriately pointed out my question isn't precise - I should have listed programming languages and emphasized my interest is largely on the technicalities of successful API extension requests, with a link to what I was referencing. Apologies, and thanks. – Mike Dolan Fliss Aug 22 '16 at 14:43
2

There are a number of government sources that you could turn to for a live list of a specific type of retailer, in lieu of scraping Google or Yelp.

For example, the FDA publishes a dataset of all tobacco retailer inspections, which could be a good source of tobacco retailer location data on a national basis.

You also could turn to specific state governments for this data because many licenses, such as tobacco retailer licenses or liquor licenses, are regulated on state level. My employer Enigma.io has already collected a lot of this state-specific data (and it's free for non-commercial use!) - for example we have a dataset of names/locations of all licensed tobacco retailers in the state of Louisiana.

  • This is very interesting, eveahe. We use many of these same datasets..., including coordinating tobacco retailer inspections / store audits, and underage buys. But not all states, unfortunately, keep records of their tobacco licensing - which is bananas, I know, but that's why it comes up. Florida, for instance, doesn't have a list of tobacco retailers... though most if not all states have that for alcohol sellers. I'll look into Enigma.io - I hadn't heard of it! – Mike Dolan Fliss Aug 17 '16 at 22:03
  • Annoying! Yes, this is the tricky part of using local-level data, even at the state level, there is so much variation in what data states collect and then released. I feel like I have come across tobacco retailer data for Florida, though, have you checked out this site? myfloridalicense.com/dbpr/sto/file_download/… – eveahe Aug 18 '16 at 22:43
  • Ah, I misspoke - FL does have a list (that's the one), but not all states do. For instance, VA and NC do not. And no states have vape list yet that we know of, and we're working directly with depts of health/human services in most cases. And even if so, those lists can often be out of date, not moving at the market speed. – Mike Dolan Fliss Aug 22 '16 at 13:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.