I want to scrape data periodically from a website and save it to a database that's should be readable by everybody. I thought about using a Google Docs Spreadsheet but the overall API management (especially the authorization) seems to be not suited to my needs. Do you know of any alternatives?

  • 1
    The easiest thing would be to store them in a CSV/TSV file somewhere. – user3856 Jun 3 '17 at 0:19
  • 1
    to @BarryCarter's point, and building on it, i would keep the scraper and the database in github. i'd do same repo but you can split it if you like, either way, run scraper periodically, then update the data. github bonus - if its csv/tsv its going to show as html table for users to pour through it. – albert Jun 3 '17 at 6:56
  • How exactly is this related to Open Data? Surely if you have to resort to scrapping a website (why haven't you contacted the admins first to provide you the data?) how is the data that you are going to "provide" open? – marfi Jun 3 '17 at 8:20
  • 1
    The data is provided from the state but old values are only kept for one week on the site. It's about the German administration and they don't give a shit about open data and will never give me the data to me voluntarily. – Johannes Filter Jun 3 '17 at 8:24
  • @marfi there are a plethora of open data repositories build atop scraping; ideally scraping would not be needed. open states is a primary example in this context. – albert Jun 5 '17 at 22:19

A quick an easy way to provide open access to your scraped data would be pushing it to GitHub or another easily accessible online repo in a text format (CSV/TSV or JSON if it's less structured -- this would also allow you to add timestamps for when data was scraped) and update it automatically. Without an explicit license to publish the data it may not be open, legally-speaking, so I would advise including any license information published on the source site as well as asking for the data and/or permission from the publishing organization. Even if they say no or ignore you, you've done your homework and may learn something important (and they can't say "yes" if you don't ask).

  • nice point regarding timestamps – albert Jun 5 '17 at 22:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.