I want to scrape data periodically from a website and save it to a database that's should be readable by everybody. I thought about using a Google Docs Spreadsheet but the overall API management (especially the authorization) seems to be not suited to my needs. Do you know of any alternatives?
-
1The easiest thing would be to store them in a CSV/TSV file somewhere.– user3856Commented Jun 3, 2017 at 0:19
-
1to @BarryCarter's point, and building on it, i would keep the scraper and the database in github. i'd do same repo but you can split it if you like, either way, run scraper periodically, then update the data. github bonus - if its csv/tsv its going to show as html table for users to pour through it.– albertCommented Jun 3, 2017 at 6:56
-
How exactly is this related to Open Data? Surely if you have to resort to scrapping a website (why haven't you contacted the admins first to provide you the data?) how is the data that you are going to "provide" open?– marfiCommented Jun 3, 2017 at 8:20
-
@marfi there are a plethora of open data repositories build atop scraping; ideally scraping would not be needed. open states is a primary example in this context.– albertCommented Jun 5, 2017 at 22:19
Add a comment
|
1 Answer
A quick an easy way to provide open access to your scraped data would be pushing it to GitHub or another easily accessible online repo in a text format (CSV/TSV or JSON if it's less structured -- this would also allow you to add timestamps for when data was scraped) and update it automatically. Without an explicit license to publish the data it may not be open, legally-speaking, so I would advise including any license information published on the source site as well as asking for the data and/or permission from the publishing organization. Even if they say no or ignore you, you've done your homework and may learn something important (and they can't say "yes" if you don't ask).
