API/machine access for Gapminder data?

Question

I'd like to be able to access the Gapminder files programmatically (e.g. via R or Python), but I can't find a way so far.

• Gapminder's downloads page offers a variety of tools and charts, but nothing for programmatic access (that I can tell)
• their data page is clicky.
• there's some discussion of the possibility of an API on their community support page, but nothing concrete
• someone wrote a Python app, but I have to admit that I don't really understand how to use it (I have installed it, and can run python3 app.py after that, but that's as far as I get ..)
• this web page has a promising name, but it's a description of an R package that contains a little bit of downloaded Gapminder data

There's a nice R package for accessing World Bank data that overlaps with the data I want, but Gapminder seems to have stuff that isn't included in the WB database (e.g. GDP per capita 1950-1960).

... thoughts ... ?

Answer 1

I downloaded http://www.gapminder.org/data/ after choosing "ALL" and noticing the spreadsheet links end in "output=xlsx". I changed the xlsx to csv to see what would happen, and did, indeed, get CSV files as expected. More specifically, I ran:

perl -nle 'while (s/href="(.*?)"//) {$x=$1; $x=~s/\&amp\;/&/g; print "curl -LO \47$x\47"}' gapminder.html | fgrep xlsx | sed 's/xlsx/csv/' | sh

zipped the results, and uploaded them to https://www.dropbox.com/s/r77tfdjt1lzkn56/gapminder.zip?dl=0

The only problem is that 519 files I thus obtained have weird names, so you may need to figure out which file is which.

Answer 2

I ended up scraping this a little more thoroughly myself (including e.g. following redirects so the real Google sheets URLs are actually available).

• The scraping code is here (GitHub pages; too long to post here)
• The assembled CSV file is here: 519 rows, starting:

indicator,provider,category,subcategory,URL
"Adults with HIV (%, age 15-49)",Based on UNAIDS,Health,HIV,https://docs.google.com/spreadsheets/d/1kWH_xdJDM4SMfT_Kzpkk-1yuxWChfurZuWYjfmv51EA/pub?output=xlsx
Age at 1st marriage (women),Various sources,Population,,https://docs.google.com/spreadsheets/d/1xYQ08p5llwPR3ZK6h900LDd-J7WuXCv_FoPlksrj-eA/pub?output=xlsx
Aged 15+ employment rate (%),International Labour Organization,Work,Employment rate,https://docs.google.com/spreadsheets/d/1jsXSKnWgsXvjYyR-4R2b9VKGh22IM70C9WQg74xK8U8/pub?output=xlsx
Aged 15+ labour force participation rate (%),International Labour Organization,Work,Labour force participation,https://docs.google.com/spreadsheets/d/16GBTNLyaFV__Bcz81ex3nQw6W6THqzKkaF9jZcCYjHw/pub?output=xlsx

• I also posted a CSV file listing countries and their corresponding regions, scraped from Wikipedia.