I'd like to be able to access the Gapminder files programmatically (e.g. via R or Python), but I can't find a way so far.

There's a nice R package for accessing World Bank data that overlaps with the data I want, but Gapminder seems to have stuff that isn't included in the WB database (e.g. GDP per capita 1950-1960).

... thoughts ... ?

  • What do you mean by "clicky"? Can't you dl everything you want in "Excel" format, convert, and use? – user3856 Jul 26 '17 at 15:42
  • I want to be able to use a program to retrieve the values rather than going to the web page in a browser and doing it by hand ... – Ben Bolker Jul 26 '17 at 15:51
  • Oh, you could use a tool like "down-them-all" or something for mass downloading. Mass converting might be harder. Of course, you can always try contacting them directly. – user3856 Jul 26 '17 at 16:47

I downloaded http://www.gapminder.org/data/ after choosing "ALL" and noticing the spreadsheet links end in "output=xlsx". I changed the xlsx to csv to see what would happen, and did, indeed, get CSV files as expected. More specifically, I ran:

perl -nle 'while (s/href="(.*?)"//) {$x=$1; $x=~s/\&amp\;/&/g; print "curl -LO \47$x\47"}' gapminder.html | fgrep xlsx | sed 's/xlsx/csv/' | sh

zipped the results, and uploaded them to https://www.dropbox.com/s/r77tfdjt1lzkn56/gapminder.zip?dl=0

The only problem is that 519 files I thus obtained have weird names, so you may need to figure out which file is which.

| improve this answer | |

I ended up scraping this a little more thoroughly myself (including e.g. following redirects so the real Google sheets URLs are actually available).

  • The scraping code is here (GitHub pages; too long to post here)
  • The assembled CSV file is here: 519 rows, starting:
"Adults with HIV (%, age 15-49)",Based on UNAIDS,Health,HIV,https://docs.google.com/spreadsheets/d/1kWH_xdJDM4SMfT_Kzpkk-1yuxWChfurZuWYjfmv51EA/pub?output=xlsx
Age at 1st marriage (women),Various sources,Population,,https://docs.google.com/spreadsheets/d/1xYQ08p5llwPR3ZK6h900LDd-J7WuXCv_FoPlksrj-eA/pub?output=xlsx
Aged 15+ employment rate (%),International Labour Organization,Work,Employment rate,https://docs.google.com/spreadsheets/d/1jsXSKnWgsXvjYyR-4R2b9VKGh22IM70C9WQg74xK8U8/pub?output=xlsx
Aged 15+ labour force participation rate (%),International Labour Organization,Work,Labour force participation,https://docs.google.com/spreadsheets/d/16GBTNLyaFV__Bcz81ex3nQw6W6THqzKkaF9jZcCYjHw/pub?output=xlsx
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.