4

The staff of the European Commission is obliged to keep a register of their meetings with lobbyists. The site Integrity Watch administered by Transparency International keeps track of these meetings on their webpage.

They pull together data from more than 90 original sources of the EC. All the links to the orginal sources can be found here. Instead of webscraping the original sources it would be great to just get the table from Integrity Watch. Unfortunately they didn't reply to my question about download options yet. My attempts to webscrape the page in R (rvest) failed because of the interactive nature of the table. Any experienced webscrapers who could give some hints and code chunks for R or Python?

Improve this question
2
  • You need to actually interact with the page (e.g. crawl through links), so it's more than just parsing HTML. For python check out mechanize or scrapy
    – philshem
    May 31, 2016 at 18:34
  • 1
    The problem with this @philshem, seems to be that there are no links, and the content delivery simply works through some Ajax magic. Anything of interest to the OP is under always integritywatch.eu/ec.html. The website states in their "about" page that "The technology [...](D3.js) was developed [...]in order to make complex datasets accessible to a wider audience. All datasets are also available for download as this platform strongly supports the principles of open software and open data." So I guess they should be willing to share without crawling/scraping, but it seems they aren't
    – sheß
    Jun 1, 2016 at 8:17

1 Answer 1

Reset to default
2

With a hint from a person from Transparency International and some further browsing through the source code this link eventually made it work:

http://static.tttp.eu/doi/data/meeting.csv

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.