I would like to know if there is a way to either download/scrape a clean CSV with coordinates and controlling group from Wikipedia conflict maps, or how to generate an ESRI shapefile from the Lua script. I have no experience using Lua.

Wikipedia has several maps for civil conflicts, which all seem to share the same structure and naming, e.g. Module:Sinai insurgency detailed map, which I've chosen due to its simplicity.

# R script for scraping text
# library

url <- "https://en.wikipedia.org/wiki/Template:Sinai_insurgency_detailed_map"
coords <- url %>%
    read_html() %>%
    html_nodes(xpath='//*[@id="mw-content-text"]/div[2]') %>%

I am currently using the above script to write a raw .txt file. I then clean it with OpenRefine to create a usable CSV in QGIS. I realize this is not the most efficient workflow.

  • I'm not sure I'm following, so you may be doing this already, but if you click 'edit' on the URL above, you get a slightly cleaner format for map marks. – Barry Carter Apr 28 '17 at 21:45
  • Thanks for commenting @BarryCarter. Unfortunately, the cleaner format is mostly just due to the spacing. If you have any advice on parsing that code to get a clean CSV, that would be great! I'm really a GIS user and not very experienced with scraping, which is why I'm using OpenRefine to clean it after scraping. – arabiccode May 2 '17 at 6:48
  • I'll give it a shot, but I write in Perl, not R, so not sure how helpful it'll be. Actually, I just realized you have a solution, you just want a cleaner one. Can't help, sorry. – Barry Carter May 2 '17 at 18:15
  • True, I have a basic solution. However, it does not seem very efficient given that the information is already structured. Somehow, Wikipedia calls the coordinates in the Lua language. Presumably, this means there is a much more direct solution. I would have included a Lua tag but I don't have the reputation. – arabiccode May 3 '17 at 7:22
  • I think I'm misunderstanding your question. Currently, you convert the file and import it into R, correct? How would using lua help? You'd still have to convert and import (unless lua is part of R?). I thought you wanted to import directly into R w/ no intermediate steps. – Barry Carter May 3 '17 at 16:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.