What is the best way to convert a taxonomy from wikipedia into a JSON-like object?

Question

I'm trying to use the Wikipedia academic fields as a JSON object in one of my projects. I couldn't find it in a useable or convertible format so I am looking into the best approach to obtain it.

I came up with 3 approaches:

• Get the data from DBPedia (https://dbpedia.org/data/Outline_of_academic_disciplines.json) but it returns a non-ordered list with no hierarchical level
• Copy-paste the markdown from the Wikipedia page and proceed with a regex/find/replace trickery but it is not as easy as it seemed at first (inconsistencies in the page to deal with across 5900 lines once prettified)
• Go the programmatic way: I import the markdown file and parse it. It feels like I'm doing a computer science exercise (a fun little recursive function) and I would gladly spend time on it if I had some to spare. The part where I go through trial-error to correct all the inconsistencies in the file doesn't look too appealing tho. :-)

I probably will go for the third option if I don't find an easier way. However, as all 3 options would require a good amount of work and I would gladly welcome a better/simpler approach.

Any ideas? Any other source to find wikimedia related data in a suitable format in general and this page in particular?

Thanks

Answer 1

I would recommend web-scraping the webpage for the data that you require.

You can either opt for scrapy, which is faster and so it should not take long at all if you have experience with it. I approximate less than 15 minutes at best.

Alternatively, you can go down the requests route and use pandas to read the tables, and then grab all the information from the tables this way. This is also a very easy route to take.

Answer 2

I don't think there's an easy, out-of-the-box, solution, but another possible path to investigate is querying Wikidata for instances of academic discipline or field of study and examining the hierarchy using the subclass of property.