Download wikipedia dump and save in raw text form

Question

I have been trying to use Wikipedia text data for my personal research. I know that crawling is not good for the Wikipedia server so I downloaded a big XML file from https://dumps.wikimedia.org/jawiki/latest/, especially I downloaded 3 files jawiki-latest-pages-articles1.xml.bz2, jawiki-latest-pages-articles2.xml.bz2, jawiki-latest-pages-articles3.xml.bz2 and it was a success.

When I check the data using my python, I noticed it is in wiki-specific format (wikitext). How can I parse wikitext into plain text? I could not find any good third-party parser. Most of what I found was no longer updated.

Answer 1

There seems to be a CPAN package for perl called WikiText, which from its description will do what you are after.

WikiText provides a parser for the WikiText markup language, and output modules to convert parsed documents into other markup languages, such as HTML, Latex or Pod.

You might need to do a further step to extract the text itself, but another article suggests that it isn't quite a straightforward process, due to the complexities in the page markup language structure.

Answer 2

As Markus D said, converting WikiText to html is not straighforward. The software that does it is MediaWiki - the sofware that runs Wikipedia - and any simpler imitation of it just does part of the work. However, you can install MediaWiki in you own server and crawl the whole Wikipedia at home. For research purposes which can benefit Wikipedia or at least serve the same goals of the Wikimedia movement you could ask for an account in Wikimedia Labs, where you could install your own software to access local dumps.

Anyway, if you don't need the full html version of Wikipedia pages, there are several parsers that could extract most of the raw text from WikiText. For example, I've used MWParseFromHell and it works quite fine.

And of course, you can access Wikipedia on-line if time is not a concern. You shouldn't be worried about overloading their servers because there is a limit for the number of queries per minute a bot can do, and that limit is safe enough for them. If I remember well, some months ago it meant for me that I could read nearly 1 page/second. That is less than two weeks for the for the whole Japanese language Wikipedia. Selecting beforehand what articles are you interested in - using searches or categories - may shorten that time a lot.