1

I'm just trying to parse Wiktionary Data Dump provided by Wikimedia. My intention is to parse that XML data dump into the MySQL database. I didn't find proper documentation regarding the structure of this XML. Also, I'm not able to open the file because it's in fact really huge (~1 GB). I thought of parsing it using some PHP script but I don't have any idea about the XML structure to proceed. So If anyone had already parsed (or has an idea about any tool to parse) this dump into MySQL using PHP, Please share the details. If nothing is in PHP, Other methods are also fine. I just followed this post (http://www.igrec.ca/lexicography/installing-a-local-copy-of-wiktionary-mysql/) but It didn't work out..:( If anybody has succeeded in this process, please help.

0

4 Answers 4

1

I found some hints for the schema here https://meta.wikimedia.org/wiki/Help:Export#Export_format

To read manually the XML, try using a viewer like:

1
  • beware of head -n when 1GB of XML is on one line!
    – philshem
    Aug 14, 2014 at 19:21
1

I wrote Wik2dict a decade ago to turn MW database dumps into the dict format. It's python code. Could help you figuring out some things.

The best way to convert this xml into mysql is by using mediawiki's xml import functionality.

If you don't have a good reason to import the dump into mysql it's better to avoid it as it's extremely slow with such a large amount of data. (Unless you're a good dba with a couple of fast machines at your disposal.)

1

The best option for the English Wiktionary is the Wiktextract project. It offers already parsed versions in JSONL format here. I did something similar and parsed every JSON line to then add it into my database.

0

The dbnary project is doing a great job at extracting a large amount of the Wiktionary data into RDF triples. If you prefer a the data as a relational database, I did further processing on the dbnary output and provide downloadable sqlite databases as part of WikDict.

If you are only interested in the English Wiktionary, I second the wiktextract recommendation, without having much experience with it myself.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.