30

Wikidata is a new Wikimedia project:

It centralizes access to and management of structured data, such as interwiki references and statistical information.

This data would be of enormous interest to the Open Data community. Does anyone know if it is possible to download a complete dump of the Wikidata database?

Update: Thanks to everyone who has pointed me to the Wikidata wiki dumps. However, I am interested in a more usable data format that doesn't require parsing to get to the actual data. An RDF dump, for example, would do nicely.

Improve this question
8
  • 2
    Not sure if you noticed that you can already download some data dumps: dumps.wikimedia.org/wikidatawiki Most of these files lack some decent description but as you can see, they're actively uploading new documents.
    – r_31415
    May 9, 2013 at 4:25
  • @RobertSmith, thanks for the link! As far as I understand, these dumps contain the Wikidata wiki, but not the Wikidata data – I know, it sounds a little confusing ;) Am I wrong? If so, which one of the dumps is the one with the relevant data?
    – Patrick Hoefler
    May 9, 2013 at 8:57
  • btw have you checked this answer?
    – RSFalcon7
    May 9, 2013 at 9:05
  • Thanks, @RSFalcon7! However, I'm interested in the data from the Wikidata project, not the database dumps from Wikipedia – though they can also be very helpful in certain cases.
    – Patrick Hoefler
    May 9, 2013 at 9:39
  • @PatrickHoefler I think you're right but it seemed to me they are just uploading their own data first.
    – r_31415
    May 10, 2013 at 15:10

8 Answers 8

Reset to default
12

The wikidata dump is already available. As of now, the last (mostly) complete dump is from 5 May 2013 and it includes the dump of pages in the important namespaces (pages-articles.xml).

Improve this answer
2
  • Thanks for the links! However, I'm not interested in the dumps of the wiki pages, but in the dumps of the data collected by the Wikidata project.
    – Patrick Hoefler
    May 22, 2013 at 17:09
  • 5
    @PatrickHoefler Well, that data is in that dump too. Wikidata stores its data as JSON page content. For example, if you look at the XML, the third <page> there is Q15 (Africa).
    – svick
    May 22, 2013 at 17:15
12

Wikidata RDF dumps have been available since 3-Aug-2013. http://www.mail-archive.com/[email protected]/msg02553.html

If you're interested in topics related to Wikidata, I'd suggest subscribing to their mailing list.

Of course, both Freebase and DBpedia have had much larger dumps available for years.

Improve this answer
0
7

The Wikidata database is currently offered for download in several different formats:

Improve this answer
0
5

Depending on your use-case Wikidata toolkit is a Java tool that allows you to iterate over all the items in a dump. In fact it auto-downloads the latest dump and translates the entire model into Java objects. This means everything is abstracted and you just have to write the call-back to be applied to each item. I wrote a tutorial on how to use it in conjunction with Python pandas here.

Improve this answer
3

Additionally to what is described in other answers, there is a Wikidata RDF dump which is used for the Wikidata query service. The format is described in RDF data format article.

Improve this answer
3

As you already found out the Dumps of wikidata are readily available and can be downloaded and unzipped in a few hours. Getting a proper RDF database server running with such dumps is another story.

At http://wiki.bitplan.com/index.php/Get_your_own_copy_of_WikiData you'll find the description of some successful and failed attempts as well as Performance reports.

Importing WikiData dumps is a moving target. Something that works today in a certain environment might not work in a few weeks or months.

As of 2020-06 a successful import has been reported by Jonas Sourlier in https://issues.apache.org/jira/browse/JENA-1909

Please note that Jonas has switched the loader to the more recent one just as i did. My recent attempt described in https://issues.apache.org/jira/browse/JENA-1908 failed.

I hope the fourth attempt I am currently describing will be successful and you could then simply follow the procedure described in the wikipage (which I'll copy here once the attempt is successful). Please stay tuned ...

http://wiki.bitplan.com/index.php/WikiData_Import_2020-08-15 reports a successful attempt.

Improve this answer
3

Max Klein describes three ways of getting the data at 3 Ways To Access Wikidata Data Until It Can Be Done Properly.

Not mentioned is the WDA tool.

Improve this answer
2

I have no information about Wikidata, but those are two ways to download structured information from Wikipedia:

Images and uploaded files are stored elsewhere, also downloadable

Also answered here

Improve this answer
2
  • Thanks for the links! However, I'm especially interested in the data from the Wikidata project, not the database dumps from Wikipedia – though they can also be very helpful in certain cases.
    – Patrick Hoefler
    May 9, 2013 at 9:33
  • 1
    This is not structured information, it's semi-structured. It's mostly prose text which has no structure but is supplemented with things like infoboxes which are structured to varying degrees and always require clean up.
    – hippietrail
    Aug 10, 2013 at 3:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.