Wikimedia: Delta-Dumps or Timestamp of Last Change

Question

I am looking for a list of all the entries of https://de.wiktionary.org (German Wiktionary, but I think this works for all Wikimedia the same way) that have had any changes since the last dump. Is there something like that? Under which URL is this list available? Or is there a list that contains the timestamp of the last change for each entry?

Background:

Wikimedia (including Wikipedia, Wiktionary etc.) provides dumps of its contents every 1 or 2 weeks.

A script I wrote loads every day the page https://dumps.wikimedia.org/dewiktionary/latest/ and reads the timestamp of the current version of the file dewiktionary-latest-all-titles-in-ns0.gz. If there is a new version of this file (usually every 1 - 2 weeks), the script downloads it.

This file contains all the titles of the sites de.wiktionary.org (only the titles, no contents). My script compares this list with the titles that are already in my database. It marks entries as deleted if they are no longer in the current list, and marks titles for download that do not yet exist in my database.

Then another script iterates through all the new titles and downloads their contents, one after the other, for example by going to this page: https://de.wiktionary.org/w/index.php?action=raw&title=Titel (where you can find the sourcecode for this site: https://de.wiktionary.org/wiki/Titel) using the respective title instead of Titel, of course.

To conserve both my own server's and Wiktionary's bandwidth, this script always pauses briefly between two titles, thus downloading the contents of a maximum of about 100,000 titles per day. There are about 1,000,000 titles in total in German Wiktionary, so the download of the whole German Wiktionary was done in less than 2 weeks.

And here's the point:

I want to keep my database as up-to-date as possible, but I also want to download as little data as possible. Above all, I don't want to download content again and again every 10 days, that already exists in exactly the same version in my database.

So ideally I only want to download the contents of those titles where there has been any change since the last dump (as provided on https://dumps.wikimedia.org/dewiktionary/latest/). Is there such a list? If so, where can I find it?

A list where each title has a timestamp of it's last change also will do.

Answer 1

There are two ways to generate a list of last-revision data for all pages. The preferable tool is probably the second one listed here, PetScan.

The first is to use the Wikimedia Quarry SQL service (quarry.wmflabs.org - documentation) to generate a list of all pages with metadata. This query (took about 25m to run, most recent results are cached) gives the page ID, page title, and latest revision ID for all ns=0 (ie main article space) pages on dewiktionary. You could also tweak the query so it picks up the actual timestamp of the latest revision ID (this is available from the revisions table, but my SQL competency to tie them together is a bit limited...).

The second is a slightly hacky approach should be "good enough" for most purposes: the PetScan tool (petscan.wmflabs.org - documentation) can generate this data for arbitrary queries like "all articles in this set of categories". It can't do "every page on the wiki", which is a pity, but as it turns out pretty much every Wiktionary article page is in a sub-category of Kategorie:Sprachen, so we can just use that as our source. This query looks for everything filed in Sprachen or up to three levels below it, takes ~60s to run, and identifies almost the same number of pages as the main one - 992644, versus 993681 in the Quarry list. You can get this list of results as a TSV or other file by selecting the relevant format in the last tab ("output") and re-running.

There are three important identifiers here. One is the page title, which you know about. This is likely to be static for Wiktionary (given it is the headword) but might change if you were eg doing an analysis on Wikipedia. The next is the page ID, which is assigned when the page is created and does not change even if the title does. The last is the latest revision ID (in Quarry) or the last changed date (in Petscan). For example, "Pferd" has a latest revision ID of 8615119, which corresponds to being edited on 22 June). This is the one you're looking for.

In terms of which source to use - well, either would work. I think the PetScan approach is likely to be preferable, as it is a much quicker system (data in ~1m not ~25m) and so is presumably a lot less expensive on the back-end systems. I think it is also possible to trigger a fresh PetScan query with wget etc, using the exact URL for the query, while Quarry may have to be manually triggered in the browser.

There is a slight risk that the PetScan approach would lose any pages that have not been categorised for whatever reason, but it looks like this is very rare. On examination they are probably redirects to another page title, which would count on the Quarry list (they exist in namespace 0) but which would not be picked up by PetScan as they are usually not placed in categories; there are currently around 1050 redirects in namespace 0 per this index, which is very close to the discrepancy we saw between the two lists.