0

I am looking for a list of all the entries of https://de.wiktionary.org (German Wiktionary, but I think this works for all Wikimedia the same way) that have had any changes since the last dump. Is there something like that? Under which URL is this list available? Or is there a list that contains the timestamp of the last change for each entry?

Background:

Wikimedia (including Wikipedia, Wiktionary etc.) provides dumps of its contents every 1 or 2 weeks.

A script I wrote loads every day the page https://dumps.wikimedia.org/dewiktionary/latest/ and reads the timestamp of the current version of the file dewiktionary-latest-all-titles-in-ns0.gz. If there is a new version of this file (usually every 1 - 2 weeks), the script downloads it.

This file contains all the titles of the sites de.wiktionary.org (only the titles, no contents). My script compares this list with the titles that are already in my database. It marks entries as deleted if they are no longer in the current list, and marks titles for download that do not yet exist in my database.

Then another script iterates through all the new titles and downloads their contents, one after the other, for example by going to this page: https://de.wiktionary.org/w/index.php?action=raw&title=Titel (where you can find the sourcecode for this site: https://de.wiktionary.org/wiki/Titel) using the respective title instead of Titel, of course.

To conserve both my own server's and Wiktionary's bandwidth, this script always pauses briefly between two titles, thus downloading the contents of a maximum of about 100,000 titles per day. There are about 1,000,000 titles in total in German Wiktionary, so the download of the whole German Wiktionary was done in less than 2 weeks.

And here's the point:

I want to keep my database as up-to-date as possible, but I also want to download as little data as possible. Above all, I don't want to download content again and again every 10 days, that already exists in exactly the same version in my database.

So ideally I only want to download the contents of those titles where there has been any change since the last dump (as provided on https://dumps.wikimedia.org/dewiktionary/latest/). Is there such a list? If so, where can I find it?

A list where each title has a timestamp of it's last change also will do.

1

There are two ways to generate a list of last-revision data for all pages. The preferable tool is probably the second one listed here, PetScan.

The first is to use the Wikimedia Quarry SQL service (quarry.wmflabs.org - documentation) to generate a list of all pages with metadata. This query (took about 25m to run, most recent results are cached) gives the page ID, page title, and latest revision ID for all ns=0 (ie main article space) pages on dewiktionary. You could also tweak the query so it picks up the actual timestamp of the latest revision ID (this is available from the revisions table, but my SQL competency to tie them together is a bit limited...).

The second is a slightly hacky approach should be "good enough" for most purposes: the PetScan tool (petscan.wmflabs.org - documentation) can generate this data for arbitrary queries like "all articles in this set of categories". It can't do "every page on the wiki", which is a pity, but as it turns out pretty much every Wiktionary article page is in a sub-category of Kategorie:Sprachen, so we can just use that as our source. This query looks for everything filed in Sprachen or up to three levels below it, takes ~60s to run, and identifies almost the same number of pages as the main one - 992644, versus 993681 in the Quarry list. You can get this list of results as a TSV or other file by selecting the relevant format in the last tab ("output") and re-running.

There are three important identifiers here. One is the page title, which you know about. This is likely to be static for Wiktionary (given it is the headword) but might change if you were eg doing an analysis on Wikipedia. The next is the page ID, which is assigned when the page is created and does not change even if the title does. The last is the latest revision ID (in Quarry) or the last changed date (in Petscan). For example, "Pferd" has a latest revision ID of 8615119, which corresponds to being edited on 22 June). This is the one you're looking for.

In terms of which source to use - well, either would work. I think the PetScan approach is likely to be preferable, as it is a much quicker system (data in ~1m not ~25m) and so is presumably a lot less expensive on the back-end systems. I think it is also possible to trigger a fresh PetScan query with wget etc, using the exact URL for the query, while Quarry may have to be manually triggered in the browser.

There is a slight risk that the PetScan approach would lose any pages that have not been categorised for whatever reason, but it looks like this is very rare. On examination they are probably redirects to another page title, which would count on the Quarry list (they exist in namespace 0) but which would not be picked up by PetScan as they are usually not placed in categories; there are currently around 1050 redirects in namespace 0 per this index, which is very close to the discrepancy we saw between the two lists.

5
  • Thank you, Andrew. This is not exactly what I expected, because I didn't want to run any reports on wikimedia servers. But I still think it's a good approach. I am not only interested in properly categorized entries but also in entries that redirect to non-existing entries and in entries with malformed level-2-headings which will not be assigned to any category and in any other quirky entries. But I will read the documentations of these tools and maybe I'll find a way. - Thanks! Jul 15 at 8:15
  • 1
    @HubertSchölnast I think I missed a trick the first time around, and you probably could do the first approach without running a report on the server - you can download the pages SQL dump and try to pick out revision IDs/timestamps from that, either with SQL or by just parsing the file. I've poked at the file and it definitely seems to have the right page ID, page title, and revision ID plus last-edited timestamps. Will try and write up an expanded answer tonight.
    – Andrew
    Jul 15 at 11:27
  • I tried Quarry and it works fine. I can run any query I want via my web browser, but I want to execute these queries form a python script running on my computer. I tumbled trough many pages of documentation, registered on some wikimedia services and still fail to connect to the server. What is the easiest way to execute such queries from a script (i.e. without a web browser)? Jul 24 at 16:45
  • @HubertSchölnast Good question. It's possible that you can't trigger a Quarry query without going through the browser - it's been a while since I did much work with it so I'm not completely sure if there's a workaround, but I can't see an obvious way to do it. Maybe parsing the pages dump each time a new one appears is the best approach after all?
    – Andrew
    Jul 24 at 17:03
  • Thank you. I already thought about this approach, but it also is a lot of work because I need to log into Quarry from my script using cookies. And before I do so I need to analyze what exactly will be sent from the client to the server. Jul 24 at 20:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.