I am looking for a list of all the entries of https://de.wiktionary.org (German Wiktionary, but I think this works for all Wikimedia the same way) that have had any changes since the last dump. Is there something like that? Under which URL is this list available? Or is there a list that contains the timestamp of the last change for each entry?
Wikimedia (including Wikipedia, Wiktionary etc.) provides dumps of its contents every 1 or 2 weeks.
A script I wrote loads every day the page
https://dumps.wikimedia.org/dewiktionary/latest/ and reads the timestamp of the current version of the file
dewiktionary-latest-all-titles-in-ns0.gz. If there is a new version of this file (usually every 1 - 2 weeks), the script downloads it.
This file contains all the titles of the sites de.wiktionary.org (only the titles, no contents). My script compares this list with the titles that are already in my database. It marks entries as deleted if they are no longer in the current list, and marks titles for download that do not yet exist in my database.
Then another script iterates through all the new titles and downloads their contents, one after the other, for example by going to this page: https://de.wiktionary.org/w/index.php?action=raw&title=Titel (where you can find the sourcecode for this site: https://de.wiktionary.org/wiki/Titel) using the respective title instead of
Titel, of course.
To conserve both my own server's and Wiktionary's bandwidth, this script always pauses briefly between two titles, thus downloading the contents of a maximum of about 100,000 titles per day. There are about 1,000,000 titles in total in German Wiktionary, so the download of the whole German Wiktionary was done in less than 2 weeks.
And here's the point:
I want to keep my database as up-to-date as possible, but I also want to download as little data as possible. Above all, I don't want to download content again and again every 10 days, that already exists in exactly the same version in my database.
So ideally I only want to download the contents of those titles where there has been any change since the last dump (as provided on
https://dumps.wikimedia.org/dewiktionary/latest/). Is there such a list? If so, where can I find it?
A list where each title has a timestamp of it's last change also will do.