Live database of checksums of all files on Wikimedia Commons

Question

Wikimedia Commons is a database of 25,000,000+ freely usable media files to which anyone can contribute.

Contributing a file to Commons takes some work (creating metadata, description, selecting categories), and it fails if the file has already been uploaded by someone else (or by forgetful me in the past).

So, a database of all checksums of Commons files would be very useful, to compare before working on metadata.

Is there such a database?

• Free
• Reasonably up-to-date, though I can bear with a week or so of delay
• API to quickly request whether a particular checksum is already on Commons or not
• No need to download the whole database before being able to use it

Answer 1

I think Elog.io meets your requirements or at least comes close. Some background on how they use checksums to find similar images is in this blog post

Answer 2

Wikimedia Commons has developed a special API exactly for this, with the keyword aisha1 which stands for "All Images with a given SHA1".

API usage

Let's say you want to check whether your file myimage.jpg is already on Wikimedia Commons or not.

1. Calculate the checksum of myimage.jpg, for instance your get 801957214aba50cb63bb6eb1b0effa50188900ba
2. Launch a request to https://commons.wikimedia.org/w/api.php?action=query&list=allimages&aisha1=801957214aba50cb63bb6eb1b0effa50188900ba
3. You receive a JSON response
4. In this JSON, if allimages is empty then myimage.jpg has never been uploaded to Wikimedia Commons. If it has, you receive a reference to the existing image, example:

JSON:

{
    "batchcomplete": "",
    "query": {
        "allimages": [
            {
                "name": "Commons-logo-beta.png",
                "timestamp": "2010-05-19T22:21:32Z",
                "url": "https://upload.wikimedia.org/wikipedia/commons/b/ba/Commons-logo-beta.png",
                "descriptionurl": "https://commons.wikimedia.org/wiki/File:Commons-logo-beta.png",
                "descriptionshorturl": "https://commons.wikimedia.org/w/index.php?curid=10378437",
                "ns": 6,
                "title": "File:Commons-logo-beta.png"
            }
        ]
    }
}

Alternatively, there seems to be a different approach with the same goal: https://commons.wikimedia.org/wiki/Commons:User_scripts/File_Analyzer