2

Wikimedia Commons is a database of 25,000,000+ freely usable media files to which anyone can contribute.

Contributing a file to Commons takes some work (creating metadata, description, selecting categories), and it fails if the file has already been uploaded by someone else (or by forgetful me in the past).

So, a database of all checksums of Commons files would be very useful, to compare before working on metadata.

Is there such a database?

  • Free
  • Reasonably up-to-date, though I can bear with a week or so of delay
  • API to quickly request whether a particular checksum is already on Commons or not
  • No need to download the whole database before being able to use it
2

I think Elog.io meets your requirements or at least comes close. Some background on how they use checksums to find similar images is in this blog post

  • Nice! This repository has "hamming distance" "blockhashes" to find images that are extremely similar. I am looking for checksums to identify the exact same image, which is probably easier than this. Anyway, great resource, I will use it if there is no database of exact checksums. – Nicolas Raoul May 26 '15 at 2:07
  • I think the parent project, Commons Machinery, has that database of exact checksums. All their code is on GitHub. – Daniel Mietchen May 26 '15 at 3:38
  • Could you please make another answer about that database of exact checksums? I haven't been able to find it (or a live hosted API to it) on their website nor on their Github. All projects I have found seem to involve hamming or perceptual distances. Thanks! – Nicolas Raoul May 26 '15 at 6:13
  • 2
    I just asked directly: twitter.com/EvoMRI/status/604372934258466816 . – Daniel Mietchen May 29 '15 at 19:46
2

Wikimedia Commons has developed a special API exactly for this.

Documentation and web UI: https://commons.wikimedia.org/wiki/Commons:User_scripts/File_Analyzer

API usage

Let's say you want to check whether your file myimage.jpg is already on Wikimedia Commons or not.

  1. Calculate the checksum of myimage.jpg, for instance your get 801957214aba50cb63bb6eb1b0effa50188900ba
  2. Launch a request to https://commons.wikimedia.org/w/api.php?action=query&list=allimages&aisha1=801957214aba50cb63bb6eb1b0effa50188900ba
  3. You receive a JSON response
  4. In this JSON, if allimages is empty then myimage.jpg has never been uploaded to Wikimedia Commons. If it has, you receive a reference to the existing image, example:

JSON:

{
    "batchcomplete": "",
    "query": {
        "allimages": [
            {
                "name": "Commons-logo-beta.png",
                "timestamp": "2010-05-19T22:21:32Z",
                "url": "https://upload.wikimedia.org/wikipedia/commons/b/ba/Commons-logo-beta.png",
                "descriptionurl": "https://commons.wikimedia.org/wiki/File:Commons-logo-beta.png",
                "descriptionshorturl": "https://commons.wikimedia.org/w/index.php?curid=10378437",
                "ns": 6,
                "title": "File:Commons-logo-beta.png"
            }
        ]
    }
}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.