10

I installed media-wiki on ec2 machine and loaded the Wikipedia page-articles dump content and other relevant data into that. I want to update the data regularly(daily) but I didn't get any resource for that. Is there any source from where I can get Wikipedia daily updates?

Note: I can get the list of modified files using bots. But I need to get the modified content also.

Improve this question

3 Answers 3

Reset to default
7

Right now, there is no good solution. The options I can think of are:

  1. Just download the normal dump whenever it becomes available. This means you won't get daily updates at all. For the English Wikipedia, a new dump is generated about once a month.

  2. Use the adds/changes dumps (already mentioned by ojdo). There are two problems with using this:

    1. It doesn't include all changes. Specifically, information about moves, deletes and some undeletes are not included.
    2. It's an experimental feature and I wouldn't be surprised if it were discontinued in the near future (because a better option is coming, see below).

  3. Use the API (possibly combined with the IRC recent changes feed) to get the text of all new revisions to a wiki. I think this might work for very small wikis, but it's certainly not feasible for huge active wikis like the English Wikipedia.

  4. Use binary incremental dumps (which is a project I built over this summer). This will do exactly what you want: it will allow you to download only changes since the last dump and it should allow creating dumps much more often (the hope is for daily dumps). The only problem is that this is not live yet, so you will have to wait before using this (I have no idea how long, but I would expect it to go live this year).

Improve this answer
5
  • It's strange that Wikimedia hasn't come up yet with an easy way to "track" changes. OpenStreetMap (diffs) seems to be in much better shape in comparison.
    – ojdo
    Commented Sep 30, 2013 at 11:17
  • Right now only the second solution helps in my project(Not completely..But something is better than nothing.). Dbpedia is getting live updates from wikipedia. I don't know how much effort it takes to make that live feed public to everyone.
    – vinod
    Commented Oct 1, 2013 at 5:15
  • @vinod I don't think DBPedia does that. For example, when I look at dbpedia-owl:wikiPageRevisionID of Albert Einstein, it points to a revision from April.
    – svick
    Commented Oct 1, 2013 at 8:05
  • @svick There are 2 versions of dbpedia. dbpedia, dbpedia-live. I was referring to dbpedia-live. Try the query SELECT * WHERE { <http://en.wikipedia.org/wiki/Albert_Einstein> ?p ?o } at "live.dbpedia.org/sparql" (DBPedia live data has been last updated on aug 29th, 2013. It is down for maintenance purpose. It should be up and running very soon.)
    – vinod
    Commented Oct 1, 2013 at 8:47
  • @vinod It seems you're right. It looks like that uses update feed service, which is not public.
    – svick
    Commented Oct 1, 2013 at 8:53
4

There exists an experimental Wikimedia adds/changes dump service. Another approach is generating the differential dump yourself using deltapack, discussed about in this thread in the Wikitech mailing list.

Improve this answer
3
  • Thanks for the references. I will look into them and get back to you.
    – vinod
    Commented Sep 30, 2013 at 9:43
  • Note that the adds/changes dumps include only new revisions, but not deletions or things like that.
    – svick
    Commented Sep 30, 2013 at 10:24
  • And generating the diff itself won't help, if there is nothing to diff (the dumps for Wikipedias are not generated every day).
    – svick
    Commented Sep 30, 2013 at 10:26
3

It wouldn't give you complete updates, but there is also dbpedia live. Dbpedia only covers the stuctured text boxes in the wikipedia entries. It is monitoring the changes on Wikipedia though.

Improve this answer
1
  • @Andra.. I tried that already but rate of error is more than we expected and they stopped providing updates from aug 29th(They told me that they would start providing them asap).
    – vinod
    Commented Oct 14, 2013 at 9:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.