1

A huge number of news related wiki pages contain article references (for example this article on the 2017 Equifax data breach) and I would like to obtain the following data from the references section at the bottom of the page:

  • The article title
  • The article url
  • The article publish date

There doesn't appear to be any way to obtain that data other than parsing the whole page. And the format seems inconsistent enough that doing so wouldn't be fruitful.

Improve this question
1
  • HTMl is machine-readable, and Wikipedia's format is extremely consistent. Looking @ the markup, with the exception of publish date, all of these are entirely doable. The publish date, which I'm assuming is the date after the title, can be accessed the exact way you would access the other data/elements, just keeping in mind that it won't exist for all of them. I would use python and I would take that into account when parsing to a dictionary. Since you don't want to scrape, I would use WikiData's API.
    – albert
    Oct 24, 2020 at 23:36

3 Answers 3

Reset to default
2

The format of the citations on Wikipedia, by and large, follows one of two standard citation models, with the exciting names Citation Style 1 (https://en.wikipedia.org/wiki/Help:Citation_Style_1) and Citation Style 2 (https://en.wikipedia.org/wiki/Help:Citation_Style_2). These are easy to parse, when they are using the standard templates. Lesser used styles (Vancouver, LSA) should not be harder, just different.

Since Wikipedia is editable by anyone, and there is no mandatory style of referencing, partly because of battles, long long ago, you cannot expect to get 100% of the information you seek with a naive parser, however you can get a very high percentage, and the more effort you put in, the higher the percentage will be.

You can find code to help you in pywikibot (python) AWB (C#) and in various other places, such as Anomie's excellent perl bot code [https://en.wikipedia.org/wiki/User:AnomieBOT/source].

I would suggest any language you use have PCRE available, if you want to craft your own tools.

Also be aware that you can download XML dumps of Wikipedia articles in any language. If you want to get the references for a very large number (or all) articles, this will be a resource saver.

You should also look at Wikidata, which has an extensive bibliographic coverage of academic papers.

You can also look at various cite completion tools for code to borrow. Refill (https://en.wikipedia.org/wiki/User:Zhaofeng_Li/reFill) is a php tool that fills in missing details in references. Citation bot (https://en.wikipedia.org/wiki/User:Citation_bot) is also written in php, and has I believe a git repository.

If you have specific questions, you can ask me on my Wikipedia talk page, or ask other at any of the usual places (https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical), https://en.wikipedia.org/wiki/Wikipedia:Help_desk or Wikipedia:Bots/Noticeboard). Most people are pretty friendly.

Improve this answer
1

Properly formatted references include COinS metadata (which does require HTML parsing, but at least it's a standardized format).

Improve this answer
0

If you know how to code with Python, perhaps this script might be helpful to you.

As for the links (but not for article titles and dates), one can use the MediaWiki API's prop=externallinks function, as in:

https://en.wikipedia.org/w/api.php?action=parse&page=2017_Equifax_data_breach&prop=externallinks

As Tgr wrote, the references should be formatted in a structured way; if you only wish to extract that formatted Wiki text, use prop=wikitext on the basis of which you could try parsing each date and article title:

https://en.wikipedia.org/w/api.php?action=parse&page=2017_Equifax_data_breach&prop=wikitext

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.