12

I'd like to grab as many soccer players names and their Wikipedia URL. For example:

"David Beckham": 'David_Beckham',
"Wayne Rooney": 'Wayne_Rooney'

The first portion is their name and the second is the extension for their Wikipedia page.

I've found a few lists like this: https://en.wikipedia.org/wiki/List_of_current_Major_League_Soccer_players

The URL above lists a few hundred/thousand names, and I suppose I could scrape it, but is there a better method since not every league has a list like that? Perhaps there is an already available index?

Improve this question
4
  • 1
    For your and anyone elses reference, wikipedia's policy on scraping en.wikipedia.org/wiki/… It's more permissive than it sounds.
    – Frames Catherine White
    Commented Jun 14, 2015 at 5:14
  • @Oxinabox Noted. If scraping isn't allowed, then surely there has to be another way of extracting such information from Wiki...
    – Lara
    Commented Jun 14, 2015 at 5:47
  • Read the linked policy page. It provides suggestions. It's not really strictly not allowed. As i said it is more permissive than it sounds.
    – Frames Catherine White
    Commented Jun 14, 2015 at 5:49
  • Since not all players may show up on such lists, perhaps you could also enhance your list by downloading the entire Wiki database and searching every page for the term footballer or better yet, the use of templates like {{Infobox footbal biographyl}}. No idea if that'd be even near feasible, but seems the way I would seek.
    – JeopardyTempest
    Commented Dec 5, 2016 at 6:56

4 Answers 4

Reset to default
6

First of all, if you want the page content, the best way to do this isn't by scraping, but by processing one of the database dumps - you can do this locally, and it'll allow you to get the raw page content rather than rendered HTML.

To identify the pages, there are a number of possible options.

First, categories - these are comparable to index pages, but dynamically generated. Taking the example of Anatole Abang, the first man on your MLS list, scroll to the bottom of the page and you'll see the article is in categories such as "Cameroonian footballers", "Major League Soccer players", etc. The category system is a hierarchical tree, and they should all be underneath "Association football players".

Unfortunately, the category tree is a little messy. If you just took everything downstream of this category, you'd pick up (among others) every fictional footballer Wikipedia knows about. A good compromise is to use something like "Association football players by club" and work downstream from there to a depth of perhaps four categories (the UK ones are pretty deeply nested).

If all you want to do is get page titles/URLs, you can use a tool like catscan or (preferably) quick-intersection for this; be prepared to wait a while for it to process everything, but it'll give you a useful html/csv/etc file. You will still get some entries that shouldn't be there, but scraping would have the same problem!

Secondly, wikidata. Wikidata is the structured-data "spine" to Wikipedia; almost every article in every language has a corresponding data element, which for Abang is Q19594142. In theory, you should be able to query Wikidata to pull out, for example, - a) every person with occupation "footballer"; b) every person with a FIFA player ID; c) every person who was a member of a team which itself was a football team; ...

Unfortunately, as of the time of writing it looks like this data has not yet been fully imported for sports players, and so it won't give you a comprehensive answer. But if you're someone reading this reply in 2017, take another look :-)

Improve this answer
5
  • Thanks for the thorough explanation of everything. That certainly helped. I did in fact find a list that is quite full of players, though, not ALL players (for instance there are no Russian teams), it's certainly a good start. en.wikipedia.org/wiki/…. You also mention raw page content etc, I'm using the Wikipedia api (en.wikipedia.org/w/…) because I really did want the rendered HTML. Is that an effective way of doing it? Thanks for your comment.
    – Lara
    Commented Jun 14, 2015 at 13:46
  • 1
    Andrew, I think scraping Wikipedia is a very popular problem here and elsewhere (precisely because direct scraping is a bad idea). Why don't you use this answer to make a 'self-answered' question about a general way of extracting structural information from Wikipedia? The others may add more methods. And such a question would be very popular here.
    – Anton Tarasenko
    Commented Jun 14, 2015 at 14:32
  • 1
    @Lara If you do really need the rendered version, sure, that's probably your best bet. If you're going to use it for a live service, though, please make sure to cache a copy locally rather than hitting the API each time!
    – Andrew is gone
    Commented Jun 14, 2015 at 19:48
  • @Lara (2): the lists are useful if they exist, but they're hand-made and so will be more likely to be incomplete/patchy/outdated (eg the Russian gaps). They may also contain "mistaken links" - someone writing [[John Smith]] without checking that the linked page is the footballer JS not the writer JS, or pages being moved without fixing the backlinks. So a slight note of caution...
    – Andrew is gone
    Commented Jun 14, 2015 at 19:51
  • @Anton: I'm not sure I know enough about it in general, to be sure. Note how vague I am about the actual scraping/category-tree construction - I know it can be and has been done, but I'd get pretty vague on the mechanisms :-). This answer mostly focuses on the selection rather than extraction, but I suppose that might be a suitable question topic.
    – Andrew is gone
    Commented Jun 14, 2015 at 19:53
4

By using WikidataQuery API you can get all items which have occupation (P106) soccer player (Q937857) and link to English Wikipedia: enwiki

http://wdq.wmflabs.org/api?q=claim[106:937857] AND link[enwiki]

Currently 134,037 soccer player Wikidata items are linked to enwiki.

After that you can use MediaWiki API for Wikidata with wbgetentities to get the title and the link for each one of these items:

https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q354|Q379|Q615|Q624|Q1634|Q1835|Q1839|Q1879&props=labels|sitelinks&languages=en&sitefilter=enwiki

where Q354|Q379|Q615|Q624|Q1634|... are the items from the first request, props=labels|sitelinks is to gets the titles and the links, and languages=en&sitefilter=enwiki are respectively their wikies (in our case English Wikipedia).

For each item the result will include this information:

"Qxxx": {
    "labels": {
        "en": {
            "value": "Item name"
        }
    },
    "sitelinks": {
        "enwiki": {
            "title": "Wikipedia link"
        }
    }
},

where "Item name" is the first portion from your example (the name) and "Wikipedia link" is the second one.

Improve this answer
3

To get a long list of soccer players who where born TODAY, ordered by birthdate , youngest first,

Go to http://dbpedia.org/sparql and enter this query into the big text box:

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX dbo: <http://dbpedia.org/ontology/>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>


SELECT distinct ?birthdate ?thumbnail ?player ?name ?description  WHERE {
?player rdf:type dbo:SoccerPlayer ;
           dbo:birthDate ?birthdate ;
           rdfs:label ?name ;
           rdfs:comment ?description 
 FILTER ((lang(?name)="en")&&(lang(?description)="en")&&(STRLEN(STR(?birthdate))>6)&&(SUBSTR(STR(?birthdate),6)=SUBSTR(STR(bif:curdate('')),6))) .
 OPTIONAL { ?player dbo:thumbnail ?thumbnail . }
} ORDER BY DESC(?birthdate)

and click on the "Run Query" button.

The resultset is a table that will look somewhat similar to this (ad blockers or modern browsers might refuse to render this page, though. Click on button "unsafe reload" at top of page).

To get wikipedia (not dbpedia) pages displayed in the table, download it, replace all occurences of "http://dbpedia.org/resource/" with "http://en.wikipedia.org/wiki/" in the text.

To get a longer list of all players, play with the "birthdate" stuff in the FILTER clause of the query.

This:

FILTER ((lang(?name)="en")&&(lang(?description)="en")&&(STRLEN(STR(?birthdate))>6)) .

would get all players, but dbpedia has a max execution time and resultset length-limit, so this won't get you all players in one fell swoop.

Improve this answer
1

If you build a "list of list" of URLs or page names that contain names of soccer players, then you can either use the database dumps or an API-like tools that the Wikipedia community offer, to list just a few:

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.