First of all, if you want the page content, the best way to do this isn't by scraping, but by processing one of the database dumps - you can do this locally, and it'll allow you to get the raw page content rather than rendered HTML.
To identify the pages, there are a number of possible options.
First, categories - these are comparable to index pages, but dynamically generated. Taking the example of Anatole Abang, the first man on your MLS list, scroll to the bottom of the page and you'll see the article is in categories such as "Cameroonian footballers", "Major League Soccer players", etc. The category system is a hierarchical tree, and they should all be underneath "Association football players".
Unfortunately, the category tree is a little messy. If you just took everything downstream of this category, you'd pick up (among others) every fictional footballer Wikipedia knows about. A good compromise is to use something like "Association football players by club" and work downstream from there to a depth of perhaps four categories (the UK ones are pretty deeply nested).
If all you want to do is get page titles/URLs, you can use a tool like catscan or (preferably) quick-intersection for this; be prepared to wait a while for it to process everything, but it'll give you a useful html/csv/etc file. You will still get some entries that shouldn't be there, but scraping would have the same problem!
Secondly, wikidata. Wikidata is the structured-data "spine" to Wikipedia; almost every article in every language has a corresponding data element, which for Abang is Q19594142. In theory, you should be able to query Wikidata to pull out, for example, - a) every person with occupation "footballer"; b) every person with a FIFA player ID; c) every person who was a member of a team which itself was a football team; ...
Unfortunately, as of the time of writing it looks like this data has not yet been fully imported for sports players, and so it won't give you a comprehensive answer. But if you're someone reading this reply in 2017, take another look :-)