20

On 18. April 2016 the ticket "More efficient SPARQL queries for sitelinks" brought us a better way: schema:isPartOf. ?sitelink schema:about ?item . ?sitelink schema:inLanguage "en" . ?sitelink schema:isPartOf <https://en.wikipedia.org/> . Your full query now looks like this: PREFIX schema: <http://schema.org/> PREFIX wikibase: <http://...


16

Try this one: prefix schema: <http://schema.org/> PREFIX wikibase: <http://wikiba.se/ontology#> PREFIX wd: <http://www.wikidata.org/entity/> PREFIX wdt: <http://www.wikidata.org/prop/direct/> SELECT ?cid ?country ?article WHERE { ?cid wdt:P31 wd:Q3624078 . OPTIONAL { ?cid rdfs:label ?country filter (lang(?country) = "...


15

Project DBpedia is a crowd-sourced community effort to extract structured information from Wikipedia. Drugs and Chemicals infoboxes are available in structured form, already. Check out http://dbpedia.org/page/Rosiglitazone


11

The dependency is supposed to be the opposite, i.e. that Wikipedia should rely on data in Wikidata. This is however not fully implemented yet. If you look at the infobox on this cheese in French, all data comes from Wikidata, but not in English (yet). We are far from full implementation, so expect to see any combination of people pulling data from Wikipedia ...


10

Wikipedia offers two interesting ways to get its own information: Complete database dumps in XML and SQL, as you wish. Special export very nice XML files downloadabe from only the categories that you specify. Images and uploaded files are stored elsewhere, also downloadable This is a XML file from the page requested using special export, the Wikipedia ...


10

You can use Google Spreadsheets ImportHTML formula as detailed in this Liberating HTML Tables (using Google Spreadsheet) tutorial on School of Data by Tony Hirst - it includes a specific walk-through for Wikipedia. The essence is to do: =importHTML("","table",N) In your case you could try: =importHTML("http://en.wikipedia.org/wiki/...


8

If you want to do this from the dumps, you can use the wb_items_per_site dump (current version, 625 MB; most recent dumps) For each Wikidata item, this contains rows with the corresponding page name on a given wiki project. For individual articles, you can use &action=info on Wikipedia as Ainali suggests, but you have to use the /w/index.php?title=...


8

No, there are no SQL versions of the XML dump files. The page Data dumps/Tools for importing on meta.wikimedia.org describes how to work around that: You can either use ImportDump.php to import the XML file directly (apparently suitable only for small wikis), or you can use a tool like mwdumper to convert the XML into SQL and then import that.


8

Extracting data from Wikipedia infoboxes will not be necessary anymore in the not-too-distant future: The people at Wikipedia are currently working on a new project called Wikidata. Wikidata is a free knowledge base that can be read and edited by humans and machines alike. It is for data what Wikimedia Commons is for media files: it centralizes access to ...


6

Right now, there is no good solution. The options I can think of are: Just download the normal dump whenever it becomes available. This means you won't get daily updates at all. For the English Wikipedia, a new dump is generated about once a month. Use the adds/changes dumps (already mentioned by ojdo). There are two problems with using this: It doesn't ...


6

The Wikipedia Special Export feature does exactly this. More details in this answer.


6

You could also use the API. The call https://en.wikipedia.org/w/api.php?action=query&prop=pageprops&format=json&titles=Tom%20Hanks will give you the id on Wikidata (as "wikibase_item").


6

You can also use the SPARQL service with query like: prefix schema: <http://schema.org/> SELECT * WHERE { <https://en.wikipedia.org/wiki/Tom_Hanks> schema:about ?item . }


5

First of all, if you want the page content, the best way to do this isn't by scraping, but by processing one of the database dumps - you can do this locally, and it'll allow you to get the raw page content rather than rendered HTML. To identify the pages, there are a number of possible options. First, categories - these are comparable to index pages, but ...


5

The wikitextparser Python module has good support for converting tables into structured formats. Example: import wikitextparser as wtp import requests import json r = requests.get('http://en.wikipedia.org/w/index.php?title=' + 'List_of_country_calling_codes&action=raw') wt = wtp.parse(r.text) print(json.dumps({'root': wt.tables[1].getdata()[0:3]}, ...


5

Using OpenRefine (previously Google Refine) you can easily convert the wikipedia table to a JSON document. The following video will walk you through the steps to clean the wikipedia table, then using the template exporter you will be able to define the JSON format of your output document. Working with OpenRefine will also allow you to extend your data ...


5

Using this SPARQL end-point: http://dbpedia.org/sparql you can enter this query: SELECT DISTINCT ?YA ?A ?YB ?B where { ?A dbo:influenced ?B. ?A a dbo:Philosopher. ?B a dbo:Philosopher. ?A dbo:birthYear ?YSA. ?B dbo:birthYear ?YSB. BIND( SUBSTR( xsd:string(?YSA), 1, 4) AS ?YA ) BIND( SUBSTR( xsd:string(?YSB), 1, 4) AS ?YB ) } LIMIT 10000 which ...


5

Wikidata doesn't automatically use data from Wikipedia. Data from Wikipedia is uploaded to Wikidata by hand or using some more or less automated tools to do batch uploads but ultimately under control of a user, as PetScan or Wikidata Game. The uploading work is in progress with thousands of articles in Wikipedias without an item in Wikidata and millions of ...


4

If you are using google chrome and google spreadsheet, there is a nice extension (still in beta development, but for the tables you mentioned it worked well) called scraper: https://chrome.google.com/webstore/detail/scraper/mbigbapnjcgaffohmbkdlecaccepngjd If you have the extension installed, just select one row of the desired table, rightclick on the ...


4

There exists an experimental Wikimedia adds/changes dump service. Another approach is generating the differential dump yourself using deltapack, discussed about in this thread in the Wikitech mailing list.


4

According to v0.4 of the Wikidata inclusion syntax, this feature has not been implemented yet. The progress of this feature is available in the Wikimedia bug tracking system.


4

As far as I can see, there is no link from the category Japanese people to the article John Andru (at least based on dump from 1 October 2013), so I'm not sure why did the tool tell you otherwise. But the category structure is pretty messed up. For example, John Andru is in the category 18th-century German writers, through chain of categories like this: ...


4

Since the 2016 season is about to start I figure I'll update this to hopefully help some others that stumble upon this page. The best resource for getting free and comprehensive nfl statistics is the nfldb and nflgame python APIs. The developer pulls the information directly from json feeds used by NFL.com's gamecenter data, which is about as accurate and ...


4

SELECT ?itemLabel ?population ?date { ?item wdt:P31 wd:Q50337; p:P1082 [pq:P585 ?date; ps:P1082 ?population] FILTER NOT EXISTS {?item p:P1082 [pq:P585 ?date_] FILTER (?date_ > ?date)} SERVICE wikibase:label { bd:serviceParam wikibase:language "en". } } Try it If you really need GROUP BY: SELECT ?itemLabel ?date ?population { { SELECT ?item ...


3

By using WikidataQuery API you can get all items which have occupation (P106) soccer player (Q937857) and link to English Wikipedia: enwiki http://wdq.wmflabs.org/api?q=claim[106:937857] AND link[enwiki] Currently 134,037 soccer player Wikidata items are linked to enwiki. After that you can use MediaWiki API for Wikidata with wbgetentities to get the ...


3

It wouldn't give you complete updates, but there is also dbpedia live. Dbpedia only covers the stuctured text boxes in the wikipedia entries. It is monitoring the changes on Wikipedia though.


3

According to DBpedia as Tables, DBpedia uses the following data to create the tables: Each instance is described by its URI, an English label and a short abstract, the mapping-based infobox data describing the instance (extracted from the English edition of Wikipedia), geo-coordinates, and external links. DBpedia basically provides two different types of ...


3

You could use Wikidata property instead of Wikipedia link, to make it more usable by other languages or other projects.. For example https://www.wikidata.org/wiki/Q859221 represents Java Swing. Perhaps you could propose a new Property at https://www.wikidata.org/wiki/Wikidata:Property_proposal/Generic and import your mapping directly on Wikidata


3

And just to add to the others, now it is available on English Wikipedia, and you would use it like this: {{#property:P1096|from=Q313}}


Only top voted, non community-wiki answers of a minimum length are eligible