11

Wikipedia is a significant source of data. Data from Wikipedia may be available in various formats via other servers (e.g., SPARQL end point). For a task of extracting all pairs of "ATC code" - "generic or brand drug name" For example data on this page

http://en.wikipedia.org/wiki/Rosiglitazone

what server and what technologies could one use?

Example data shown graphically: example

15

Project DBpedia is a crowd-sourced community effort to extract structured information from Wikipedia.

Drugs and Chemicals infoboxes are available in structured form, already. Check out http://dbpedia.org/page/Rosiglitazone

10

Wikipedia offers two interesting ways to get its own information:

Images and uploaded files are stored elsewhere, also downloadable

This is a XML file from the page requested using special export, the Wikipedia marckup specification is very nice for scripting. For the particular case of the ATC code (I'm assuming it is always ATC_prefix immediately followed by ATC_suffix) I did:

grep -e 'ATC' Wikipedia_file.xml | awk '{print $4}' | tr -d '\n'

PS: I'm oversimplifying because this file only have a single page, for many more it is not too different.

Anyway, the "infoboxes" tables are always at the beginning of the page and it is well structured.

8

Extracting data from Wikipedia infoboxes will not be necessary anymore in the not-too-distant future: The people at Wikipedia are currently working on a new project called Wikidata.

Wikidata is a free knowledge base that can be read and edited by humans and machines alike. It is for data what Wikimedia Commons is for media files: it centralizes access to and management of structured data, such as interwiki references and statistical information. Wikidata contains data in every language supported by the MediaWiki software.

The data for your Rosiglitazone example will be located at https://www.wikidata.org/wiki/Q424771

Currently, it only lists very rudimentary data as well as the links to all language versions of the Wikipedia entry for Rosiglitazone. However, it is only a matter of time until the Wikidata entry will get expanded to include additional relevant data that currently resides in the infobox. You can look at the project progress regarding infoboxes data at https://www.wikidata.org/wiki/Wikidata:Phase_2

  • 3
    promising project but doesn't answer the question – RSFalcon7 May 9 '13 at 6:10
  • @RSFalcon7, that really depends on how soon the data is needed: The more people get engaged with Wikidata, the sooner extracting data from Wikipedia infoboxes will become a thing of the past. – Patrick Hoefler May 9 '13 at 13:52
  • 2
    It doesn't answer the question yet. – ojdo Apr 29 '14 at 15:35
3

Wikipedia has an API to fetch raw content of the articles - http://www.mediawiki.org/wiki/API:Main_page

There are plenty of libraries to interact with it, if you use Python I can recommend -https://github.com/callison-burch/wikipydia

It seems that data you look for is standardized as {{drugbox}}/{{chembox}} template - http://en.wikipedia.org/wiki/Wikipedia:WikiProject_Chemicals/Chembox_validation

Basically you need to fetch articles for drugs you interested in via API and then parse HTML/Wikipedia markup to get drugbox/chembox element and your data is inside. Here is format for Chembox - http://en.wikipedia.org/wiki/Template:Chembox looks like it should be fairly easy to parse.

You can run such script on server or on your own machine, depending on volume/number of articles you want to extract and process.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.