1

Given a list of medical terms, I want to identify and remove the general (less granular) terms.

For example consider the below mentioned word list:

 - kartagener s syndrome 
 - disease treatment
 - human 
 - blood 
 - ischaemia
 - acute respiratory distress syndrome
 - hospital 
 - adrenergic blocking drug 
 - finger 
 - symptom 
 - therapy
 - extracorporeal membrane oxygenation 
 - hand

I mean word such as;

 - human
 - blood
 - disease treatment
 - hospital
 - finger
 - symptom
 - therapy
 - hand

as general/less granular terms.

I tried to detect these terms using the following two statistical measures.

  • frequency: i.e. I assumed the more frequent the word in the corpus is, the less granular it is
  • IDF (Inverse Document Frequency)

However, since they are only statistical measures and does not consider the meaning (semantics) of the word, it worked poorly in my dataset.

I tried to use wikipedia categories to filter these terms. However, I could not find a proper way of using categorical details to facilitate my problem. Therefore, I am wondering if there is any way to detect general terms using wikipedia (or its variants such as DBpedia, Wikidata etc.)

For those, who would like to check a long list of concepts, I have attcahed a long list herewith: https://docs.google.com/document/d/1BYllMyDlw-Rb4uMh89VjLml2Bl9Y7oUlopM-Z4F6pN0/edit

NOTE: I am not expecting the solution to work 100% (if the proposed algorithm is able to detect many of the general concepts that is enough for me)

I am happy to provide more details if needed.

  • 1
    w.wiki/3Rm – Stanislav Kralin Apr 28 at 19:04
  • 1
    Can you state why "finger" is in your list of less granular terms? I don't see a more specific concept. Also, your question states that wikipedia/data/dbpedia have to be used. While these are great resources, you may get more complete answers by using tools that use the UMLS, BioPortal, or Open Bio Ontologies (OBO). – cmungall Apr 29 at 1:36
  • 1
    Would it be possible for you to create a corpus of the general terms (I don't think it should be too big), if yes then the issue will automatically be resolved, the issue is at this point I am not certain what constitutes as a general term. Stainslav Kralsin's solution does look promising, but I haven't really used WQS, so I don't really know how that works, my suggestion would be to not use AI in this, rather create a corpus of general words and then use a regex approach. – anand_v.singh Apr 29 at 6:23
  • @StanislavKralin I was able to understand most of the parts of your code. However, I am not clear what is meant by ?s, ?p in ?s ?p ?item .. Could you please let me know what is denoted using this line? I look forward to hearing from you :) – EmJ Apr 29 at 13:13
  • 2
    The query just count the number of incoming links. Terms that are often "referenced" could be considered as "general". – Stanislav Kralin Apr 29 at 19:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.