Given a list of medical terms, I want to identify and remove the general (less granular) terms.
For example consider the below mentioned word list:
- kartagener s syndrome - disease treatment - human - blood - ischaemia - acute respiratory distress syndrome - hospital - adrenergic blocking drug - finger - symptom - therapy - extracorporeal membrane oxygenation - hand
I mean word such as;
- human - blood - disease treatment - hospital - finger - symptom - therapy - hand
as general/less granular terms.
I tried to detect these terms using the following two statistical measures.
- frequency: i.e. I assumed the more frequent the word in the corpus is, the less granular it is
- IDF (Inverse Document Frequency)
However, since they are only statistical measures and does not consider the meaning (semantics) of the word, it worked poorly in my dataset.
I tried to use wikipedia categories to filter these terms. However, I could not find a proper way of using categorical details to facilitate my problem. Therefore, I am wondering if there is any way to detect general terms using wikipedia (or its variants such as DBpedia, Wikidata etc.)
For those, who would like to check a long list of concepts, I have attcahed a long list herewith: https://docs.google.com/document/d/1BYllMyDlw-Rb4uMh89VjLml2Bl9Y7oUlopM-Z4F6pN0/edit
NOTE: I am not expecting the solution to work 100% (if the proposed algorithm is able to detect many of the general concepts that is enough for me)
I am happy to provide more details if needed.