I wonder how PubMed obtains papers' metadata.

In particular, I am interested in the metadata used for these filters: http://www.ncbi.nlm.nih.gov/pubmed?term=%22Review%22[pt]

enter image description here

  • This seems like a rather specific question for the creators of NCBI/PubMed. I'd consider tweeting at them at @NCBI and/or emailing them at info@ncbi.nlm.nih.gov and reporting back to answer this for the rest of us :) – Mark Silverberg Jun 27 '16 at 11:35

The Frequently Asked Questions about Indexing for MEDLINE give some interesting insights:

How is MEDLINE produced?

MEDLINE is the product of many information specialists at the National Library of Medicine: serials librarians who obtain journal subscriptions and check in the individual journal issues; data review specialists who assure the quality of all supplied and created bibliographic citations; biomedical subject specialists who analyze the subject content of articles and index the concepts that are discussed, using the Medical Subject Headings (MeSH) controlled vocabulary; and computer and information specialists who develop and maintain the various systems, including the retrieval system. Every journal issue and article cited in MEDLINE has been reviewed and inspected by many individuals.

Do you scan in the citations?

Bibliographic citations in MEDLINE are currently created by two different methods. Many journal publishers supply NLM with citation and abstract data electronically, tagged with XML (Extensible Markup Language) codes to identify different data elements. Article citations from print journals for which data are not supplied are created with a combination of optical character recognition (OCR) and keystroking. Citations that are received electronically may still require a considerable amount of human intervention to add supplementary data and to make other data uniform.

As an author, how should I select keywords so that my article is retrieved in MEDLINE?

The National Library of Medicine does not use author-assigned keywords for MEDLINE indexing. The terms that are used are descriptors from MeSH (Medical Subject Headings) – the National Library of Medicine’s controlled vocabulary thesaurus. MESH includes more than 27,000 descriptors, most of which may be qualified with one or more of over 80 MeSH subheadings. In addition, indexed terms may be derived from the more than 200,000 headings called Supplementary Concept Records (SCRs); these records (representing chemicals and rare diseases) exist within a separate thesaurus and are mapped to MESH descriptors. NLM’s policy for indexing is to use the most specific terms possible which best describe the content and substance of the indexed item.

Since early 2013, PubMed has displayed publisher-supplied keywords in the KEYWORDS field of the abstract display. Authors who wish to supply those keywords using the MeSH vocabulary can consult the MeSH Browser (available at http://www.nlm.nih.gov/mesh/MBrowser.html). They may also use a tool called MeSH on Demand (available at http://www.nlm.nih.gov/mesh/MeSHonDemand.html) that identifies MeSH terms in text using the NLM Medical Text Indexer program. After processing the text, MeSH on Demand returns a list of MeSH terms relevant to the text that was input.

If authors wish their articles to be retrieved by their preferred terminology, they should ensure that these words appear in the title or abstract, where they will be searchable as text words.

What types of articles are selected to be indexed?

NLM indexes only the substantive content of journals selected for MEDLINE indexing. In journals selected for cover-to-cover indexing, for instance, NLM does not index items such as book reviews or abstracts.

Furthermore, a number of journals are multidisciplinary and publish articles about non-life science fields such as astronomy, geology, mathematics, and the like. Other journals cover many aspects of a subject field, including non-life science aspects (e.g., chemical journals may contain articles about physical chemistry as well as biochemistry). Because NLM indexes only articles that pertain to the life sciences, articles in these selectively indexed journals that do not pertain in any way to the life sciences are considered “out of scope” and appear as unindexed citations in PubMed.

What is a Publication Type? How does it differ from descriptors?

> Since 1991, Publication Types (PTs) have been assigned by NLM indexers to describe the form of presentation of materials that are indexed. Some PTs reflect the format and editorial practices of the individual journal; others reflect the indexer's analytical judgment. Items indexed prior to 1991 have had a limited number of PTs added by machine rather than by re-indexing.

PTs such as "letter" or "editorial" are determined by the style or section heading used by the journal. "Letter" is assigned to routine letters to the editor as well as to lengthier articles if the journal labels them as "scientific correspondence" or similar caption. Likewise, some journals label lead articles as "editorial," while others reserve this caption for expressions of opinion. NLM follows the journal's practice and assigns the PT "editorial" rather than "journal article" in both these cases. A "classical article" is one that has been reprinted by a journal because of its significance; NLM makes no independent determination of an article's "classical" status.

A published item may be assigned more than one PT. The default PT is "journal article"; however, a journal article may also be a "review" or a "randomized controlled trial." Certain PTs never co-occur. For instance, an item cannot be both a journal article and letter, editorial, or news item.

PTs should not be confused with Main Headings (MHs) or descriptors, although many PTs have a corresponding MH. PTs describe the form of an article; MHs describe its contents.

Can NLM add recently adopted new terminology to my article published several years ago?

NLM does not re-index articles as new terms are added to the MeSH vocabulary. New terms are transparently linked to existing terms as they are added so that searchers using the new terms will retrieve the older citations.

What online journals are indexed for MEDLINE? What journals have online versions?

Electronic, or online, journals come in two basic types: journals that are online only and those that are published both in online and print formats. Of the latter, many titles have identical content in both formats; some have content unique to the electronic medium. The ISSN that displays on the MEDLINE citation reflects the version of the title from which NLM indexes the issue (print or online). Currently, about 81% of journals indexed for MEDLINE are indexed from the online version; a list of these journals is available at http://www.nlm.nih.gov/bsd/journals/online.html.

Who are the indexers, and what are their qualifications?

> Most MEDLINE indexers are either Federal employees or employees of firms that have contracts with NLM for biomedical indexing. A prospective indexer must have no less than a bachelor's degree in a biomedical science. A reading knowledge of certain modern foreign languages is typically sought. An increasing number of recent recruits hold advanced degrees in biomedical sciences. Federal employees must be United States citizens, but citizenship is not mandatory for contractors.

Indexers are trained in principles of MEDLINE indexing, using the Medical Subject Headings (MeSH) controlled vocabulary as part of individualized training. The initial part of the training is based on an online training module (partially available to the public at http://www.nlm.nih.gov/bsd/indexing/index.html), followed by a period of practice indexing. NLM does not accept other indexing training programs as a substitute.

> About 1% of MEDLINE indexing is performed by indexers at the International MEDLARS Centers in Sweden and Brazil.

How can I become an indexer?

Indexing vacancies at the National Library of Medicine are filled competitively. Any such vacancy is listed on NLM's Web site. Indexers generally are classified as Technical Information Specialists, in the GS-1412 job series. The normal career ladder goes from GS-9 to GS-12.

For information about applying for work as an indexer with NLM's contractors, please contact the Index Section for a listing of all firms with a current indexing contract.

I've heard that I can do indexing work at home. How do I apply for this type of position?

Contract indexers work from their homes once their training has been completed. Indexers who are Federal employees may qualify to telework for two or three days each week under the NIH Telework Program. See the preceding paragraph for information on applying for either type of indexing position.

How can I apply for a job as an abstract writer for MEDLINE?

> No original abstracts are created for MEDLINE. Author abstracts in English published in a journal are input for MEDLINE.

How can I apply for a job as a translator with the Index Section?

> Although journals from many languages are represented in MEDLINE, the National Library of Medicine does not translate the articles. Indexers who perform subject analysis of the articles typically have a reading knowledge of scientific terminology in one or more foreign languages. They read and comprehend the articles, but they do not need to translate them in order to index them.

Regarding MESH terms, it's highly possible that the human indexers uses the NLM Medical Text Indexer (MTI) for suggestions.

enter image description here

Use of MeSH in Indexing contains a bit more information but I wish the reference paper was a bit more recent than 1978.

The MeSH vocabulary is designed for use by NLM to index the MEDLINE database of journal citations and other media, and to search the MEDLINE data using PubMed. The following outlines how MeSH is used by indexing of the MEDLINE journal articles. For more detailed information see Bachrach and Charen (1). See also Use of MeSH in Online Retrieval and Use Of Medical Subject Headings For Cataloging.

  1. Combining specific MeSH Descriptors to index complex subjects: Co-ordination

Rather than creating a MeSH Descriptor for every subject that might discussed in a journal article, multiple MeSH Descriptors or Qualifiers are combined to index the desired subject. For example, jejunitis may be expressed by the use of the Descriptors Jejunum and Enteritis. Similarly, MeSH Qualifiers can be used in conjunction with appropriate Descriptors. For example, a deficiency of monoamine oxidase may be indexed by the Descriptor Monoamine Oxidase, combined with the Qualifier /Deficiency. This combining procedure is called "coordinated" indexing.

Note that not all complex subjects are indexed by co-ordination. When a particular complex subject occurs frequently, a "pre-coordinated" Descriptor may be created. For example, for a subject of arm injuries, instead of combining the Descriptor Arm with the Qualifier /Injuries, the single Descriptor Arm Injuries is used.

  1. Multiplicity

Articles may discuss multiple subjects in which case indexers supply Descriptors for all subjects. For example, "an article in an endocrinology journal does not relieve the indexer of covering the non-endocrine aspects of the text." (Bachrach and Charen, p. 25)

  1. Specificity

Indexers generally use the most specific MeSH subject heading available, rather than each of the broader subjects in which a searcher may be interested. For example, an article about pulmonary pathology is indexed under the Descriptor Lung rather than the more general Respiratory System. The reasons for indexing to the most specific are: (a) it records more information, and (b) it avoids multiple, redundant, indexing since the broader subjects are linked to the more specific subjects by the hierarchical MeSH Tree Structures. For example, in PubMed a search for Respiratory System by default includes articles indexed to Lung and other more specific Descriptors.

(1) Bachrach, C. A. and Charen, Thelma. "Selection of MEDLINE contents, the development of its thesaurus, and the indexing process." Med Inform (Lond). 1978 Sep;3(3):237-54. Note especially "Indexing principles, pp. 250ff.

MTI First Line (MTIFL) Indexing:

MTI First Line (MTIFL) indexing partially automates the standard indexing process at the US National Library of Medicine. MTIFL provides the initial indexing for a citation. A human indexer then reviews this indexing and modifies it as required by adding any missed terms, removing any incorrect terms, and supplying Publication Types. The process of the human curation of MTIFL results is called MTIFL Completion.

In 2010, the Indexing Initiative team and the NLM Index Section conducted a series of three experiments with MTI to determine the feasibility of using MTI recommended MHs as first-line indexing for selected subject areas. Journals for the three experiments were chosen from fields where MTI was performing well (for example, Microbiology, Anatomy, Botany, and Medical Informatics). The experiments measured the accuracy of MTI indexing and the amount of time required to index and revise both the manual and MTI First Line (MTIFL) Completion. The results of the experiments showed that MTIFL Completion was successful given the right circumstances, namely journals with a low potential for the need of manually created chemical flags and GeneRIFs that are normally added by the indexer. In the case of MTIFL, the burden of creating the chemical flags and GeneRIFs would shift to the reviser which would be time consuming and undesirable.

In February 2011, fourteen journals were included in a pilot evaluation of MTIFL Completion. Since that time, almost 400 journals have been selected for indexing by MTIFL Completion, and new journals are added each month. The process of evaluating journals for indexing by MTIFL Completion is ongoing and dynamic, with journals added and removed based on continued refinement of the evaluation criteria and experience with MTIFL Completion results. One outcome of the MTIFL experiments was that the timing information showed it took indexers longer to remove wrong MTI recommendations than to add missing ones. In talking with the indexers, the reason for this extra time when removing a bad recommendation is that they have to take time and decide if they missed something in the article or not before removing it. So, MTIFL journals are processed with MTI's Balanced Recall/Precision Filtering option providing a smaller, more precise indexing list than with the regular MTI processing. The average F1 measure increases by 0.1252 (2015) when journals are incorporated into the MTIFL program. This increase is likely due to the extra filtering and indexing policies specific to MTIFL Completion.


enter image description here

I would be interested to see the F1, recall, and precision

Also of interest:

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.