I'm currently working on a project for University and need to source a free and open database of scientific research papers to work on, the larger the better. It also needs to contain bibliographic data as we want apply a version of PageRank and various clustering algorithms on the data and for this we will need the paper citations.

So, are there any known complete databases we may be able look at? I've been looking for a while now but cant find anything immediately obvious.

If not, which is the more likely scenario:

We need to scrape together many existing databases on small subsets of scientific communities to create a larger more complete one?


There may be a online database with some nice APIs we could query for this task?

Any other useful scenarios?

I have no experience in a task such as this, so any tips is greatly appreciated, such as any future road bumps we could hit if we take a particular path such as querying APIs.

8 Answers 8


CiteSeerX and DBPL are two of the most commonly used data sets used in the academic literature.


The the arXiv repository contains almost 1 million of scientific articles, whose bibliographic information is accessible through a documented API, including links to other papers.


There is also PLOS Public Library of Science http://api.plos.org/solr/faq/#solr_api_examples PLOS is open access.


Pubmed has data about 23 million biology and medicine papers. Unfortunately I think you need to request access and given US politics at the moment it could take some time till there a human available to give you that access.

  • 1
    There are a subset of Pubmed papers, at PubMed Central (PMC), that are all open, and in full-text access. Other subset is at SciELO, with equal openess... See explanation. Nov 29, 2015 at 16:50

EDIT 2017

PubMed Central changed all adressess and some methods.

Only tips:

  • To bulk access you can use complex FTP procedures similar to the described below in the "OLD" section... Or a simple perl (or shell) script to loop over API.

  • The basic GET API are showed at https://www.ncbi.nlm.nih.gov/pmc/tools/get-full-text/

    • To GET by shell you must to use Accept: application/xml!

    • Example: curl -s -H 'Accept: application/xml' 'https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=pmc&id=PMC3970601

    • To list all IDs (PMC-ID) that you need in a complex query, see a good tutorial about complex-query APIs https://www.ncbi.nlm.nih.gov/books/NBK25499/

    • To insert XML into a database like PostgreSQL, not so difficult using some curl and sed.


    psql -h localhost -U postgres obsjats -c "
     INSERT INTO article (pmcid,content) VALUES (
        '`curl -s -H 'Accept: application/xml' 'https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=pmc&id=PMC3970601' | sed -e "s/'/''/g" `'

Simplest perl to loop over IDs

Suppose a basic PostgreSQL database,

CREATE TABLE article (
  content xml  -- in general XML JATS

This is myXML-insert.pl:

#!/usr/bin/perl -w

@lines = <STDIN>;
$p = "PGPASSWORD=postgres psql -h localhost -U postgres obsjats -c";
foreach $url (@lines) {
 $url =~ s/^\s+|\s+$//g ;
 print "\n$p \"INSERT INTO article (uri,content) VALUES ('$url',xmlparse(document '\`curl -s -H 'Accept: application/xml' '$url' | sed -e$

So, you can produce a list of URLs (one per line) that you sampled with browser, and loop over all by cat myList.txt | perl myXML-insert.pl > myInsert.sh. If all good at the generated myInsert.sh you can run sh myInsert.sh to insert all in your database.


The largest Open Data sources are the XML JATS files, the sources of scientific papers... There are big JATS repositories at SciELO or PubMed Central. The JATS files are full content and the most complete and reliable source of a legal deposit.

(aleatory) Example of research paper (see also this other answer),


It is a paper at SciELO, where the XML JATS data is at the "Article in xml format" link in the same page (this link).

See (in that XML) the <article-meta> tag , with all metadata information about the article. All the article citations are in the <ref-list> tag ... So, all imaginable bibliographic data is there!

At PubMed Central the bulk data are at http://www.ncbi.nlm.nih.gov/pmc/tools/ftp/

You need to start with a "candidate article selection", because the universe is so big... The same article is also at PubMed Central,


where is also possible to get XML... But the procedure is like a "hacker task"... Procedure to get the full XML data at PubMed Central:

  1. get the list of all articles that are in the FTP of PubMed Central

  2. filter articles by ID (in this example is PMC3706994) or a list of articles by journal name, volume, etc.

  3. download it

  4. analyse the XML

So, in a UNIX/Linux terminal

   # step 1
   wget -c ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/file_list.txt

   # step 2
   cat file_list.txt | grep PMC4612608
   # or (slower) 
   cat file_list.txt | grep -i -E genet.mol.biol.2015.+353 | more

   # step 3     
   wget -c "ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/82/0d/Genet_Mol_Biol_2015_Aug_21_Jul-Sep_38(3)_353-365.tar.gz"
   tar -vzxf Genet_Mol_Biol_2015_Aug_21_Jul-Sep_38\(3\)_353-365.tar.gz

   # step 4
   more Genet_Mol_Biol_2015_Aug_21_Jul-Sep_38\(3\)_353-365/1415-4757-gmb-38-3-353.nxml

There are a lot of tools for JATS analysis and data extraction.


To update Cristian's answer, with regard to PubMed, you don't need to be from the USA. There are a couple of ways to get the data.
1) PubMed, go to http://www.ncbi.nlm.nih.gov/pubmed/?term=%22all%22%5Bfilter%5D (which today hosts 25M articles - not OpenAccess) choose send to -> file -> abstract or xml. I downloaded the XML which is kind of larger - 298.298.250 lines containing the 25M articles (summaries, methods, conclusions) 2) PMC (PubMed central) which can be queried like PubMed but a bit different http://www.ncbi.nlm.nih.gov/pmc/?term=%22all%22%5Bfilter%5D in that PMC needs to be queried instead of pubmed. This results in 3.7M open articles. But there's a (small) catch, in that about 2/3 of these open articles only the abstracts (and metadata) can be downloaded for XML and TXT. These 2/3 are open, but only for humans (sort of).
3) The 1/3 remaining from the 3.6M from PMC (about 1.04M) can be downloaded, also, from FTP: ftp.ncbi.nlm.nih.gov/pub/pmc . Here you can download the XMLs (NXML) for all OA (OpenAccess) files, which may be downloaded in bulk - hence containing the entire article in NXML (articles..tar.gz) or TXT (articles.txt..tar.gz)... The other files in this directory link PMC_ids to the location where the other data (images and the like) can be downloaded in this folder -> {2hex}/{2hex}/{inf}.


For papers on computational linguistics, there is an archive of 24,300 papers at http://aclweb.org/anthology/ , with an active community interested in better retrieving this data such as DFKI and DERI.


RePEc data:

Only economics, but coverage, quality, and ease of access are outstanding.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.