EDIT 2017
PubMed Central changed all adressess and some methods.
Only tips:
To bulk access you can use complex FTP procedures similar to the described below in the "OLD" section... Or a simple perl (or shell) script to loop over API.
The basic GET API are showed at https://www.ncbi.nlm.nih.gov/pmc/tools/get-full-text/
To GET by shell you must to use Accept: application/xml!
Example: curl -s -H 'Accept: application/xml' 'https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=pmc&id=PMC3970601
To list all IDs (PMC-ID) that you need in a complex query, see a good tutorial about complex-query APIs https://www.ncbi.nlm.nih.gov/books/NBK25499/
To insert XML into a database like PostgreSQL, not so difficult using some curl and sed.
Example:
psql -h localhost -U postgres obsjats -c "
INSERT INTO article (pmcid,content) VALUES (
'PMC3970601',
xmlparse(document
'`curl -s -H 'Accept: application/xml' 'https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=pmc&id=PMC3970601' | sed -e "s/'/''/g" `'
))
"
Simplest perl to loop over IDs
Suppose a basic PostgreSQL database,
CREATE TABLE article (
uri text NOT NULL PRIMARY KEY,
content xml -- in general XML JATS
);
This is myXML-insert.pl:
#!/usr/bin/perl -w
@lines = <STDIN>;
$p = "PGPASSWORD=postgres psql -h localhost -U postgres obsjats -c";
foreach $url (@lines) {
$url =~ s/^\s+|\s+$//g ;
print "\n$p \"INSERT INTO article (uri,content) VALUES ('$url',xmlparse(document '\`curl -s -H 'Accept: application/xml' '$url' | sed -e$
}
So, you can produce a list of URLs (one per line) that you sampled with browser, and loop over all by cat myList.txt | perl myXML-insert.pl > myInsert.sh. If all good at the generated myInsert.sh you can run sh myInsert.sh to insert all in your database.
OLD
The largest Open Data sources are the XML JATS files, the sources of scientific papers... There are big JATS repositories at SciELO or PubMed Central. The JATS files are full content and the most complete and reliable source of a legal deposit.
(aleatory) Example of research paper (see also this other answer),
http://dx.doi.org/10.1590/S1415-475738320140376
It is a paper at SciELO, where the XML JATS data is at the "Article in xml format" link in the same page (this link).
See (in that XML) the <article-meta> tag , with all metadata information about the article. All the article citations are in the <ref-list> tag ... So, all imaginable bibliographic data is there!
At PubMed Central the bulk data are at http://www.ncbi.nlm.nih.gov/pmc/tools/ftp/
You need to start with a "candidate article selection", because the universe is so big... The same article is also at PubMed Central,
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4612608/
where is also possible to get XML... But the procedure is like a "hacker task"... Procedure to get the full XML data at PubMed Central:
get the list of all articles that are in the FTP of PubMed Central
filter articles by ID (in this example is PMC3706994) or a list of articles by journal name, volume, etc.
download it
analyse the XML
So, in a UNIX/Linux terminal
# step 1
wget -c ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/file_list.txt
# step 2
cat file_list.txt | grep PMC4612608
# or (slower)
cat file_list.txt | grep -i -E genet.mol.biol.2015.+353 | more
# step 3
wget -c "ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/82/0d/Genet_Mol_Biol_2015_Aug_21_Jul-Sep_38(3)_353-365.tar.gz"
tar -vzxf Genet_Mol_Biol_2015_Aug_21_Jul-Sep_38\(3\)_353-365.tar.gz
# step 4
more Genet_Mol_Biol_2015_Aug_21_Jul-Sep_38\(3\)_353-365/1415-4757-gmb-38-3-353.nxml
There are a lot of tools for JATS analysis and data extraction.
