I'm developing a final degree project and the first step would be to get a big amount of (public) academic papers with citations included. The purpose of the system is to calculate bibliometric indexes based on citations (that's why citations must be included in the data set). The thing is I'm struggling to find a way to get this dataset. I would really appreciate some help or advice on where to get it. It doesn't matter the subject of the papers. Thanks.
-
1Google Scholar? Inspire HEP?– astronatAug 7, 2018 at 7:10
-
adsabs.harvard.edu; arxiv.org– corey979Aug 7, 2018 at 7:22
-
1Web of Science maybe?– Richard EricksonAug 7, 2018 at 15:57
-
Any requirements for the quality of the data? I'm asking specifically because of all these fake (not peer reviewed) 'scientific magazines' that exist nowadays.– user4293Aug 8, 2018 at 8:07
Add a comment
|
6 Answers
There are a couple of options. I work for Europe PMC - a database of biomedical literature. For biomedical articles you can retrieve article info, including references/citation from Europe PMC via an API: https://europepmc.org/RestfulWebService. You can use the search module of the API (https://europepmc.org/RestfulWebService#!/Europe32PMC32Articles32RESTful32API/search) with the following query to retrieve all articles that have been cited at least once: (CITED:* NOT CITED:0). Then get the article iD (PMID) and plug it into the citations module (https://europepmc.org/RestfulWebService#!/Europe32PMC32Articles32RESTful32API/citations) to retrieve information about the citing articles. That gives you about 18 million papers to work with. If you need the full text, check the open access subset - it can be downloaded for further analysis (https://europepmc.org/downloads/openaccess) but it does not include citation info - this needs to be retrieved via the API. For articles from other disciplines you might want to check http://opencitations.net/index/coci for a full citation network (does not include article information), or use Crossref API for citation info: https://search.crossref.org/help/api
Web of Science has citation information, as does Scopus, your university may have access to both of them. WoS only allows 500 reference downloads at a time through the web interface, but depending on how much data you need to download that may be feasible. Scopus may have similar limits. Google scholar doesn't have an API but scholar.py is a Python module including a querier and parser for Google Scholar, which can at least access google scholar data including cited-by lists. Working with Google Scholar is notoriously annoying though - see https://www.nature.com/articles/d41586-018-04190-5
Academic Torrents is "a distributed system for sharing enormous datasets - for researchers, by researchers. The result is a scalable, secure, and fault-tolerant repository for data, with blazing fast download speeds."
I would suggest Wikidata at https://www.wikidata.org. The data is available via API, dump files and a SPARQL endpoint. The academic paper information in Wikidata is certainly not complete, but might be sufficient for your application.
Our Scholia webservice at https://tools.wmflabs.org/scholia/ displays some statistics about the information in Wikidata: We got over 86 million citations, over 17 million PubMed identifier links and over 13 million DOI links.
In Scholia, you can see various ways to get citation information out of Wikidata via the Wikidata Query Service, i.e., the SPARQL endpoint. For instance, the page https://tools.wmflabs.org/scholia/work/Q41799194 shows the in-going and out-going citations and a partial citation graph for the "Scholia, Scientometrics and Wikidata" paper (click on the link in the lower left corner of each panel to see the associated SPARQL query). The paper also describes Scholia in more depth.
The data in Wikidata is CC0. Many other citation databases are not so open as Wikidata.
answered Aug 15, 2018 at 17:38
I can suggest research gate or mendeley or Zotero API to get the dataset. Have you tried to contact them or use their API ? Otherwise ORCID, OpenScienceFramework or worldcat could help you.
If your university has a Scopus subscription, you should be able to access this through the Scopus API. There are several python wrappers (google "scopus API"). See https://dev.elsevier.com/ for details including acceptable use, and use cases. Assuming this is an academic research project, the acceptable use policy seems to allow downloading a large amount of data as long as it is limited in scope to a specific discipline - they do expressly state that no mining of the entire Scopus dataset is permitted.
