Does anyone know where I can find the dataset of all possible research publications ? I would like to get only titles and abstracts for those papers.
-
one research field or every research field?– philshemCommented Mar 10, 2014 at 8:54
-
2Do you really mean "all possible"? Even "all" would be difficult to answer without some limitations (e.g., in peer-reviewed journals).– igelkottCommented Mar 10, 2014 at 10:36
-
For now I want to deal with Computer Science research papers only. I know "all possible" is a bit too much, but as many as I can get will be great.– pbathalaCommented Mar 10, 2014 at 16:08
-
1Try Crossref. Here's their web services labs.crossref.org– sckottCommented Mar 11, 2014 at 6:30
-
After the Aaron Swartz incident, JSTOR started allowing people access to their corpus for text mining -- I don't know if they'd necessarily allow this sort of access, though.– JoeCommented Jun 19, 2014 at 16:39
Add a comment
|
5 Answers
Restricting this question to "Computer Science" (as suggested in the comments) and reducing "all possible" to the ones which were actually published, one resource is The DBLP Computer Science Bibliography. Note that this source follows ODC-BY 1.0.
The DBLP has a search interface with an attempt to identify coauthor communities. They also offer bulk downloads in XML format.
Some other research subjects have good sources, such as PubMed for biology, medicine, etc.
First to be able to better answer: what is your definition of open scientific publications?
Second: there is for sure nothing comparable to Wikipedia or so in terms of cross-disciplinarity and amount of content. Most of this data is in the hands of big (publishing) companies. There is no central register for scientific literature. One of the best ways is to get bibliographic data and find the DOI (digital object identifier). With this, you can search in different databases for it.
But I know some big open repositories, which have a lot of open access:
Another good spot to look for Open Access publications is the Directory of Open Access Journals (DOAJ).
BASE (Bielefeld academic search engine) is one of the world's most voluminous search engines especially for academic open access web resources. It claims 80 million documents, and offers an API interface (with registration).
"all possible research publications" is way too broad of a search.
It's estimated that there are over 26,000 peer reviewed journals, and 21,000 'non-journals', not to mention all of the journals that aren't peer reviewed.
It's possble that some of those only published one issue, and went defunct ... but even then the number of articles is huge
Searching Google Scholar for the letter 'a' yields 8,460,000 results. The Astrophysics Data System is at least 9,250,000 records based on a poster from this week's LISA conference.
To start, I'd probably try to use the various disciplinary bibliographic systems -- ADS for astronomy & physics, and PubMed Central for biology & health.
For books, you might try OCLC, but I think their 'open license' applies to their members, not necessarily the general public. ... that also wouldn't necessarily have 'abstract' info, except for a few phrases or sentence you might in the front matter of more recent books books.
Answering because I came to this from Google, even though it was asked ages ago. Check out constellate.org. Recent release by JSTORLabs, they integrate multiple databases, and allow access to filtered sets of up to 25k papers at a time.
