I am looking for a very large collection ( > 10 GB) of biomedical text documents to run some unsupervised experiments with (for recognizing drug names, etc). Do we have access to such a thing for free? Or is it legal to crawl pubmed and use the crawled dataset for research experiments (no idea about how the licensing/copyright is at pubmed)?
2 Answers
There is an open access subset in PubMed Central which can be crawled via an API and might be what you are looking for: http://www.ncbi.nlm.nih.gov/pmc/tools/openftlist/. There are currently around 1 million articles in the open access subset, which I imagine would be getting towards your GB target. It looks like versions of the subset are available on torrent sharing sites.
"is it legal to crawl pubmed and use the crawled dataset for research experiments?"
The open access subset includes articles with a range of licenses, some of which restrict "commercial use" and creation of "derivatives". If you are worried about legality, you could focus your search on specific licenses (e.g. there are over 500K CC-BY licensed articles).
Europe PMC would be worth considering as an alternative source of content: https://europepmc.org/Help#doesEPMChaveallPMandPMC. It covers the majority of PubMed Central content, plus additional content like NHS Clinical Guidelines.
There is an Open Access Journal site called PLOS. The amount of material maybe a little (but not much) short for your requirements, but it is Open Access.
See https://www.plos.org/ for more information.