I want to test machine learning tasks on time-divided textual data set. For this purpose, I want to use a common text data set which is already validated and "good" for use. I already found a Web of Science data set from this source:

K. Kowsari, D. E. Brown, M. Heidarysafa, K. Jafari Meimandi, M. S. Gerber and L. E. Barnes, "HDLTex: Hierarchical Deep Learning for Text Classification", 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 364-371. doi: 10.1109/ICMLA.2017.0-134

Unfortunately, the data set does not include data about the time of the publications, which I really need for my algorithms. Can anyone recommend a common textual data set for me which is divided into time windows?


It depends what kind of text you're looking for. I can think of the following three sources for text data with years of publication:

  • Project Gutenberg is a repository containing 60,000 books in the public domain. It includes a lot of literature classics. I don't think the original publication date is included but for many books it's not to hard to retrieve it from Wikipedia.
  • Google NGrams is a massive collection of n-grams extracted from one million books. Each n-gram is provided with the year of the book it comes from, but there is no full text available.
  • Medline and/or PubMed Central abstracts (around 20 millions) and/or full papers (around 2 millions) from scientific biomedical articles. The year of publication is included in the metadata.
| improve this answer | |

In addition to the answer by @Erwan, I came across

Both are curated lists of text datasets for NLP tasks. A few datasets that may fit your requirement of temporal variation:

Since you're looking for datasets that have been used in previous NLP studies, I suggest searching these lists for data from Kaggle competitions or academic papers.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.