Common text data sets in form of panel data

Question

I want to test machine learning tasks on time-divided textual data set. For this purpose, I want to use a common text data set which is already validated and "good" for use. I already found a Web of Science data set from this source:

K. Kowsari, D. E. Brown, M. Heidarysafa, K. Jafari Meimandi, M. S. Gerber and L. E. Barnes, "HDLTex: Hierarchical Deep Learning for Text Classification", 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 364-371. doi: 10.1109/ICMLA.2017.0-134

Unfortunately, the data set does not include data about the time of the publications, which I really need for my algorithms. Can anyone recommend a common textual data set for me which is divided into time windows?

Answer 1

In addition to the answer by @Erwan, I came across

• nlp-datasets, collected by Nicolas Iderhoff;
• Awesome Public Datasets, collected by AwesomeData.

Both are curated lists of text datasets for NLP tasks. A few datasets that may fit your requirement of temporal variation:

• Cheng-Caverlee-Lee September 2009 - January 2010 Twitter Scrape: Timestamped tweets (3,844,612 training, 5,156,047 testing) for NLP geolocation task. See paper here.
• NIPS2015 Papers (version 2) [Kaggle]: full text of all NIPS2015 papers
• Reddit comments: I have every publicly available Reddit comment for research. ~ 1.7 billion comments @ 250 GB compressed. Any interest in this?, Reddit, July 2015;
• YouTube videos: 1.7 millions Youtube videos' description on Reddit, August 2015.

Since you're looking for datasets that have been used in previous NLP studies, I suggest searching these lists for data from Kaggle competitions or academic papers.

Answer 2

It depends what kind of text you're looking for. I can think of the following three sources for text data with years of publication:

• Project Gutenberg is a repository containing 60,000 books in the public domain. It includes a lot of literature classics. I don't think the original publication date is included but for many books it's not to hard to retrieve it from Wikipedia.
• Google NGrams is a massive collection of n-grams extracted from one million books. Each n-gram is provided with the year of the book it comes from, but there is no full text available.
• Medline and/or PubMed Central abstracts (around 20 millions) and/or full papers (around 2 millions) from scientific biomedical articles. The year of publication is included in the metadata.