4

I am looking for short text classifications (multilabel is OK). By short text I mean ~50 words max.

The data set used in Xin Li, Dan Roth. Learning Question Classifiers. COLING. 2002. is a good example: The data can be obtained at the group's website. The training data consists of 5500 labelled questions. The test data is 500 questions taken from the TREC 10 set. In both the training and test data, there are a total of 50 different question classes.

Excerpt:

DESC:manner How did serfdom develop in and then leave Russia ?
ENTY:cremat What films featured the character Popeye Doyle ?
DESC:manner How can I find a list of celebrities ' real names ?
ENTY:animal What fowl grabs the spotlight after the Chinese Year of the Monkey ?
ABBR:exp What is the full form of .com ?
HUM:ind What contemptible scoundrel stole the cork from my lunch ?
HUM:gr What team did baseball 's St. Louis Browns become ?
Improve this question

3 Answers 3

Reset to default
3

The following repository contains 20000 questions from StackOverflow. https://github.com/jacoxu/StackOverflow . The data is provided by Kaggle. There are true labels as well. So you can evaluate the clustering algorithms easily.

Improve this answer
2

Below are three datasets for a subsset of text classification, sequential short text classification. All three datasets are for speech act prediction.

  • Switchboard Dialog Act Corpus. [Jurafsky et al.1997]
  • MRDA: ICSI Meeting Recorder Dialog Act Corpus (Janin et al., 2003; Shriberg et al., 2004)
  • Dialog State Tracking Challenge 4's data set.

In terms of size, here is an overview:

enter image description here

In case anyone is interested, we presented an overview of the state-of-the art results on these three datasets in Ji Young Lee, Franck Dernoncourt, Sequential Short-Text Classification with Recurrent and Convolutional Neural Networks. NAACL 2016.

We found a few more interesting data sets for sequential short text classification mentioned in the literature, but we could not access it.

  • [Jurafsky et al.1997] Dan Jurafsky, Elizabeth Shriberg, and Debra Biasca. 1997. Switchboard SWBDDAMSL shallow-discourse-function annotation coders manual. Institute of Cognitive Science. Technical Report, pages 97–102.
  • [Janin et al.2003] Adam Janin, Don Baron, Jane Edwards, Dan Ellis, David Gelbart, Nelson Morgan, Barbara Peskin, Thilo Pfau, Elizabeth Shriberg, Andreas Stolcke, et al. 2003. The ICSI meeting corpus. In Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP’03). 2003 IEEE International Conference on, volume 1, pages I–364. IEEE.
  • [Shriberg et al.2004] Elizabeth Shriberg, Raj Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey. 2004. The ICSI meeting recorder dialog act (MRDA) corpus. Technical report, DTIC Document.
Improve this answer
1

Zhang et al. built several large datasets for short text.

Here is the link

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.