14

I am looking for paraphrase data sets.

I am aware of the following:

  1. PPDB: The Paraphrase Database (Ganitkevitch, Juri, Benjamin Van Durme, and Chris Callison-Burch. "PPDB: The Paraphrase Database." HLT-NAACL. 2013.). Its English portion, PPDB:Eng, contains over 220 million paraphrase pairs.

  2. The Microsoft Research Paraphrase Corpus (2005):

5800 pairs of sentences which have been extracted from news sources on the web, along with human annotations indicating whether each pair captures a paraphrase/semantic equivalence relationship. No more than 1 sentence has been extracted from any given news article. We have made a concerted effort to correctly associate with each sentence information about its provenance and any associated information about its author. If any attribution information is incorrect or missing, please send email to [email protected] and we will update the file.

  1. The DIRT Paraphrase Collection (Dekang Lin and Patrick Pantel. 2001. Discovery of Inference Rules for Question Answering. Natural Language Engineering 7(4):343-360.)
Improve this question
2
  • Hi, do you know how i get obtained the DIRT Paraphrase Collection? Can't seem to find it on the net :(
    – snowflake
    Commented Sep 7, 2019 at 5:53
  • You can also use ParaBank, which contains more than 50 million paraphrastic sentence pairs (in English).
    – Mohammad Dehghan
    Commented Nov 26, 2021 at 19:58

4 Answers 4

Reset to default
9

You definitely have the big ones with MSRP and PPDB. A few others, grouped by maintainers.

CLiC at the Universitat de Barcelona

Maintain several highly annotated corpora. These are available from their site, by request/registration. It is a request they grant with almost no questions asked. (My experience was that after registering online I had to sent them an email).

P4P (paraphrases for plagiarism detection)

The P4P corpus contains a partition of the plagiarism cases in the PAN-PC-10* corpus manually annotated with the paraphrase phenomena they contain. It is composed of 847 source-plagiarism pairs in English.

This is not the hyperlink from the quote, but unlike that one does go to right address.

These are multi-sentence paraphrases. PAN-PC-10 is a Plagiarism corpus, for context

See Paper: A. Barrón-Cedeño, M. Vila, M. A. Martí, and P. Rosso. 2013. Plagiarism meets paraphrasing: Insights for the next generation in automatic plagiarism detection. Computational Linguistics, 39(4):917-947

MSRP-A (annoated MSRP)

MSRP-A stands for “Microsoft Research Paraphrase” corpus “Annotated”. The MSRP-A corpus contains the positive examples in the MSRP corpus manually annotated with the paraphrase phenomena they contain. It is composed of the 3,900 paraphrase pairs in English.

I believe the annotations are similar to those in P4P; but have not looked so I am uncertain. There is no additionally utility here if you don't want annotations.

See Paper: W. B. Dolan and C. Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP 2005), pages 9-16.

WRPA Relational Paraphrase Acquisition from Wikipedia

WRPA stands for “Relational Paraphrase Acquisition from Wikipedia” corpus. The WRPA corpus contains relational paraphrases extracted by the WRPA system from Wikipedia.

It has several types of relational paraphrases. By which the authors mean to the are paraphrases about a particular relations. Eg paraphrases about people person and their place of birth.

See Paper: Vila, Marta, Horacio Rodríguez, and M. Antònia Martí. "WRPA: A system for relational paraphrase acquisition from Wikipedia." Procesamiento del lenguaje natural 45 (2010): 11-19.

Own work: Paraphrased Grouped Opinosis Subcorpus

For my own work I required a corpus which had a large number of paraphrases for each meaning. For this I constructed a corpus based on Opinosis. Opinosis is a highly redundant corpus of opinions -- that did not originally have the paraphrases marked. (I can attest there is not a lot of validation of the accuracy of this corpus, given it was only annotated by one person (being me).)

On same pages also links my own automatically grouped MSRP subcorpus, that is less useful, because it it countains paraphrases you already have got from MSRP.

See Paper: Lyndon White, Roberto Togneri, Wei Liu, and Mohammed Bennamoun, 2015: How Well Sentence Embeddings Capture Meaning, Proceedings of the 20th Australasian Document Computing Symposium

Improve this answer
6
Improve this answer
4

Google recently released paraphrase dataset: Paraphrase Adversaries from Word Scrambling (PAWS).

https://github.com/google-research-datasets/paws

This dataset is superior to many paraphrase datasets. As stated in their page,

Existing paraphrase identification datasets lack sentence pairs that have high lexical overlap without being paraphrases. Models trained on such data fail to distinguish pairs like flights from New York to Florida and flights from Florida to New York.

The dataset has two subsets, one based on Wikipedia and the other one based on the Quora Question Pairs (QQP) dataset.

There is an accompanying research paper for this dataset:

PAWS: Paraphrase Adversaries from Word Scrambling

Improve this answer
1
  • Hi, and welcome to the Open Data Stack Exchange. Thanks for contributing an answer! If you can spare the time, consider adding a bit more context to your answer, to help protect against link rot, and to potentially save readers here a click.
    – Joe Germuska
    Commented May 6, 2019 at 20:45
3

You can also check out the Quora Question Pairs Dataset which was used in the recent Kaggle competetition.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.