7

Most of the parallel corpora (Opus, EuroParl, OpenSubtitles) have only human translations.

src_txt | human_trans_of_src_txt_into_trg_lng (We assume all the translations are good.)

Is there a corpus of machine translations, annotated with human eval scores?

src_txt | machine_trans_of_src_txt_into_trg_lng | score * Any score is fine (label vs number, sentence-level vs word-level)

OR

src_txt | machine_trans_of_src_txt_into_trg_lng*

*Where all translations are known to be good (or all known to be bad)

Improve this question
2

1 Answer 1

Reset to default
1

Such a dataset was used to create the BLEU metric, but I do not know if it is available. Maybe you could try asking the authors of the paper that introduced it if they could share their data?

Papineni, K.; Roukos, S.; Ward, T.; Zhu, W. J. (2002). BLEU: a method for automatic evaluation of machine translation. ACL-2002: 40th Annual meeting of the Association for Computational Linguistics. pp. 311–318. http://aclweb.org/anthology/P/P02/P02-1040.pdf

The paper is quite old so I suspect the dataset might be a bit dusty though. The links from the comments above seem to be more current.

Improve this answer
1
  • Confusingly, quality evaluation and quality estimation are subtly different - corpus/system vs sentence level. The former assumes a human translation ie a parallel corpus, but the metric used is traditionally very rough, because it just needs to be correct on average over the corpus. The latter may have no human translation, but the score is given by humans. I am working on leveraging parallel corpora for sentence-level quality estimation, but in combination with other approaches to generating data.
    – Adam Bittlingmayer
    Oct 10, 2017 at 15:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.