Most of the parallel corpora (Opus, EuroParl, OpenSubtitles) have only human translations.

src_txt | human_trans_of_src_txt_into_trg_lng (We assume all the translations are good.)

Is there a corpus of machine translations, annotated with human eval scores?

src_txt | machine_trans_of_src_txt_into_trg_lng | score * Any score is fine (label vs number, sentence-level vs word-level)

OR

src_txt | machine_trans_of_src_txt_into_trg_lng*

*Where all translations are known to be good (or all known to be bad)

Such a dataset was used to create the BLEU metric, but I do not know if it is available. Maybe you could try asking the authors of the paper that introduced it if they could share their data?

Papineni, K.; Roukos, S.; Ward, T.; Zhu, W. J. (2002). BLEU: a method for automatic evaluation of machine translation. ACL-2002: 40th Annual meeting of the Association for Computational Linguistics. pp. 311–318. http://aclweb.org/anthology/P/P02/P02-1040.pdf

The paper is quite old so I suspect the dataset might be a bit dusty though. The links from the comments above seem to be more current.

  • Confusingly, quality evaluation and quality estimation are subtly different - corpus/system vs sentence level. The former assumes a human translation ie a parallel corpus, but the metric used is traditionally very rough, because it just needs to be correct on average over the corpus. The latter may have no human translation, but the score is given by humans. I am working on leveraging parallel corpora for sentence-level quality estimation, but in combination with other approaches to generating data. – A. M. Bittlingmayer Oct 10 '17 at 15:40

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.