I'm looking for a dataset of human translated sentences.

The ideal dataset would look like this:

1, en, The weather is nice today.
1, de, Das Wetter ist heute schön.
1, es, El clima es agradable hoy.
1, el, Ο καιρός είναι καλός σήμερα. 
...

for as many languages as possible and for sentences of varying length and complexity. It's important that the text is human translated.

An existing dataset or database would be great, but I can also scrape websites or hook up to an API.

(For programming, RosettaCode does a nice job of this.)

Note: I've suggested Anki for these types of questions (one, two), but that doesn't work well in this case because it's 2-languages only.

  • 1
    There are so many ways to say the same concept, even if you did get such a dataset, I'm not sure you would get EVERY way of saying that same thing. Example: - It's cold out. - It's cold outside. - It's cold today. - It's cold right now. - It's cold now. - I'm cold. - I'm not very warm. - I'm not happy with the temperature. – Bulrush Oct 25 '14 at 14:47
  • I understand. Thank you for not being snarky. But I do hope this helps the OP as I was involved in machine translation for a project. We discovered there is extensive manual checking. Since there are so many ways to say something in English, we had a spreadsheet with 10,000+ lines, and 2 columns: English and French. We had a translator person translate each English phrase, then our computer replaced the English with French. Many were repetitions of the same concept, but the customer did not want to standardize to one phrase for that concept. – Bulrush Oct 29 '14 at 11:45
  • Thanks for your feedback. I am only looking for human translations. – philshem Oct 29 '14 at 18:40
up vote 5 down vote accepted

Tatoeba.org is exactly that: http://tatoeba.org

Their data is human-edited and used by dozens of products/websites including electronic dictionaries, so it is of reasonable quality.

It has 471468 English sentences and 179 languages.
(not all sentences are translated to all languages though, far from that)

The structure is not a simple 1-1-1, a sentence can have several translations.

For your example, see http://tatoeba.org/eng/sentences/show/406004 :

Tatoeba.org

The whole data is open (Creative Commons CC-BY) and downloadable at http://tatoeba.org/eng/downloads in various formats.

Disclaimer: I am a member of Tatoeba (I mentored a GSoC that created a webapp to enrich your Anki decks using Tatoeba)

One option would be to use movie subtitles. As described here, OpenSubtitles.org can be used to find parallel text in many languages.

I've tried this before in the past and I find that it's a great way to collect data, but it's very hard to synchronize the different text files (even with the movie timestamp)

enter image description here

Have you considered using some book that's been widely translated, but is out of copyright?

The bible comes to mind, as the verse numbering scheme would make it easy to match up sentances. Some translations may still be copyrighted (eg, the NIV), but older translations like the King James would be out of copyright.

  • 1
    Great idea! This list is helpful - en.wikipedia.org/wiki/… which brought me to here ohchr.org/EN/UDHR/Pages/SearchByLang.aspx – philshem Oct 25 '14 at 19:41
  • Anderson's Fariy Tales and Pinnocio ... interesting, and both out of copyright (in the US) ... the only question is if the translations are. – Joe Oct 26 '14 at 13:04
  • 1
    That's why the UN Human Rights works well. It's a direct translation (article by article), and should be a forgiving license. – philshem Oct 29 '14 at 7:58
  • 1
    If you use the Bible, you would still have to do some sentence tokenizing because verses can contain more than one sentence or multiple verses can be one sentence. – Suzana Jun 27 '15 at 19:18

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.