13

I'm looking for a dataset of human translated sentences.

The ideal dataset would look like this:

1, en, The weather is nice today.
1, de, Das Wetter ist heute schön.
1, es, El clima es agradable hoy.
1, el, Ο καιρός είναι καλός σήμερα. 
...

for as many languages as possible and for sentences of varying length and complexity. It's important that the text is human translated.

An existing dataset or database would be great, but I can also scrape websites or hook up to an API.

(For programming, RosettaCode does a nice job of this.)

Note: I've suggested Anki for these types of questions (one, two), but that doesn't work well in this case because it's 2-languages only.

Improve this question
3
  • 1
    There are so many ways to say the same concept, even if you did get such a dataset, I'm not sure you would get EVERY way of saying that same thing. Example: - It's cold out. - It's cold outside. - It's cold today. - It's cold right now. - It's cold now. - I'm cold. - I'm not very warm. - I'm not happy with the temperature.
    – Bulrush
    Commented Oct 25, 2014 at 14:47
  • I understand. Thank you for not being snarky. But I do hope this helps the OP as I was involved in machine translation for a project. We discovered there is extensive manual checking. Since there are so many ways to say something in English, we had a spreadsheet with 10,000+ lines, and 2 columns: English and French. We had a translator person translate each English phrase, then our computer replaced the English with French. Many were repetitions of the same concept, but the customer did not want to standardize to one phrase for that concept.
    – Bulrush
    Commented Oct 29, 2014 at 11:45
  • Thanks for your feedback. I am only looking for human translations.
    – philshem
    Commented Oct 29, 2014 at 18:40

4 Answers 4

Reset to default
10

Tatoeba.org is exactly that: http://tatoeba.org

Their data is human-edited and used by dozens of products/websites including electronic dictionaries, so it is of reasonable quality.

It has 471468 English sentences and 179 languages.
(not all sentences are translated to all languages though, far from that)

The structure is not a simple 1-1-1, a sentence can have several translations.

For your example, see http://tatoeba.org/eng/sentences/show/406004 :

Tatoeba.org

The whole data is open (Creative Commons CC-BY) and downloadable at http://tatoeba.org/eng/downloads in various formats.

Disclaimer: I am a member of Tatoeba (I mentored a GSoC that created a webapp to enrich your Anki decks using Tatoeba)

Improve this answer
5

One option would be to use movie subtitles. As described here, OpenSubtitles.org can be used to find parallel text in many languages.

I've tried this before in the past and I find that it's a great way to collect data, but it's very hard to synchronize the different text files (even with the movie timestamp)

enter image description here

Improve this answer
1
  • From what I can understand of the Portuguese, this must be a different version of the movie 🤣
    – philshem
    Commented Oct 25, 2019 at 6:48
4

Have you considered using some book that's been widely translated, but is out of copyright?

The bible comes to mind, as the verse numbering scheme would make it easy to match up sentances. Some translations may still be copyrighted (eg, the NIV), but older translations like the King James would be out of copyright.

Improve this answer
4
  • 1
    Great idea! This list is helpful - en.wikipedia.org/wiki/… which brought me to here ohchr.org/EN/UDHR/Pages/SearchByLang.aspx
    – philshem
    Commented Oct 25, 2014 at 19:41
  • Anderson's Fariy Tales and Pinnocio ... interesting, and both out of copyright (in the US) ... the only question is if the translations are.
    – Joe
    Commented Oct 26, 2014 at 13:04
  • 1
    That's why the UN Human Rights works well. It's a direct translation (article by article), and should be a forgiving license.
    – philshem
    Commented Oct 29, 2014 at 7:58
  • 1
    If you use the Bible, you would still have to do some sentence tokenizing because verses can contain more than one sentence or multiple verses can be one sentence.
    – Suzana
    Commented Jun 27, 2015 at 19:18
1

https://opus.nlpl.eu/ has a huge list of possible resources (with varying quality).

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.