OPUS provides great moses/tmx files of parallel texts. However, many xml files that are accompanied by xces linking files have mistakes in tokens order.

 <chunk type="NP" id="c0-15">
 <w hun="DT" tree="DT" lem="the" pos="DT" id="w10.29">the</w>
 <w hun="JJ" tree="JJ" lem="early" pos="JJ" id="w10.30">early</w>
 <w hun="CD" tree="CD" lem="@card@" pos="CD" id="w10.31">1980</w>
 <w hun="VBZ" tree="NN" lem="S" pos="NNS" id="w10.33">s</w>
 <w hun="NN" tree="POS" lem="'" id="w10.32">’</w>
 <w hun="." tree="SENT" lem="." pos="." id="w10.34">.</w>

The mistakes are with abbreviations like "it's" and "wouldn't". The apostrophe symbol is usually in the wrong place.

 <chunk type="NP" id="c7-4">
 <w hun="NN" tree="NN" lem="today" pos="NN" id="w17.5">today</w>
 <w hun="VBZ" tree="NN" lem="S" pos="VBZ" id="w17.7">s</w>
 <w hun="NN" tree="NN" lem="price" pos="NN" id="w17.8">price</w>
 <w hun="NN" tree="POS" lem="'" id="w17.6">’</w>
 <chunk type="VP" id="c7-5">
 <w hun="VBZ" tree="VBZ" lem="be" pos="VBZ" id="w17.9">is</w>

However, the other parallel files such as moses and TMX are alright. How to generate correct parallel sentences out of xces files?

Note: I used opus2text tool and it did not fix the token order problem.

Note: The use case is for example generating parallel paragraphs (not sentences) or generating multiparallel corpus via opus2multi tool then multi parallel text.

Note: The examples are taken from the "news-commentary v16" corpus I checked the corpus named "Books" found similar issues.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.