Language data are either texts or spoken utterances. By default, any text is copyrighted. We have Open Data for relatively recent time periods, where text were written intentionally with an open licence (like Wikipedia articles), or for relatively old time periods where the copyright has expired and the texts are public domain now. In between, there is a long period where no open texts are available.

My question is: What are generally accepted (or even better: legally proven) methods to create open data from copyrighted texts? The methods are probably in some way lossy and prevent the restauration of the original texts from the open data.

  • 1
    Q: do you 1) want to know where to obtain text versions of copyrighted text or 2) have text versions and want to create a text corpus? Also, consider visiting law.stackexchange.com. If you already have the text, your derivative work would probably fall under the legally protected category of "transformative use". I am not a lawyer. – Barry Carter Apr 23 '18 at 22:07
  • It's 2) I have text that is copyrighted and I want to create a distributable corpus out of it. I am aware of methods like scrambling the sentence order and arbitrarily cutting "holes" in the text (leaving chunks of words out) – jknappen Apr 24 '18 at 9:37
  • OK, to me, a corpus would be: 1) a list of words and their frequency, and 2) a list of sequential word pairs and their frequency. You may be able to use sequential three word triplets as well. What is your "vision" of the corpus? – Barry Carter Apr 24 '18 at 15:59
  • My vision of a corpus is a maximal distributable amount of plain text with linguistic annotation (part-of-speech tagging, lemma annotation, maybe syntax parse annotation). I know that some loss is unavoidable and some interesting analyses cannot be done on a crippled text; But leaving a substantial amount of sentences intact is enough for a lot of stuff. What you describe is a language model (in particular, a unigram model, a bigram model and a trigram model) in linguistic terminology. I see no particular difficulty in distributing such beasts. – jknappen Apr 24 '18 at 16:11
  • Check with law.stackexchange.com, but "leaving a substantial amount of sentences intact" is one of the tests used in copyright law: if you're doing that, you're probably violating copyright law. – Barry Carter Apr 24 '18 at 16:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.