Skip to main content
edited tags
Link
Franck Dernoncourt
  • 7.8k
  • 10
  • 40
  • 86
Tweeted twitter.com/#!/StackOpenData/status/618138560378613760
edited tags
Link
philshem
  • 17.7k
  • 7
  • 69
  • 171
Source Link

Where can I find a part-of-speech corpus free for commercial use?

I'm looking to teach myself more about NLP. I started with NLTK and I see that it has potential to eventually become something I could get paid for.

From there, my journey continued to reading this blog post by Matthew Honnibal. So I'm interested in writing my own NLP algorithms.

The main issue I have is with licenses, specifically with training data. I don't want to ship software to clients that have any licensing issues.

I've narrowed my search down to three: the Brown corpus (license), the CoNLL2000 corpus and the portion of the Penn Treebank that NLTK provides on their downloads page. I understand that each of these downloads has a small snippet about licenses, but it doesn't describe fully what commercial use cases it supports. They only briefly describe that it's free for academic or learning purposes.

Perhaps it's a non-issue?

If I extract statistics from these words, is that a derivative work also under the same copyright? Perhaps I can train my new tagger on the results of an older tagger?

I know that the CoNLL2000 corpus is derived from the Brill tagger. Since the Brill tagger is in the public domain, does that necessarily mean that all of the data it produces is also public domain?