I am working on textual analysis of reports in English language. The documents are official records. I requires the sentence level segmentation of text. One way is through regular expression. Other way is through classifier trained on tagged data-set and then predict whether a period is sentence boundary or not? But for that i will require tagged data-set. Is there any sentence boundary tagged data-set freely available?
2 Answers
Some Corpora in the NLTK have a method sents() that returns each sentence as a list of words, for example the Brown corpus:
import nltk
nltk.download() # this opens a GUI to download all corpora needed
from nltk.corpus import brown
sentences = brown.sents()
The British National Corpus, "a 100 million word collection of samples of written and spoken language from a wide range of sources", is free of charge and segmented, but i never used it.
