4

I need to perform stemming in my application, but mostly for a small subset of stems. I.e. to check if a given word is of a certain stem out of a small list of stem words. Maybe a hundred stem words only. There's few stemming projects out there, but they all look a bit huge. I am looking for one that is lightweight and predictable, that is preferrably based on a definite list of words belonging to each stem, not on a heursitc algorithm.

That's because I assume large stemming packages rely more on algorithm (total newbie assumption) rather than holding databases of words, and thus their results may be a bit heuristic and less accurate.

Does anyone know of such a software package, that stems by a 'dictionary' of words, or any open database that can be derived from, for the same purpose?

a package can be better than a database, if it's speedy for large volume processing, as I need to figure, in my application, whether each and every word of a given long text, has the same stem as any of a short list of stem words, which requires an efficient algorithm.

  • If someone's answer solved your problem, you might want to accept it as the answer using the big checkbox. – ojdo Oct 15 '13 at 9:36
  • 1
    I think all of them are great, and therefore I'm reluctant to say any of them is the answer. At least until I finished implementing the solution to my case, which may add some more insight. – Matan Oct 17 '13 at 12:28
2

If you are really looking for this dictionary approach, you can just roll your own.

In python this would be something like,

stem_dict = ...
stemmed_doc = []
for word in original_doc :
    stemmed_doc.append(stem_dict[word])
2

If you have to treat long, natural English, I would go for the Porter stemming algorithm (with implementations) in order to stem both the text and your dictionary and then simply do (Python/pseudo-code):

for word in stemmed_document:
     if word in stemmed_dictionary:
         pass # do something with a matching word stem
2

Edit: Here are some useful Ruby packages I've used:

https://github.com/aurelian/ruby-stemmer

https://github.com/romanbsd/fast-stemmer

There's also a full suite package named Treat which contains stemming among other things: https://github.com/louismullie/treat

Truth be told, Python has the better packages for this, but just in case you're trying to do it in Ruby/Rails...

(not relevant links below)

Checkout the data set of words (not sure if you need plain English text or if a list of words is sufficient) that Peter Norvig uses for his writeup on natural language processing:

http://norvig.com/ngrams/

Quoting from his summary, the data contains:

  • 4.9 MB count_1w.txt The 1/3 million most frequent words, all lowercase, with counts. (Called vocab_common in the chapter, but I changed file names here.) 5.6 MB count_2w.txt The 1/4 million most frequent two-word (lowercase) bigrams, with counts.
  • 0.0 MB count_2l.txt Counts for all 2-letter (lowercase) bigrams.
  • 0.2 MB count_3l.txt Counts for all 3-letter (lowercase) trigrams.
  • 0.0 MB count_1edit.txt Counts for all single-edit spelling correction edits, from the file spell-errors.txt.
  • 0.5 MB spell-errors.txt A collection of "right: wrong1, wrong2" spelling mistakes, collected from Wikipedia and Roger Mitton.

The following files are not referenced in the chapter, but may be useful to you.

  • 0.3 MB count_big.txt A word count file (29,136 words) for the big.txt file from my spell correction article.
  • 4.3 MB shakespeare.txt The complete works of Shakespeare, tokenized so that there is a space between words and punctuation. From John DeNero.
  • 3.0 MB sowpods.txt The SOWPODS word list (267,750 words) -- used by Scrabble players (except in North America) and in other word games.
  • 1.9 MB TWL06.txt The Tournament Word List (178,690 words) -- used by North American Scrabble players.
  • 1.9 MB enable1.txt The ENABLE word list (172,819 words) -- also used by word game players. Words with Friends uses a variant of this.
  • 2.7 MB word.list The YAWL (Yet Another Word List) word list (263,533 words) formed by combining the above.
  • Thanks this is helpful in general, But I'm not sure how it's directly related to stemming. Care to elaborate? – Matan Oct 13 '13 at 19:52
  • 1
    I'm sorry, I misread your question or maybe I was reading the others' answers and thought you were trying to create your own stemmer. Let me add some software links to the answer I made. – Dan Oct 14 '13 at 2:10
2

Given that language is not a fixed thing, I'd hesitate to put much stock in a fixed database of "definite" stems.

Here's the source code for the NLTK (python) Porter stemmer (GPL). It looks like it has no serious dependencies on anything else in NLTK -- just an interface that you could discard and some stuff for unicode compatibility that you could adapt. So if you're using Python (and can work within the GPL), this could get you started pretty quickly without the weight of the entire NLTK library.

If you still feel that the stemmer is likely to make mistakes, it wouldn't be hard to add an override that checks a dictionary of 'definite' stems before dropping back to the algorithm. Maybe you deploy it with an empty dictionary and then add corrections as you find errors.

PS this Stack Overflow post "Stemming algorithm that produces real words" looked like it might relate to your question.

2

As I can understand, you want something custom. I don't know any database or any library where you can have access on the dictionary of the words, but you can see here a greek version where it works like you describe on your question. If you know python, you can change it a bit and make it to fit on your problem

Link

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.