This is a bit specific. I want to train a support vector machine or naive Bayes classifier to spot advertisements in HTML so I can clean them before LDA analysis. Only a few percentage points of bad data exist, (spam, web menus, headlines; etc. after stripping the text and cleaning with rules). These lines are throwing off my categorization attempts as the good data is lumped together and the bad data is separated by categories, likely because the resulting distance is quite far from the important information. Increasing the number of 'topics' does not appear to help.

Is it possible to use a set such as the Enron spam set to classify advertisements? After finding the root words and removing stopwords, they look fairly similar. Is there a better corpus or approach?

1 Answer 1


After long hours of searching, I could not find one. The short answer was to crawl many different websites, separate the data, and use multi-nomial bayes or a radial basis function depending on how awful the results were to generate my own corpus via cluster/grouping with the manually classified data forming a kernel.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.