2

I'm looking for big data set which is suitable to be used for document classification task. The data set which I'm looking for should composed of the frequency of the words which exist in each document, so the main goal of the task is classifying each document to one category such as Sport, Political etc. All what I found in Internet is some small data sets, if anyone can help me to find a big data set that can be used for a such task, I will be very thankful.

Note: all the attributes values should composed of integer number which represent the frequency of word in each document.

edit: I'm looking for CSV file to use it as input for multinomial naive bayes classifier.

Improve this question
4
  • Are you looking for documents to classify, or documents that have already been classified? Typically, you take whatever you have existing that's been manually classified, and use a portion of that as your training set. (holding some back to use to verify that the classifications are good)
    – Joe
    Jan 27, 2016 at 1:45
  • Actually, I am looking for a data set that already have been classified, but as I said I need it to be composed of attributes that represent the word occurrences within texts in order to use it to build a Multinomial Naive Classifier. @Joe (sorry but I don't have permission to write comments)
    – Eyad
    Jan 27, 2016 at 1:53
  • Please specify more clearly what you want to do and what language the texts should be. Note that text is by default copyrighted, if you want free texts for Open Data your choice is restricted. Is wikipedia (using the categories as tags) an option for you?
    – user6083
    Jan 27, 2016 at 7:01
  • I think you are looking for Document-Term-Matrices (or TermDocument-Matrices - same thing). You can create these yourself from document collections. Actually, that is the easy part.
    – knb
    Jan 27, 2016 at 8:59

1 Answer 1

Reset to default
0

You should be able to use this data set, just use the folder name "politics", "sports" as the desired output and ignore the summaries angle.

https://www.kaggle.com/pariza/bbc-news-summary

This associated notebook creates csv files from those txt files. Copy the code starting aroud "Part 2 and 3 : Match Data and convert to csv file":

https://www.kaggle.com/anirudhp4/dataframe

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.