I am working on a NLP project, and for that I need a dataset of English words (words typically found in dictionary). Could somebody please guide me how to put together such dataset?
Is there any online resource for such requirements? If not, what is the best way to go about making such dataset on our own?
You can also use ConceptNet5, it a list of concepts (~ words) linked together using relations. It takes its data from DBPedia, wordnet and wiktionary. It's free and available in several languages.
Definitions are not included, but surface text (see the image) can give a hint of the meaning of words and relations. Actually the point of conceptnet is to store the meaning of the words in the structure of the graph. The web interface allows to surf and query the data with a browser.
More complex REST queries are possible, have a look at the REST API.
If you hit the query limit or some other reasons, you can build the database for yourself using the official build script or using the docker image (recommended). You can also directly use the database dumps (that's what i do).
There is also a mailling list.
I also recommend you read the paper ConceptNet, a practical commonsensereasoning tool-kit even if it's for a previous version of ConceptNet the idea is the same.
Free wordlists are available from SCOWL. Those might suit your purpose depending what data you need.
There are probably many good existing datasets, but if you want to make your own, here is a little Python 2.7 code that takes a text file as input, and then prints the most common N words.
Let's use a sample book from Project Gutenberg - Complete Works of William Shakespeare (5.3 MB .txt file, pg100.txt)
# -*- coding: utf-8 -*-
import string
from collections import Counter
def get_words(infile,N):
# open file and read it into a string
with open(infile,'rb') as textfile:
text = textfile.read()
# remove punctuation - http://stackoverflow.com/q/265960/2327328
text = text.translate(string.maketrans("",""), string.punctuation)
# make all text lower case
text = text.lower()
# split text into array of words
words = text.split()
return Counter(words).most_common(N)
# run the function with these input params (file_name, N most common words)
print get_words('pg100.txt',10)
gives as output:
[('the', 27824), ('and', 26791), ('i', 20681), ('to', 19261), ('of', 18289), ('a', 14667), ('you', 13716), ('my', 12481), ('that', 11135), ('in', 11027)]
Of course, to print all words, then just make N really large (10**10, aka 10^10). The timing is based on reading the entire file, so it shouldn't change much by giving much more output (takes 0.7 seconds for N=10 and 1.1 seconds for N=10**10 on my old laptop).
See here for instructions about downloading the entire project. You can then iterate over each text file.
