I'm looking for a source of character-level ngrams and counts that include spaces. I've found character-level ngram datasets, but they don't include spaces. Ideally, the dataset will include up to 8-grams and be based on a broad swath of English language text.
For example, "the quic" should be one of the 8-grams in the dataset.
Based on @NeilSlater's comment, you can easily calculate character N-grams with a few lines of code.
In this snippet, I use Python's Collections library, which is quite fast for these types of applications:
from collections import defaultdict
def make_char_ngram(text,N):
data = defaultdict(int) # for speed
for i in xrange(len(text)-N+1):
x = text[i:i+N] # actual N-gram
data[x] += 1 # add 1 to this N-gram key
return data
print make_char_ngram('ABC the quick brown fox the quick brown fox the quick brown fox XYZ',8)
gives you character 8-grams as a dictionary, where the key is the character N-gram, and the value is the count of that case:
defaultdict(<type 'int'>, {'brown fo': 3, 'ck brown': 3, 'n fox th': 2, 'own fox ': 3, ' the qui': 3, ' quick b': 3, 'rown fox': 3, 'uick bro': 3, 'k brown ': 3, ' fox the': 2, 'e quick ': 3, 'fox the ': 2, ' fox XYZ': 1, 'ox the q': 2, 'wn fox X': 1, ' brown f': 3, 'ick brow': 3, 'BC the q': 1, 'the quic': 3, 'quick br': 3, 'ABC the ': 1, 'wn fox t': 2, 'C the qu': 1, 'n fox XY': 1, 'he quick': 3, 'x the qu': 2})
In terms of performance, this code went through a 6.2MB text file with 128k lines in 4.5 seconds on my laptop (but not writing the results).
If you want to parse massive amounts of text, consider writing these dictionaries to a NoSQL database like MongoDB. In this case, you can parse text in pieces and don't need to run the thing from the start every time. Writing a Python dictionary to MongoDB should be easy (perhaps transform the dict to JSON and then the import is direct).
