3

I'm looking for a source of character-level ngrams and counts that include spaces. I've found character-level ngram datasets, but they don't include spaces. Ideally, the dataset will include up to 8-grams and be based on a broad swath of English language text.

For example, "the quic" should be one of the 8-grams in the dataset.

  • 2
    Any reason you want ngrams and not original text? It is probably easier to get a well-understood "broad swath" of text than the same text pre-processed to a model. Generating the ngrams is also pretty easy once you have the source text. – Neil Slater Apr 21 '15 at 19:35
2

Based on @NeilSlater's comment, you can easily calculate character N-grams with a few lines of code.

In this snippet, I use Python's Collections library, which is quite fast for these types of applications:

from collections import defaultdict

def make_char_ngram(text,N):
    data = defaultdict(int) # for speed

    for i in xrange(len(text)-N+1):
        x = text[i:i+N] # actual N-gram
        data[x] += 1    # add 1 to this N-gram key
    return data

print make_char_ngram('ABC the quick brown fox the quick brown fox the quick brown fox XYZ',8)

gives you character 8-grams as a dictionary, where the key is the character N-gram, and the value is the count of that case:

defaultdict(<type 'int'>, {'brown fo': 3, 'ck brown': 3, 'n fox th': 2, 'own fox ': 3, ' the qui': 3, ' quick b': 3, 'rown fox': 3, 'uick bro': 3, 'k brown ': 3, ' fox the': 2, 'e quick ': 3, 'fox the ': 2, ' fox XYZ': 1, 'ox the q': 2, 'wn fox X': 1, ' brown f': 3, 'ick brow': 3, 'BC the q': 1, 'the quic': 3, 'quick br': 3, 'ABC the ': 1, 'wn fox t': 2, 'C the qu': 1, 'n fox XY': 1, 'he quick': 3, 'x the qu': 2})

(similar question)


In terms of performance, this code went through a 6.2MB text file with 128k lines in 4.5 seconds on my laptop (but not writing the results).


If you want to parse massive amounts of text, consider writing these dictionaries to a NoSQL database like MongoDB. In this case, you can parse text in pieces and don't need to run the thing from the start every time. Writing a Python dictionary to MongoDB should be easy (perhaps transform the dict to JSON and then the import is direct).

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.