5

I've been looking for a table of the frequencies of all 128 ASCII characters, not just letters. The best table I've been able to find is this one, but that one only includes printable ASCII characters, and is still missing some very important ones (namely \n and \r). Is there anywhere I can find the frequency table for all ASCII characters?

  • 2
    You do realize that this is context dependent, right? Not only by spoken language, but someone who's a travel writer will have a dramatically different frequency than a computer programmer (and that'll vary by what computer language they write in) – Joe Feb 9 '16 at 19:51
  • 1
    In addition to @Joe's comment, it is also dependent upon (1) locality and (2) dialect. An obvious one for lacality:- US English has more "z", UK English has more "s" (e.g. specialize/specialise). With regard to dialect, Hoch Deutsch versus Bayerish have different distributions based on their dialect. – Marcus D Feb 15 '16 at 21:24
  • I'm comparing ASCII coding with Huffman coding, and your link is actually useful to me! Do you still need to do a frequency Analysis? I could do it over some text or file for you, or share a code. – Leopoldo Sanczyk Oct 21 at 4:57
3

You can pretty easily create character (or word) frequency data based on a few lines of code. Here is python 2.7 code that collects counts of each character

# -*- coding: utf-8 -*-

import string      # definitions of ascii printable chars
from collections import defaultdict     # fast counting

d = defaultdict(int)    # define dictionary for counting frequencies

# define text - see below for how to download from a url
text = '''Lorem ipsum dolor sit amet,\tconsectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.'''

for ch in text:       # loop over each character 
    if ch in string.printable:     # is the character in the ascii/printable set?
        d[ch] += 1    #   if so, add 1 to that characters frequency counter

print d     # print all frequencies

The output is then (formatted):

{
   "\t":1,
   " ":67,
   ",":4,
   ".":4,
   "E":1,
   "D":1,
   "L":1,
   "U":1,
   "a":29,
   "c":16,
   "b":3,
   "e":37,
   "d":18,
   "g":3,
   "f":3,
   "i":42,
   "h":1,
   "m":17,
   "l":21,
   "o":29,
   "n":24,
   "q":5,
   "p":11,
   "s":18,
   "r":22,
   "u":28,
   "t":32,
   "v":3,
   "x":3
}

The string.printable object is all ascii characters, including \t, etc. You can read more here.

The collections library has very high performance. You can go through gigabytes of text data in seconds.


Find large samples of text files is not so hard - here are a couple examples


I was curious. The entire works of Shakespeare takes 8 seconds to download and run on my old laptop. The results are

defaultdict(<type 'int'>, {u' ': 1293934, u'(': 628, u',': 83174, u'0': 299, u'4': 93, u'8': 40, u'<': 468, u'@': 8, u'D': 15683, u'H': 18462, u'L': 23858, u'P': 11939, u'T': 39800, u'X': 606, u'`': 1, u'd': 133779, u'h': 218406, u'l': 146161, u'p': 46525, u't': 289975, u'x': 4688, u'|': 33, u'#': 1, u"'": 31069, u'/': 5, u'3': 330, u'7': 41, u';': 17199, u'?': 10476, u'C': 21497, u'G': 11164, u'K': 6196, u'O': 33209, u'S': 34011, u'W': 16496, u'[': 2085, u'_': 71, u'c': 66688, u'g': 57035, u'k': 29212, u'o': 281391, u's': 214978, u'w': 72894, u'\n': 124456, u'"': 470, u'&': 21, u'*': 63, u'.': 78025, u'2': 366, u'6': 63, u':': 1827, u'>': 441, u'B': 15413, u'F': 11713, u'J': 2067, u'N': 27338, u'R': 28970, u'V': 3580, u'Z': 532, u'b': 46543, u'f': 68803, u'j': 2712, u'n': 215924, u'r': 208894, u'v': 33989, u'z': 1099, u'~': 1, u'!': 8844, u'%': 1, u')': 629, u'-': 8074, u'1': 928, u'5': 82, u'9': 948, u'=': 1, u'A': 44486, u'E': 42583, u'I': 55806, u'M': 15872, u'Q': 1178, u'U': 14129, u'Y': 9099, u']': 2077, u'a': 244664, u'e': 404621, u'i': 198184, u'm': 95580, u'q': 2404, u'u': 114818, u'y': 85271, u'}': 2})

To run the code above with a web-url, replace the line text= with

import requests
text = requests.get('http://ocw.mit.edu/ans7870/6/6.006/s08/lecturenotes/files/t8.shakespeare.txt').text
  • 1
    You can also just do Counter(text). – user76284 Nov 1 at 23:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.