5

I'm looking for a csv file containing the utf-8 encoding table. This file would contain the code point, the escaped html entity and the symbol itself.

I already found websites providing these correspondances, like this one or this other one. However, they do not provide an offline access and pasting all the content would be, while technically possible, a little bit tedious.

Would you know where such a file is available ?

5
  • You may want to ask on stackoverflow.com, too.
    – philshem
    Apr 9, 2014 at 14:55
  • The best website for Unicode characters is fileformat.info Apr 11, 2014 at 22:14
  • Be aware that some Unicode "characters" are "combining marks", which bind with the preceding character to form a grapheme (e.g., an accent that combines with a latin letter) Apr 11, 2014 at 22:17
  • 2
    If I were you, I would base my work on the official Unicode documents at www.unicode.org, in particular unicode.org/reports and even more in particular the UnicodeData.txt file Apr 14, 2014 at 7:18
  • 2
    Numeric HTML entities are easily built as &#x<HEX>;, where <HEX> is the Unicode codepoint in hex (the first column of the UnicodeData.txt file, with or without leading zeros) Apr 15, 2014 at 16:18

1 Answer 1

5

With python you can lookup each unicode character by its integer code using unichr.

import sys
with open('unicode.csv','wb') as output:
    for i in xrange(sys.maxunicode):
        output.write(unicode(i))
        output.write(u',')
        output.write(unichr(i).encode('utf-8'))
        output.write(u',')
        output.write(unichr(i).encode('ascii', 'xmlcharrefreplace'))
        output.write(u'\n')
print sys.maxunicode

This gives you a file (unicode.csv) which has the unciode integer, unicode representation, unicode character, and HTML escaped character (for non-ascii).

For example, each line looks like this:

64058,u'\ufa3a',墨,&#64058;

I put the code and the unicode.csv file on github for easier access.

Note: Because the unicode character set includes newline characters, CSV is not really the best format. (See lines 10 to 13.) I also added a python code to generate a JSON file, which is more safe than CSV for storing unicode characters.

1
  • The best solution would be to generate the lookup on the fly (my code takes about 1 second, but the load times for CSV and JSON are much longer).
    – philshem
    Apr 9, 2014 at 14:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.