unicodedata — character properties and normalization

unicodedata.normalize with NFC and NFKC, plus name, category, and combining, checked on Python 3.13.12. Normalization works on code points, not user-perceived glyphs.

The unicodedata module looks up Unicode character properties and applies Unicode Normalization Forms. It works one code point at a time (plus the normalize() algorithm over a string). It does not implement grapheme-cluster segmentation, shaping, or a font. A user-perceived character may be several code points.

NFC and NFD are the canonical forms: they rearrange and compose or decompose according to canonical mappings, and they are meant for testing canonical equivalence. NFKC and NFKD apply compatibility mappings first. Those compatibility mappings can be destructive: a typographic ligature, circled digit, or fullwidth letter may become a plain ASCII sequence that cannot be reversed uniquely.

Examples were run against Python 3.13.12. Outputs list code points in hexadecimal so combining marks stay visible.

Examples were checked against Python 3.13. See the official documentation for the complete API.

Python reference overview →

NFC, NFD, and canonical equivalence

The precomposed string "café" and the decomposed sequence "cafe" plus U+0301 COMBINING ACUTE ACCENT are not the same Python str: they have different lengths and compare unequal. After NFC they are equal; after NFD they are also equal to each other, as a five-code-point sequence.

Use NFC when you want a composed form for storage or comparison. Use NFD when you need the decomposed form. Neither form tells you how many glyphs a renderer will draw.

normalize NFC and NFD

unicodedata.normalize(form, unistr)

NFC and NFD implement canonical composition and decomposition. Equal appearance does not mean equal Python strings until they are normalized the same way.

ParameterDescription
formOne of "NFC", "NFD", "NFKC", or "NFKD".
unistrThe Unicode string to normalize.

Returns: A new normalized str.

Raises

  • ValueError — Raised when form is not one of the four supported names.
python
import unicodedata

composed = 'café'
decomposed = 'cafe\u0301'
print(composed == decomposed)
print(unicodedata.normalize('NFC', composed) == unicodedata.normalize('NFC', decomposed))
for form in ('NFC', 'NFD'):
    for label, value in (('composed', composed), ('decomposed', decomposed)):
        out = unicodedata.normalize(form, value)
        print(form, label, len(out), [hex(ord(ch)) for ch in out])

Expected output

False
True
NFC composed 4 ['0x63', '0x61', '0x66', '0xe9']
NFC decomposed 4 ['0x63', '0x61', '0x66', '0xe9']
NFD composed 5 ['0x63', '0x61', '0x66', '0x65', '0x301']
NFD decomposed 5 ['0x63', '0x61', '0x66', '0x65', '0x301']

Related: Unicode inspector tool, str methods

Official documentation →

NFKC can be lossy, and properties are per code point

NFKC applies compatibility mappings and then composes. TRADE MARK SIGN becomes the two letters TM, a ligature becomes two letters, a circled digit becomes a plain digit, and a fullwidth Latin letter becomes ASCII. Those conversions are not reversible from the result alone, so do not use NFKC as a lossless identifier fold unless you accept that loss.

name(), category(), combining(), and east_asian_width() describe a single code point, not a grapheme cluster. combining() is 0 for a base character and a combining class for a combining mark. east_asian_width("宽") is "W" (wide), which is a property of that code point, not a measurement of screen columns in a terminal or a browser.

NFKC, name, category, and combining

unicodedata.normalize(form, unistr)
unicodedata.name(chr, default=...)
unicodedata.category(chr)
unicodedata.combining(chr)
unicodedata.east_asian_width(chr)

NFKC compatibility mappings can destroy typographic distinctions. Character APIs take one code point, not a glyph cluster.

ParameterDescription
chrA Unicode character (a str of length one). These functions do not accept a grapheme cluster of several code points as a single character.
defaultIf supplied to name(), it is returned when the character has no official name. Without it, a missing name raises ValueError.

Returns: name() returns a str. category() and east_asian_width() return short property strings. combining() returns an integer class, or 0 if the character is not combining.

Raises

  • ValueError — name() raises ValueError when the character is unnamed and no default is given.
  • The final print is the code-point length of NFD "café", which is 5, even though a reader may see four letters.
python
import unicodedata

for char in ('™', 'fi', '①', 'A'):
    print(char, unicodedata.normalize('NFKC', char), unicodedata.name(char), unicodedata.category(char))
print(unicodedata.combining('\u0301'))
print(unicodedata.east_asian_width('宽'))
print(len(unicodedata.normalize('NFD', 'café')))

Expected output

™ TM TRADE MARK SIGN So
fi fi LATIN SMALL LIGATURE FI Ll
① 1 CIRCLED DIGIT ONE No
A A FULLWIDTH LATIN CAPITAL LETTER A Lu
230
W
5

Related: Unicode inspector tool, NFC and NFD

Official documentation →