Python 3.15 Unicode Normalization and Case Folding

Unicode normalization and case folding help compare text consistently.

How It Works

Different code point sequences can look identical. unicodedata.normalize can bring them to a common form, and str.casefold() provides a stronger case-insensitive transformation than lower() for many languages.

Example

import unicodedata
first_word = "café"
second_word = "cafe\u0301"
print(first_word == second_word)
print(unicodedata.normalize("NFC", first_word) == unicodedata.normalize("NFC", second_word))

The comparison is initially False but becomes True after normalization.

Expected Output

False
True

Important Notes

Normalization and case folding do not solve all language-specific collation or security problems.

Try It Yourself

Normalize two equivalent spellings and compare their lengths.

Common Beginner Mistakes

• Normalization and case folding do not solve all language-specific collation or security problems.
• Comparing visibly identical strings without first checking their underlying normalization forms.
• Treating casefold() as a universal replacement for locale-specific collation or validation.

Key Takeaways

• Unicode normalization and case folding help compare text consistently.
• Different code point sequences can look identical. unicodedata.normalize can bring them to a common form, and str.casefold() provides a stronger case-insensitive transformation than lower() for many languages.
• Normalize when appropriate, then compare in a way that matches the application's language requirements.