Preparation of Internationalized Strings ("stringprep")
obsoleted by
- rfc-7564 — PRECIS Framework: Preparation, Enforcement, and Comparison of Internationalized Strings in Application Protocols
Extracted elements (30)
Allowing queries (but not stored strings) to contain unassigned code points eliminates the need for explicit versioning in protocols: old query applications can still find new stored strings because the unassigned code points pass through, while stored strings remain stable and comparable across profile versions.
Case folding maps uppercase to lowercase (rather than lowercase to uppercase) because there is a tradition of using lowercase in Internet applications and protocols. This choice is documented to explain a non-obvious directional convention.
NFKC was chosen over NFC because it maps 'compatibility characters' to their base equivalents, causing more strings that users expect to match to actually match. Some input mechanisms allow entering compatibility characters that look identical to base characters but have different code points.
The bidirectional rules (section 6) are optional in profiles, but omitting them makes it easy to create two strings with the same characters in different order that display identically, creating security-relevant user confusion analogous to look-alike character attacks.
The system using stringprep MUST be prepared to receive a longer or shorter string than the one input, because mapping and normalization can be one-to-none, one-to-many, or many-to-one transformations.
When two protocols using different profiles of stringprep interoperate, there may be conflict about which characters are allowed in the final string. Protocol developers are strongly encouraged to re-use existing registered profiles rather than creating new ones.
A profile of stringprep MUST include: the intended applicability, the character repertoire, which mapping tables are used, any additional profile-specific mappings, the Unicode normalization form (if any), the prohibited character tables used, the bidirectional checking used (if any), and any additional profile-specific prohibited characters.
Before output, text MUST be checked for prohibited code points. If any prohibited code point is found, only an error is returned; the process never emits both an error and a string. Implementations MUST map based on the tables in Appendix C, not the prose descriptions.
Each character in the input stream MUST be checked against a mapping table. The mapping table SHOULD come from this document, though a profile MAY add to or alter it. Mapped characters are not re-scanned during the mapping step.
Each profile of stringprep MUST be registered with IANA and MUST be reviewed by the IESG before registration. The IESG MAY change a profile before registration.
For backward compatibility, a subsequent version of a profile MUST NOT move code points from AO (allowed output), MN (mapped/normalized away), or D (disallowed) categories to any other category. Only unassigned (U) code points may change category in a newer profile version.
If a profile uses Unicode normalization, it MUST use normalization form KC (NFKC), not form C. A profile that specifies normalization MUST use the normalization tables associated with the version of Unicode specified for that profile.
In any profile that specifies bidirectional character handling, all three requirements MUST be met: (1) characters in section 5.8 MUST be prohibited; (2) if a string contains any RandALCat character, it MUST NOT contain any LCat character; (3) if a string contains any RandALCat character, a RandALCat character MUST be both the first and last character of the string.
Profiles SHOULD NOT use the stringprep process to 'correct' perceived mis-features or bugs in current character standards. Custom mapping tables in profiles SHOULD be based on UTR21 and SHOULD map from uppercase to lowercase.
Protocols that have sets of non-allowed characters or sequences MUST check for those non-allowed characters or sequences after stringprep processing, because stringprep mapping may introduce new characters.
Stored strings embedded in structures that cannot be changed (such as the signed parts of digital certificates) MUST NOT contain any unassigned code points.
Stored strings using a stringprep profile MUST NOT contain any unassigned code points. Queries for matching strings MAY contain unassigned code points. This is the only place where requirements for queries differ from requirements for stored strings.
The four preparation steps (Map, Normalize, Prohibit, Check bidi) MUST be performed in the stated order to comply with the specification.
The list of unassigned code points in Unicode 3.2 given in Appendix A MUST be used by implementations. If there are discrepancies between Appendix A and the Unicode 3.2 specification, Appendix A always takes precedence.
Appendix B.2 provides case-folding mapping tables for use with NFKC normalization; Appendix B.3 provides case-folding tables for use without normalization. Both map uppercase to lowercase and are derived from the Unicode 3.2 CaseFolding.txt file. B.2 adds entries for Greek characters and symbols containing Latin characters that B.3 omits.
Categories of prohibited output characters include: space characters (C.1), control characters (C.2), private-use code points (C.3), non-character code points (C.4), surrogate codes D800–DFFF (C.5), characters inappropriate for plain text (C.6), ideographic description characters (C.7), characters that change display properties or are deprecated (C.8), and tagging characters (C.9).
Characters commonly mapped to nothing (deleted from input) include: SOFT HYPHEN (00AD), ZERO WIDTH SPACE (200B), WORD JOINER (2060), ZERO WIDTH NO-BREAK SPACE (FEFF), COMBINING GRAPHEME JOINER (034F), ZERO WIDTH NON-JOINER (200C), ZERO WIDTH JOINER (200D), and Variation Selectors FE00–FE0F. These are listed in Table B.1.
Code points are categorized as: AO (allowed in output), MN (eliminated by mapping or normalization), D (disallowed by prohibition step), and U (unassigned). Stored strings MUST only contain AO code points; queries treat U code points as if they were AO.
Surrogate codes D800–DFFF are permanently reserved for UTF-16 encoding and will never be assigned to characters; they are prohibited output in all stringprep profiles.
The stringprep preparation pipeline consists of four ordered steps: (1) Map each character using mapping tables, (2) Normalize using Unicode normalization, (3) Prohibit disallowed code points, and (4) Check bidirectional string requirements. Each step may transform or reject the string.
IANA maintains a registry of stringprep profiles as a text file listing each profile's name, the RFC defining it, and whether it is the current version. All versions of a profile remain listed permanently; superseded versions have their current-version indicator turned off.
Applications using UTF-8 or UTF-16 MUST reject any string that is an illegal or ill-formed octet sequence. Failure to validate encoding (e.g., accepting 0xC0 0xAB as U+002B) is a known vulnerability class in Unicode-aware applications.
Some Unicode code points are invisible. Protocols that permit invisible characters (by not mapping them out or prohibiting them) can cause user confusion when two identical-looking strings do not compare as equal.
Unicode contains many visually similar characters. Stringprep does not map similar-looking characters together or prohibit characters because they resemble others, as doing so without full font context would be unreliable. User confusion from visual similarity (spoofing) remains a residual risk that applications must address independently.
Unicode normalization and encoding conversion can cause strings to grow or shrink. Programs using fixed-size buffers or making assumptions about buffer bounds are likely to fail insecurely when processing Unicode normalization or encoding conversions.