Unicode Character Repertoire Subsets
Extracted elements (20)
Private-use code points (Definition D49) are not classified as problematic because Unicode considers them assigned characters whose use may be determined by private agreement among cooperating users.
Subset restrictions apply to data content, not textual representation: escaping techniques such as JSON \uXXXX sequences cannot be used to circumvent restrictions on field values. A JSON text may be syntactically conforming even when the data it encodes contains prohibited code points.
The 'set of all Unicode code points' is not a good choice for text fields because Unicode recognizes different code point types, not all of which are appropriate in protocols or even associated with characters. Specifications should reference a named subset excluding problematic code points.
The XML Characters subset retains C1 controls because they are less likely than C0 controls to appear erroneously in data and have not been observed as a frequent source of problems; noncharacters above U+FFFF are also retained because they are rarely encountered.
Three increasingly restrictive subsets are defined — Unicode Scalars, XML Characters, and Unicode Assignables — so that specifications can choose an appropriate level of restriction. Unicode Assignables is the new recommended subset that excludes all problematic code points and is a proper subset of the other two.
Unassigned code points are generally not excluded from subsets because they regularly become assigned when new Unicode characters are added; specifying their avoidance would break forward compatibility with future Unicode versions.
JSON (RFC 8259) allows any Unicode code point — including all problematic types — in object member names and string values. A JSON-based protocol can restrict field values to one of the three named subsets to improve robustness; equivalent restrictions apply to I-JSON, YAML, CBOR, and XML.
JSON's permissiveness means that some libraries silently parse strings containing surrogates and generate ill-formed UTF-8 output. The example field value containing NUL, a C1 control, an unpaired surrogate, and a noncharacter is a conforming JSON text yet cannot be serialized into well-formed UTF-8.
The Unicode Scalars subset is the default for CBOR (RFC 8949) and corresponds to Definition D76 from the Unicode Standard. JSON, by contrast, permits all code points including surrogates and noncharacters, creating implementation divergence when libraries parse strings containing such values.
BCP 18 (RFC 2277) mandates that 'Protocols MUST be able to use the UTF-8 charset', making UTF-8 the required encoding form for any protocol or data format that specifies a single encoding.
Control codes comprise 65 code points: C0 controls (U+0000–U+001F), C1 controls (U+0080–U+009F), and DEL (U+007F). The 'useful controls' are newline (U+000A), carriage return (U+000D), and tab (U+0009); all others are 'legacy controls' with obsolete or non-interoperable semantics.
Noncharacters include the last two code points in each of the 17 Unicode planes (U+FFFE, U+FFFF through U+10FFFE, U+10FFFF) and the range U+FDD0–U+FDEF; they are not designed for open interchange.
Surrogates are 2,048 code points in the range U+D800–U+DFFF, used in UTF-16 to represent code points above U+FFFF. A surrogate occurring in any encoding other than UTF-16 has no meaning, and UTF-8 forbids encoding surrogates.
Silently deleting an ill-formed part of a string is a known security risk. The Unicode Standard (Section 3.2) recommends signaling an error or replacing problematic code points with U+FFFD (REPLACEMENT CHARACTER) rather than silently discarding them.
The behavior of software that encounters surrogates is unpredictable and differs among programming-language implementations, even between different API calls in the same language. Industry experience shows surrogates can appear in uncontrolled input even when encodings forbid them.
The PRECIS framework's IdentifierClass and FreeformClass exclude many code points beyond the problematic ones defined here. Specifications of fields designed for display to or interaction with humans should consider PRECIS's more-restrictive subsets, as they address spoofing and homograph attacks.
Unicode Security Considerations (TR36) surveys attacks including those involving surrogates and noncharacters that can contribute to human-deceiving exploits. Unicode Source Code Handling (TR55) addresses related security issues in programming-language contexts.
The 'Unicode Assignables' subset is expressed in ABNF spanning all 17 planes, excluding surrogates, all legacy controls (C0 except useful controls, C1, and DEL), and all noncharacters (U+FDD0–U+FDEF plus last two per plane). It is the most restrictive of the three subsets and the one newly introduced by this document.
The 'Unicode Scalars' subset is expressed in ABNF as %x0-D7FF / %xE000-10FFFF. It excludes surrogates but retains legacy controls and noncharacters. This is the default for CBOR (RFC 8949).
The 'XML Characters' subset is expressed in ABNF as %x9 / %xA / %xD / %x20-D7FF / %xE000-FFFD / %x10000-10FFFF. It excludes surrogates, legacy C0 controls, and the noncharacters U+FFFE and U+FFFF, but retains C1 controls and noncharacters above U+FFFF.