Unicode Format for Network Interchange
Extracted elements (26)
CRLF was retained as the sole line-ending sequence (rather than adopting LS U+2028) because of the large installed base of CRLF usage and the importance of forward compatibility with NVT and protocols that depend on it.
NFC normalization was selected over NFKC and other forms because it is closest to actual use in practice, minimizes side-effects from treating characters as equivalent that may not be in all situations, and typically requires the least work converting from non-Unicode encodings.
The BOM is prohibited because, unlike UTF-16, UTF-8 has no byte-order ambiguity; including the BOM would waste bytes and create interoperability problems with receivers that treat it as data.
The 'Stable NFC' approach (option 4 in Section 5.2) — rejecting unassigned code points rather than mapping them to themselves — was identified as the most reasonable compromise for NFC stability under ongoing Unicode expansion.
This specification applies starting with Unicode 5.0 and not to any version that changes UTF-8 encoding or NFC stability. Freezing on an older version (like Stringprep's 3.2) was rejected because library-based application implementations make version linkages nearly meaningless in practice.
UTF-8 was chosen over UTF-16 and UTF-32 because of its good compatibility properties with ASCII and for reasons discussed in the IETF character set policy (RFC 2277), enabling a smooth upgrade path from NVT ASCII.
Applications typically rely on OS or language library functions for Unicode conversion and normalization; there may be no plausible way for an application to know which Unicode version or normalization procedure it is using, nor to guarantee consistency between the two.
Receivers of Net-Unicode strings should be prepared to receive unnormalized strings and not react excessively, because some implementations rely on OS libraries over which they have little control.
The Telnet IAC octet (%xFF) cannot appear in valid UTF-8, so it poses no ambiguity; however, other Telnet command-introducer octets may be valid UTF-8 subsequences, requiring caution in Telnet-over-Net-Unicode implementations.
This specification is intended for protocols that have not yet defined how to use Unicode; it is not intended for specifications that already allow UTF-8 and precisely define that use (e.g., MIME content types with explicit charset labels).
Before transmission, all character sequences SHOULD be normalized according to Unicode normalization form NFC.
Characters MUST be encoded in UTF-8 as defined in RFC 3629.
CR MUST NOT appear unless it is immediately followed by LF or NUL; the CR NUL sequence SHOULD be avoided because NUL is hostile to programming languages that use it as a string delimiter.
If the protocol has the concept of 'lines', line-endings MUST be indicated by the sequence CR (U+000D) followed by LF (U+000A). CR SHOULD NOT appear except when followed by LF.
IND (U+0084), NEL (U+0085), line separator (U+2028), and paragraph separator (U+2029) MUST NOT be used as line-ending indicators; lines end in CRLF only.
Strings that are received by a protocol referencing this specification and that appear to be UTF-8 and are not otherwise identified SHOULD be treated as using UTF-8 in conformance with this specification.
Systems conforming to this specification MUST NOT transmit any string containing any code point that is unassigned in the version of Unicode on which they are dependent. The version of NFC and the version of Unicode used MUST be consistent.
The Byte Order Mark (BOM) signature MUST NOT appear at the beginning of Net-Unicode text strings.
The C1 Controls (U+0080 through U+009F) MUST NOT appear in Net-Unicode text.
Unicode private-use code points SHOULD be avoided in strings intended for Internet interchange because they lack standard definitions or normalization interpretations.
Unicode SHOULD be used with the Net-Unicode text encoding when there is a choice, in preference to double-byte 'extended ASCII' or per-language character coding systems.
ASCII control characters U+0000–U+001F and U+007F–U+009F SHOULD generally be avoided; SP (U+0020), CR, LF, and FF (U+000C) are exceptions, though use of all but SP requires care.
Net-Unicode is defined as UTF-8 encoded text with CRLF line endings, NFC normalization, no BOM, no C1 controls, and no unassigned code points. It serves as the internationalized successor to NVT ASCII.
If assumptions about future Unicode versions, NFC normalization stability, or UTF-8 processing per RFC 3629 are violated, strings considered equivalent may not compare equal; robust implementations must be prepared for these possibilities.
NUL (CR NUL sequence) may be a trap for programming languages that use NUL as a string terminator; this is an additional reason to avoid CR except when followed by LF.
Programs receiving Net-Unicode streams should use extreme caution about assuming incoming data are normalized; attackers may deliberately send unnormalized text or invalid UTF-8 to evade naive text-matching systems such as firewalls.