UTF-8, a transformation format of ISO 10646
Extracted elements (22)
The MIME charset label 'UTF-8' is version-independent by design: a label without a version number allows older applications to process newer data correctly when no new characters are present, whereas an unknown version-specific label would cause outright failure. The 'Korean mess' incompatible change is deemed an acceptable historical exception because no significant data using the old Hangul encoding existed.
US-ASCII characters (U+0000–U+007F) are encoded as single octets with their normal ASCII values, and no multi-byte sequence ever produces a byte in the 0x00–0x7F range. This preserves full compatibility with software that parses based on ASCII byte values while being transparent to non-ASCII bytes.
UTF-8's range is restricted to U+0000–U+10FFFF (the UTF-16 accessible range), eliminating the 5- and 6-byte sequences previously allowed by ISO/IEC 10646 for code points up to U+7FFFFFFF. This aligns UTF-8 with Unicode's scalar value range and reduces buffer-overflow risk.
UTF-8 was devised in September 1992 by Ken Thompson under design criteria from Rob Pike, with the goal of supporting UCS in the Plan 9 operating system without disrupting existing ASCII-based software. The encoding scheme ensures lexicographic byte-order sorting matches Unicode code-point order and allows character boundaries to be found from any position in an octet stream.
In 1996, Amendment 5 to ISO/IEC 10646 and Unicode 2.0 moved and expanded the Korean Hangul block, making all prior Hangul-containing data invalid under the new version. This 'Korean mess' is the only known incompatible change; the relevant committees have committed never to repeat such a change.
When encoding to UTF-8 from UTF-16 data, the UTF-16 data must first be fully decoded to Unicode code points (resolving surrogate pairs), which are then encoded in UTF-8. Directly encoding UTF-16 code unit values produces CESU-8, which is not valid UTF-8.
A protocol SHOULD forbid use of U+FEFF as a signature for textual protocol elements that are mandated to always be UTF-8 (since the signature function is useless there), and SHOULD also forbid it when the protocol already provides character-encoding identification mechanisms used consistently by implementations.
A protocol SHOULD NOT forbid use of U+FEFF as a signature for protocol elements where no encoding-identification mechanism exists, where a ban would be unenforceable, or where implementations cannot be expected to always use such mechanisms properly—as is often the case with large MIME entities obtained from file systems or FTP.
A U+FEFF character appearing at any position other than the beginning of a stream MUST be interpreted as 'ZERO WIDTH NO-BREAK SPACE' and MUST NOT be interpreted as a signature.
Implementations of the UTF-8 decoding algorithm MUST protect against decoding invalid sequences. A naive implementation might decode overlong sequence C0 80 as U+0000 or surrogate pair ED A1 8C ED BE B4 as U+233B4; such behavior can have security consequences.
It is RECOMMENDED to avoid stripping an initial U+FEFF interpreted as a BOM signature without a good reason, to ignore it instead of stripping it when appropriate (e.g., for display), and to strip it only when truly necessary, because stripping may affect external processes such as digital signatures or character counts.
When a protocol forbids use of U+FEFF as a signature for a given protocol element, any initial U+FEFF in that element MUST be interpreted as 'ZERO WIDTH NO-BREAK SPACE'. When a protocol does NOT forbid it, implementations SHOULD be prepared to handle a signature and react appropriately.
CESU-8 is a UTF-8-like encoding that encodes UTF-16 code unit values (16-bit quantities) rather than Unicode code points, producing different results for characters above U+FFFF. CESU-8 encodings of characters above U+FFFF are NOT valid UTF-8 and are not intended for Internet use.
The Byte Order Mark (BOM) is the UCS character U+FEFF 'ZERO WIDTH NO-BREAK SPACE'. In UTF-8 it always appears as the 3-octet sequence EF BB BF. It may be prepended to a stream as a signature to indicate UTF-8 encoding, but its signature function is less useful in UTF-8 than in multi-byte encodings because byte order is not at issue.
The MIME charset label for UTF-8 is the string 'UTF-8'. It labels media types containing text from the ISO/IEC 10646 repertoire (including all amendments through amendment 5) encoded per the UTF-8 scheme, and is suitable under the MIME 'text' top-level type.
The valid UTF-8 code point range is U+0000 to U+10FFFF (the UTF-16 accessible range). Code points U+D800 through U+DFFF (UTF-16 surrogate pairs) are explicitly prohibited in UTF-8 and must not be directly encoded.
The IANA charset registry entry for 'UTF-8' has been updated to point to RFC 3629. The charset parameter value is the string 'UTF-8' and applies to ISO/IEC 10646 characters encoded per the UTF-8 scheme defined in this memo.
Buffer overflow risk exists when encoding to UTF-8 if the allowed character number range is not explicitly limited to U+10FFFF, because the ISO/IEC 10646 description historically allowed sequences up to 6 bytes for code points up to U+7FFFFFFF. Buffer sizing must account for the actual maximum sequence length.
Parsers that perform security-critical checks against the UTF-8 encoded form but accept illegal overlong sequences are vulnerable to attack. The exploit of allowing C0 80 as U+0000 (to bypass NUL checks) or 2F C0 AE 2E 2F to bypass '/../' path traversal checks was actively used in a 2001 Web-server virus, demonstrating this is a real-world threat.
The same user-visible character can be represented by multiple distinct Unicode character sequences (e.g., precomposed U+00E9 vs. U+0065 U+0301), which can undermine string matching in security-sensitive contexts such as credential identifiers versus access-control list entries. Unicode Normalization Forms (UAX #15) are the recommended mitigation.
ABNF syntax for valid UTF-8: UTF8-1 = %x00-7F; UTF8-2 = %xC2-DF UTF8-tail; UTF8-3 handles ranges %xE0/%xE1-EC/%xED/%xEE-EF with appropriate tail constraints; UTF8-4 handles %xF0/%xF1-F3/%xF4 with appropriate tail constraints; UTF8-tail = %x80-BF. Octet values C0, C1, and F5–FF are excluded.
UTF-8 encodes code points U+0000–U+10FFFF as 1 to 4 octets. Single-octet sequences have bit pattern 0xxxxxxx (U+0000–U+007F); 2-octet sequences start with 110xxxxx; 3-octet with 1110xxxx; 4-octet with 11110xxx. Continuation octets always have pattern 10xxxxxx, providing 6 payload bits each.