UTF-8, a transformation format of ISO 10646
obsoleted by
- rfc-3629 — UTF-8, a transformation format of ISO 10646
obsoletes
- rfc-2044 — UTF-8, a transformation format of Unicode and ISO 10646
Extracted elements (16)
If future incompatible changes occur to ISO/IEC 10646, the 'UTF-8' MIME charset label will remain aligned with the version prior to the incompatible change unless the IETF explicitly decides otherwise. The 'Korean mess' (Amendment 5 Hangul repositioning) is treated as a one-time exception justifying the legacy 'UNICODE-1-1-UTF-8' label rather than versioning the primary label.
The MIME charset label 'UTF-8' is intentionally version-independent. A MIME charset label's sole purpose is to enable interpretation of received bytes into characters; version tags serve no purpose when changes are compatible additions, and an older application encountering an unknown version-tagged label would fail entirely, whereas a generic known label triggers mostly correct processing.
UTF-8 was originally designed as a File System Safe UCS Transformation Format (FSS-UTF) for UNIX compatibility. Key design properties include: lexicographic sort order preservation for UCS-4 strings; Boyer-Moore search algorithm compatibility; and reliable auto-detection since the probability of a non-UTF-8 byte string appearing as valid UTF-8 decreases rapidly with string length.
ISO/IEC 10646 Amendment 5 moved and expanded the Korean Hangul block, rendering any pre-amendment data containing Hangul invalid under the new code point assignments. Unicode 2.0 contains the same incompatible difference from Unicode 1.1. The relevant committees have pledged not to make such incompatible changes again.
Any UTF-8 data not containing Hangul syllables encoded with pre-Amendment 5 ISO/IEC 10646 assignments SHOULD NOT use the charset label 'UNICODE-1-1-UTF-8'. Creating new Hangul-containing data without taking Amendment 5 into account is strongly discouraged.
Implementations of the UTF-8 decoding algorithm must protect against decoding invalid byte sequences. A naive decoder may wrongly interpret the illegal sequence C0 80 as U+0000 or 2F C0 AE 2E 2F as '/../', creating exploitable inconsistencies when security checks operate on the decoded form.
Encoding from UCS-4 to UTF-8 proceeds in three steps: (1) determine octet count from the character value range; (2) set the high-order prefix bits per the encoding table; (3) fill the x-bits from the character value starting at the least significant bits into the last octet and working left. There is exactly one valid encoding for each UCS-4 value; the rows of the table are mutually exclusive.
The first octet of a multi-byte sequence encodes the sequence length via the count of leading 1-bits (e.g., 110xxxxx means 2 bytes, 1110xxxx means 3 bytes). This enables character boundary resynchronization from any arbitrary position in an octet stream without rescanning from the start.
The octet values FE and FF never appear in any valid UTF-8 byte stream. This provides a reliable sentinel for encoding detection and distinguishes UTF-8 from UCS-2/UCS-4 byte sequences.
US-ASCII characters (U+0000–U+007F) are encoded as single octets identical to their ASCII values (00–7F). Octet values in the range 00–7F can only appear as encodings of US-ASCII characters and never as continuation or leading bytes of multi-byte sequences, ensuring full backward compatibility with ASCII-transparent software.
UTF-16 surrogate pairs — UCS-2 values in the range D800–DFFF — represent UCS-4 characters above U+FFFF and require special treatment when encoding UCS-2 to UTF-8. The UTF-16 transformation must first be reversed to recover the underlying UCS-4 code point, which is then encoded as a 4-byte UTF-8 sequence.
RFC 2279 provides the basis for registering 'UTF-8' as a MIME charset parameter value per the IANA Charset Registration Procedures (BCP 19, RFC 2278). The label applies to text containing characters from ISO/IEC 10646 including all amendments through at least Amendment 5, encoded in UTF-8.
The charset label 'UNICODE-1-1-UTF-8' is also proposed for registration, exclusively for UTF-8 data containing Hangul syllables encoded using pre-Amendment 5 ISO/IEC 10646 code point assignments. Its use is otherwise deprecated.
A parser that performs security-critical checks on the UTF-8 encoded form of input but also interprets illegal octet sequences as valid characters is vulnerable to bypass attacks. Examples: accepting C0 80 as U+0000 (NUL) while blocking 00, or accepting 2F C0 AE 2E 2F as '/../' while blocking 2F 2E 2E 2F — both allow security filters to be circumvented via overlong encodings.
Overlong encodings — representing a character using more octets than the minimum required — are illegal in UTF-8 and must be rejected by compliant decoders. Permitting them while simultaneously enforcing character-level restrictions creates exploitable inconsistencies that an attacker can leverage.
UTF-8 encodes UCS-4 characters as 1 to 6 octets using a self-synchronizing prefix scheme. The encoding table maps: U+0000–U+007F → 0xxxxxxx (1 byte); U+0080–U+07FF → 110xxxxx 10xxxxxx (2 bytes); U+0800–U+FFFF → 1110xxxx 10xxxxxx 10xxxxxx (3 bytes); U+10000–U+1FFFFF → 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx (4 bytes); with 5- and 6-byte forms extending to U+7FFFFFFF. Continuation octets always have the pattern 10xxxxxx.