RTP Payload Format for H.264 Video
obsoleted by
- rfc-6184 — RTP Payload Format for H.264 Video
Extracted elements (29)
Aggregation packets (STAP and MTAP) were introduced to bridge dramatically different MTU sizes: wireline IP networks (typically ~1500-byte Ethernet MTU) versus IP/non-IP wireless systems such as ITU-T H.324/M (preferred ≤254 bytes). Without aggregation, transcoding between these environments would be necessary at gateways.
Application-layer fragmentation via FUs was preferred over IP-level fragmentation for two reasons: it permits NAL units larger than 64 KB (possible in HD pre-recorded content) to traverse IPv4 networks, and it enables applying generic Forward Error Correction (FEC, RFC 2733) at the RTP level on individual NAL unit fragments.
The Decoding Order Number (DON) enables interleaved packetization where NAL units are transmitted out of decoding order to support robust packet scheduling — for example, sending redundant coded pictures ahead of their decoding time or pre-transmitting intra pictures. A 16-bit modular counter is used because the sender may not know at assignment time how many NAL units will be encoded before a later one arrives, allowing gaps in DON values.
Because H.264 decoding order may differ from display order, RTP timestamps for H.264 streams may not be monotonically non-decreasing as a function of RTP sequence numbers. As a result, the RTCP interarrival jitter calculation defined in RFC 3550 section 6.4.1 may not reliably reflect actual network performance.
Single NAL Unit Mode (packetization-mode=0) is required for interoperability with ITU-T H.241-based video telephony systems. The MANEs' use of F=1 to signal detected bit errors extends H.264 semantics, since the H.264 specification requires F=0; receivers aware of this extension can use F=1 as a hint to discard and conceal the affected NAL unit.
A 90 kHz clock rate MUST be used for the RTP timestamp. For NAL units without their own timing properties (e.g., parameter sets and SEI NAL units), the RTP timestamp is set to the timestamp of the primary coded picture of the access unit in which the NAL unit is included.
Aggregation packets MUST NOT contain fragmentation units. Aggregation packets MUST NOT be nested (an aggregation packet MUST NOT contain another aggregation packet). Total size SHOULD be chosen so the resulting IP packet is smaller than the path MTU.
All receivers MUST support Single NAL Unit Mode (packetization-mode=0 or absent). In this mode only single NAL unit packets MAY be used; STAPs, MTAPs, and FUs MUST NOT be used, and transmission order MUST comply with NAL unit decoding order.
An H.264 encoder MUST set NRI per the H.264 specification (subclause 7.4.1) for nal_unit_type 1-12, and SHOULD set NRI=11 for sequence parameter sets (type 7), picture parameter sets (type 8), and IDR coded slice NAL units (type 5).
DON-related fields (DON, DONB, DOND) MUST produce a decoding order that conforms to NAL unit decoding order per H.264 section 7.4.1.2. NAL units whose order is switched without preserving decoding order MUST NOT have the same DON value.
Each octet of a NAL unit MUST be part of exactly one fragment. Fragments of the same NAL unit MUST be sent in consecutive ascending RTP sequence number order with no intervening packets from the same stream. STAPs and MTAPs MUST NOT be fragmented; FUs MUST NOT be nested.
For aggregation packets, the RTP timestamp MUST be set to the earliest NALU-time of all aggregated units. The F bit MUST be cleared only if all aggregated NAL units' F bits are zero, otherwise MUST be set. The NRI field MUST be the maximum NRI of all contained NAL units.
Implementations MUST follow the congestion control requirements for RTP as specified in applicable RTP profiles. Senders must reduce their transmission rate when congestion is detected to avoid contributing to network collapse.
In interleaved mode, FU-B MUST be used for the first fragmentation unit of a fragmented NAL unit; FU-B MUST NOT be used in any other case. All subsequent fragments of that NAL unit MUST use FU-A.
MANEs MUST NOT duplicate any NAL unit except sequence or picture parameter set NAL units. Any allowed parameter set duplication MUST NOT affect the contents of any active parameter set and SHOULD be performed at the application layer rather than by duplicating RTP packets with identical sequence numbers.
Non-Interleaved Mode (packetization-mode=1) SHOULD be supported. Only single NAL unit packets, STAP-As, and FU-As MAY be used; STAP-Bs, MTAPs, and FU-Bs MUST NOT be used. Interleaved Mode (packetization-mode=2) requires STAP-Bs, MTAPs, FU-As, and FU-Bs; STAP-As and single NAL unit packets MUST NOT be used.
The RTP marker bit MUST be set for the very last packet of the access unit indicated by the RTP timestamp. For aggregation packets, the M bit MUST reflect what the last aggregated NAL unit's M bit would have been if transported alone. Decoders MAY use the M bit as an early indicator but MUST NOT rely on this property.
The Decoding Order Number (DON) is a 16-bit field in the range 0-65535 (wrapping around) used in interleaved mode to convey NAL unit decoding order when it differs from transmission order. The function don_diff(m,n) uses modular arithmetic with 32768 as the half-period to determine relative ordering between two DON values.
The MIME type video/H264 is defined for this payload format. Key MIME parameters include: packetization-mode (0/1/2), sprop-parameter-sets (base64-encoded SPS/PPS for out-of-band delivery), profile-level-id (3-octet hex), sprop-interleaving-depth, sprop-deint-buf-req, deint-buf-cap, sprop-init-buf-time, and sprop-max-don-diff.
The NRI (nal_ref_idc) field in the NAL unit octet indicates relative transport priority for MANEs: value 11 (binary) is highest, 10 and 01 are intermediate, and 00 is lowest (non-reference content that can be discarded without harming reference picture integrity).
The payload format defines three packet structures identified by the first byte (NAL unit type): Single NAL Unit Packet (types 1-23), Aggregation Packets (STAP-A=24, STAP-B=25, MTAP16=26, MTAP24=27), and Fragmentation Units (FU-A=28, FU-B=29). Types 0 and 30-31 are undefined; receivers MUST ignore them.
Three packetization modes are defined: Single NAL Unit Mode (packetization-mode=0, default) for H.241-compliant conversational systems; Non-Interleaved Mode (packetization-mode=1) for low-delay applications; and Interleaved Mode (packetization-mode=2) for systems tolerating higher latency. Each mode restricts which NAL unit types 24-29 are permitted.
This document registers the MIME media type video/H264 with IANA for the H.264/AVC RTP payload format described herein. The registration includes all MIME parameters defined in section 8.1.
The payload format carries compressed video and provides no inherent security mechanisms. SRTP (RFC 3711) is RECOMMENDED for confidentiality, integrity, and replay protection. MANEs that parse and modify RTP payloads must be trusted entities; when SRTP is in use, they require access to decrypted content to perform their manipulation functions.
A Single NAL Unit Packet contains exactly one H.264 NAL unit; the NAL unit type octet (F|NRI|Type) is the first byte and also serves as the RTP payload header. Only nal_unit_type values 1-23 are valid in this packet type.
FU-A structure: 1-byte FU indicator (F|NRI|Type=28), 1-byte FU header (S|E|R|Type), and variable-length FU payload. FU-B is identical but inserts a 16-bit DON field between the FU header and payload. S=1 marks the first fragment; E=1 marks the last; both MUST NOT be set simultaneously in one FU.
MTAP16/MTAP24 payloads begin with a 16-bit DONB, followed by multi-time aggregation units each containing: 16-bit NAL unit size, 8-bit DOND, n-bit timestamp offset (16 bits for MTAP16, 24 bits for MTAP24), and the NAL unit. The NAL unit's DON equals (DONB + DOND) % 65536. The earliest NAL unit's timestamp offset MUST be zero.
STAP-A payload: NAL unit header byte followed by one or more single-time aggregation units, each a 16-bit unsigned size (network byte order) prefixed to the NAL unit. STAP-B prepends a 16-bit DON field before the aggregation units; each successive NAL unit's DON is (prior DON + 1) % 65536.
The NAL unit type octet serves as both the NAL header and the RTP payload header. It contains three fields: F (1-bit forbidden_zero_bit), NRI (2-bit nal_ref_idc indicating reference importance), and Type (5-bit nal_unit_type identifying the payload structure).