RTP Payload Format for H.264 Video
obsoletes
- rfc-3984 — RTP Payload Format for H.264 Video
Extracted elements (30)
Aggregation packets (STAP/MTAP) were introduced to handle the dramatic MTU size difference between wireline IP networks (~1500-byte Ethernet) and wireless systems (often ≤254 bytes preferred transmission units), allowing efficient packing of small NAL units without media transcoding at the boundary.
Application-layer fragmentation (FUs) was chosen over IP-layer fragmentation to support NAL units exceeding 64 KB (possible in HD video with limited slice counts), and to enable per-fragment forward error correction. IP fragmentation cannot be controlled at the application layer and is generally avoided in modern networks.
H.264 parameter sets (SPS and PPS) are designed to be transmitted out-of-band (e.g., in SDP via sprop-parameter-sets) to create self-contained packets and allow parameter changes without synchronization with the slice packet stream. In-band transport is also supported for applications lacking a suitable out-of-band channel.
The Decoding Order Number (DON) was introduced to support interleaved transmission, where NAL units are intentionally sent out of decoding order to improve error resilience — for example, transmitting reference frames earlier than their decoding position. Without DON, a receiver has no way to reconstruct correct decoding order from RTP sequence numbers alone.
RFC 6184 obsoletes RFC 3984. Key additions include sprop-deint-buf-req and deint-buf-cap SDP parameters, clarified offer/answer procedures, and expanded packetization-mode semantics. Section 15 notes that endpoints implementing only RFC 3984 single NAL unit mode remain interoperable with RFC 6184 implementations using the same mode.
Aggregation packets MUST NOT contain fragmentation units, and aggregation packets MUST NOT be nested (an aggregation packet MUST NOT contain another aggregation packet). The total data MUST fit within an IP packet.
An H.264 encoder MUST set NRI according to H.264 Subclause 7.4.1 for nal_unit_type 1–12; NRI SHALL equal 0 for types 6, 9, 10, 11, and 12. Encoders SHOULD set NRI to 11 for sequence/picture parameter sets (types 7/8) and IDR coded slice NAL units (type 5).
DON field values MUST be such that the decoding order they imply conforms to the H.264 NAL unit decoding order. In single NAL unit and non-interleaved modes, the transmission order of packets MUST conform to NAL unit decoding order.
For all aggregation packets, the RTP timestamp MUST be the earliest NALU-time among all aggregated NAL units; the F bit MUST be cleared if all aggregated NAL unit F bits are zero, otherwise MUST be set; and the NRI value MUST be the maximum of all aggregated NAL unit NRI values.
Fragments of the same NAL unit MUST be sent in consecutive order with ascending RTP sequence numbers, with no other packets of the same RTP stream interleaved. A NAL unit MUST be reassembled in RTP sequence number order. The S and E bits MUST NOT both be set to 1 in the same FU header (a NAL unit MUST NOT be transmitted as a single FU).
In interleaved mode, FU-B MUST be used for the first fragmentation unit of each fragmented NAL unit and MUST NOT be used in any other case (subsequent fragments use FU-A). STAPs and MTAPs MUST NOT be fragmented. FUs MUST NOT be nested.
In interleaved mode (packetization-mode=2), STAP-Bs, MTAPs, FU-As, and FU-Bs MAY be used. STAP-As and single NAL unit packets MUST NOT be used in this mode.
In non-interleaved mode (packetization-mode=1), only single NAL unit packets, STAP-As, and FU-As MAY be used; STAP-Bs, MTAPs, and FU-Bs MUST NOT be used. Transmission order of NAL units MUST comply with NAL unit decoding order.
In single NAL unit mode (packetization-mode=0 or absent), only single NAL unit packets MAY be used; STAPs, MTAPs, and FUs MUST NOT be used. All receivers MUST support this mode. Transmission order MUST comply with NAL unit decoding order.
MANEs MUST NOT duplicate any NAL unit except for sequence or picture parameter set NAL units. Parameter set duplication MUST NOT affect any active parameter set, and duplication SHOULD be performed at the application layer rather than by duplicating RTP packets with identical sequence numbers.
Reserved NAL unit types (type 0 and types 30–31) SHOULD NOT be sent as packet payloads, as aggregation units, or as fragmented units; they MUST be ignored by receivers.
The RTP marker bit MUST be set for the very last packet of each access unit. For aggregation packets, the marker bit MUST reflect the value the last aggregated NAL unit's marker bit would have if transported individually. Decoders MAY use this as an early end-of-access-unit indicator but MUST NOT rely on it.
The RTP timestamp MUST use a 90 kHz clock rate. For NAL units without inherent timing (e.g., parameter sets, SEI NAL units), the RTP timestamp is set to the sampling timestamp of the primary coded picture of the access unit in which the NAL unit is included.
The Decoding Order Number (DON) is a 16-bit unsigned field (range 0–65535, wrapping modulo 65536) used in interleaved mode to signal NAL unit decoding order when transmission order differs. The don_diff(m,n) function uses modulo-65536 arithmetic to determine relative ordering; positive values mean n follows m in decoding order.
The NRI (nal_ref_idc) field serves dual purpose: zero indicates the NAL unit is not needed for reference picture reconstruction and may be discarded; non-zero values convey relative transport priority (11=highest, 10, 01, 00=lowest) that MANEs can use for selective packet dropping under congestion.
Three packetization modes are defined: single NAL unit mode (packetization-mode=0, targeting H.241-compatible conversational systems), non-interleaved mode (packetization-mode=1, low-delay), and interleaved mode (packetization-mode=2, tolerating higher latency). Each mode restricts which packet types are permitted.
Three RTP payload structures are defined: Single NAL Unit Packet (NAL unit types 1–23), Aggregation Packet (STAP-A=24, STAP-B=25, MTAP16=26, MTAP24=27), and Fragmentation Unit (FU-A=28, FU-B=29). The NAL unit type field in the first byte of the RTP payload identifies which structure is present.
This document registers the MIME media type 'video/H264' with IANA (updating RFC 3984), including SDP parameters: profile-level-id, packetization-mode, sprop-parameter-sets, max-mbps, max-fs, max-cpb, max-dpb, max-br, redundant-pic-cap, sprop-deint-buf-req, deint-buf-cap, sprop-interleaving-depth, sprop-max-don-diff, and sprop-init-buf-time.
H.264 RTP streams are subject to RTP security considerations (RFC 3550); SRTP (RFC 3711) SHOULD be used to provide confidentiality, integrity, and authentication. MANEs that manipulate RTP payloads must be trusted entities within SRTP-protected sessions, as they require access to the media content. The F bit can be used by MANEs to signal detected bit errors.
The de-interleaving receiver operates in two states: initial buffering (entered at RTP session start) and buffering-while-playing (entered after decoding/playback begins). In both states NAL units are stored in reception order with computed DON values; transition to buffering-while-playing is triggered when sufficient initial data is accumulated.
A Single NAL Unit Packet carries exactly one H.264 NAL unit; the first byte of the RTP payload is the NAL unit header (F|NRI|Type, 1 octet) followed by bytes 2..n of the NAL unit payload and optional RTP padding.
FU-A (type 28) payload: 1-octet FU indicator (F|NRI|Type=28), 1-octet FU header (S|E|R|Type), then fragmentation payload. FU-B (type 29) is identical but includes an additional 16-bit DON field (network byte order) between the FU header and payload. The FU header S bit marks the start and E bit marks the end of a fragmented NAL unit.
MTAP16 (type 26) and MTAP24 (type 27) payloads begin with a 16-bit DONB field. Each multi-time aggregation unit contains: 16-bit NAL unit size, 8-bit DOND, 16-bit (MTAP16) or 24-bit (MTAP24) timestamp offset, then the NAL unit. The DON of each unit equals (DONB + DOND) % 65536; the earliest aggregation unit's timestamp offset MUST be zero.
STAP-A (type 24) payload consists of one or more single-time aggregation units, each prefixed with a 16-bit unsigned NAL unit size (network byte order) followed by the NAL unit including its type byte. STAP-B (type 25) additionally prepends a 16-bit DON field before the aggregation units.
The NAL unit header is a single octet with three fields: F (forbidden_zero_bit, 1 bit), NRI (nal_ref_idc, 2 bits), and Type (nal_unit_type, 5 bits). This octet co-serves as the RTP payload header for all payload structures defined in this memo.