RTP Payload Format for Transport of MPEG-4 Elementary Streams
updated by
- rfc-5691 — RTP Payload Format for Elementary Streams with MPEG Surround Multi-Channel Audio
Extracted elements (29)
CTS is encoded as a 2's complement offset (delta) from the RTP header timestamp, and DTS is encoded as a 2's complement offset from the CTS. This differential encoding keeps time stamp coding efficient while avoiding redundant transmission of the first AU's timestamp (which is already in the RTP header).
Interleaving of Access Units is supported to increase error resiliency against packet loss at a modest cost in latency and implementation complexity. The receiver does not need to know the sender's interleaving pattern a priori; it only uses the AU-Index information to place each AU into the correct position in the decoding queue.
Payload fields are made configurable via MIME format parameters rather than fixed, because their use strongly depends on the type of MPEG-4 elementary stream being carried. A length of zero indicates absence of a field, allowing a minimal 'basic' configuration with empty AU-header and Auxiliary sections when no optional features are needed.
Specific transport modes (CELP-cbr, CELP-vbr, AAC-lbr, AAC-hbr, generic) are required to allow the design of simplified, dedicated receivers that need only implement one mode, rather than requiring all receivers to be fully configurable for any MPEG-4 stream.
This payload format can be configured to be nearly identical to RFC 3016 (MPEG-4 video). Receivers compliant with this specification SHOULD be able to decode RFC 3016 payloads, and it is strongly RECOMMENDED that implementations be able to ignore 'in band' video decoder configuration packets found in RFC 3016 streams.
An RTP packet SHALL either carry one or more complete Access Units or a single fragment of one AU. Packets MUST NOT contain fragments of multiple Access Units.
Each additional mode definition MUST be in full compliance with this specification and MUST be registered with IANA. A mode MUST NOT specify new default values for MIME parameters, and MIME parameters that configure the RTP payload MUST be present even if redundant for a given mode.
If an Access Unit is fragmented, the RAP-flag MUST be set to 0 for each non-first fragment of the AU. The Stream-state field MUST be incremented by one whenever the stream state changes.
Receivers capable of receiving modes that support interleaving MUST be able to decode interleaved Access Units. When interleaving is applied, senders MUST signal the maximum displacement in time via the MIME parameter 'maxDisplacement'.
The applied mode MUST be signaled via the MIME format parameter 'mode'. Receivers that support only specific modes must be able to determine whether the applied mode is supported before processing payloads.
The CTS-flag field MUST be 0 in the first AU-header (since the first AU's timestamp is carried by the RTP timestamp) and MAY be 1 in non-first AU-headers. The CTS-delta encodes the CTS as a 2's complement offset from the RTP header timestamp using the same clock rate.
The DTS-flag field MUST be present in each AU-header if the DTS-delta field length is signaled as greater than zero, and MUST have the same value for all fragments of an Access Unit. The DTS-delta field MUST also have the same value for all fragments of an AU.
The marker bit (M) is set to 1 to indicate that the RTP packet payload contains either the final fragment of a fragmented Access Unit or one or more complete Access Units. Fragments of the same AU have the same timestamp but different RTP sequence numbers.
The RTP timestamp MUST carry the sampling instant of the first AU (fragment) in the RTP packet. The sender SHALL ensure that RTP timestamps are identical only when referring to fragments of the same Access Unit.
The signaled de-interleave buffer size MUST be large enough to contain all 'early' AUs at any point during the session. If the de-interleaveBufferSize parameter is present, receivers MUST allocate at least that size; otherwise receivers use the calculated size based on maxDisplacement and max bit rate.
When multiple Access Units are carried in an RTP packet, each AU MUST be complete (the number of AUs MUST be integral). Additionally, an AU MUST NOT be repeated in other RTP packets; repetition is only possible via duplicate RTP packets.
When transmitting Access Units of variable duration, the 'constantDuration' parameter MUST NOT be present, the transmitter MUST use AU-Index to encode ordering information, and CTS-delta MUST be coded in the AU-header for each non-first AU in the RTP packet.
The AAC-hbr mode (mode=AAC-hbr) supports variable-size AAC frames up to 8191 octets, with fragmentation and interleaving permitted. AU-headers use 13 bits for AU-size and 3 bits for AU-Index(-delta), giving a 2-octet AU-header; each AU-Index field MUST be coded with value 0.
The AU-Index-delta field encodes the serial number of the associated AU as an unsigned integer difference from the previous AU's serial number: AU-Index(n) = AU-Index(n-1) + AU-Index-delta(n) + 1. A value of 0 indicates consecutive decoding order; a value greater than 0 signals interleaving.
The CELP-cbr mode (mode=CELP-cbr) transports one or more fixed-size CELP frames per RTP packet; interleaving is prohibited, fragmentation is prohibited, and both the AU Header Section and Auxiliary Section MUST be empty. The MIME parameter constantSize MUST be provided.
The RTP payload is divided into three octet-aligned sections: the AU Header Section (containing one or more AU-headers), the Auxiliary Section (containing auxiliary data), and the Access Unit Data Section (containing AU data). The first two sections MAY be empty, but the Access Unit Data Section MUST NOT be empty.
RFC 3640 establishes an IANA registry for mode definitions used with the mpeg4-generic MIME type. New mode registrations require a specification in the form of an RFC, MPEG standard, or other permanent reference (Specification Required policy per RFC 2434).
RFC 3640 registers the MIME subtype 'mpeg4-generic' under the media types 'video', 'audio', or 'application'. Required MIME parameters are streamType, profile-level-id, config, and mode. Optional parameters include objectType, constantSize, constantDuration, maxDisplacement, de-interleaveBufferSize, sizeLength, indexLength, indexDeltaLength, CTSDeltaLength, DTSDeltaLength, randomAccessIndication, streamStateIndication, and auxiliaryDataSizeLength.
Implementations that support MPEG-4 scripts (a subset of ECMAScript) MUST ensure that script actions are limited solely to the domain of the single presentation in which they reside, disallowing session-to-session communication, access to local resources and storage, and unrestricted network access.
Non-compliant MPEG streams can be injected to overload receiver/decoder buffers, potentially compromising or crashing the receiver. This is particularly dangerous for MPEG-4 Systems streams carrying executable content (OD commands, BIFS commands, MPEG-J Java bytecode, ECMAScript). Senders SHOULD ensure MPEG compliance; receivers should implement protective filters.
MPEG-4 system streams define crucial and non-crucial AU states tracked by a stream-state modulo counter. A decoder starts in a corrupted state until a random access point AU is received. On receiving an AU: if RAP-flag=1 and stream-state changes, it is a crucial RAP and MUST be decoded; if RAP-flag=1 and stream-state unchanged, it is a non-crucial RAP and SHOULD be decoded only if the stream is corrupted; if RAP-flag=0, the AU MUST be decoded unless the stream is corrupted.
Each AU-header may contain the following fields in order: AU-size, AU-Index or AU-Index-delta, CTS-flag, CTS-delta, DTS-flag, DTS-delta, RAP-flag, and Stream-state. The length of each field (except the flag fields) is configured via MIME format parameters; a default value of zero indicates the field is absent.
The AU Header Section begins with a 2-octet AU-headers-length field specifying the length in bits of the immediately following AU-headers, excluding padding bits. If the AU-header is configured empty, the AU-headers-length field SHALL NOT be present. Up to 7 zero-padding bits are appended to achieve octet-alignment.
The Auxiliary Section consists of an auxiliary-data-size field (whose bit length is configured by a MIME parameter) followed by the auxiliary-data field. Up to 7 zero-padding bits are appended for octet-alignment. The default length of zero for the size field indicates the entire Auxiliary Section is absent.