Sending Multiple Types of Media in a Single RTP Session
updates
Extracted elements (23)
Applications requiring significantly different network QoS or RTCP configuration for different streams are better served by separate RTP sessions; this specification is only applicable when all media streams can be treated equally at the network layer.
RTP was originally restricted to single media types per session because RTCP timing rules assume all media flows in a session have broadly similar reporting requirements; mixing audio and video can result in either excessive or insufficient RTCP for some flows depending on configuration.
The motivation for multiplexing multiple media types into a single RTP session is to reduce transport-layer flows, which decreases setup delay, middlebox state consumption, and risk of flow establishment failure in NAT/firewall-heavy environments.
The per-SSRC media type restriction exists because each SSRC has its own sequence number and timestamp spaces; allowing media type changes would make packet loss detection impossible and clock rate determination ambiguous for a given SSRC.
Implementing a gateway that interconnects endpoints using separate RTP sessions per media type with endpoints using a single multi-media-type session is generally not feasible, due to SSRC collision risks and the difficulty of remapping SSRCs across all possible RTCP packet types.
In SSRC-multiplexed retransmission mode (RFC 4588), when multiple original RTP streams are sent in one session, original and retransmission stream association must be done via heuristics or via distinct payload types using the 'apt' parameter.
Session-multiplexed retransmission mode (RFC 4588) is unaffected by multiple media types in a single session, since each media type uses a different SSRC which is preserved in the retransmission session. SDP grouping uses BUNDLE plus FID grouping, requiring a separate 'm=' line per retransmitted original source.
Some RTP/RTCP extensions rely on multiple RTP sessions or are specific to a single media type and cannot be used when multiple media types are combined in a single session; applications must avoid such extensions.
The ssrc-group 'FEC' semantic from RFC 5576 cannot be used with generic FEC (RFC 5109/2733) in a multi-media-type session, because generic FEC requires the repair packets to share the same SSRC as the source packets being protected.
This specification is not compatible with unmodified RFC 3550 endpoints; use requires signalling and mutual agreement by all participants. Declarative signalling may attempt to enforce this, but solutions lacking feature negotiation may still encounter compatibility failures.
When using redundant audio encoding (RFC 2198) with generic FEC in a multi-media-type session, each media type requires a distinct payload type for its FEC stream (e.g., audio/ulpfec and text/ulpfec separately), which can rapidly exhaust the 7-bit RTP payload type space.
An SSRC in an RTP session MUST NOT change its media type during its lifetime. It can change between different formats of the same media type (e.g., audio codecs), but cannot switch from audio to video.
Different SSRCs MUST be used for different media sources when multiple media types are configured in a single RTP session, mirroring the requirement for multiple sources of the same media type.
For generic FEC sent as a separate stream, it is RECOMMENDED to use the 'application/ulpfec' or 'application/parityfec' media type rather than media-specific payload format names, to avoid unnecessary payload type consumption.
Payload types of different media types SHALL NOT be multiplexed within a single RTP session unless RFC 8860 is used and the application conforms to its applicability constraints (updating RFC 3551).
Separate audio and video media sources SHOULD NOT be carried in a single RTP session unless the guidelines of RFC 8860 are followed (updating RFC 3550's prohibition).
The RTP payload type MUST be unique across all payload configurations independent of media type within an RTP session carrying multiple media types.
When sending multiple types of media with different rates in a single RTP session, endpoints MUST follow the RTCP guidelines in Section 7 of RFC 8108.
RTP demultiplexing within a multi-media-type session proceeds by SSRC first (identifying the RTP stream), then payload type is used solely to select the appropriate media decoder—payload type is never used to distinguish RTP streams.
Signalling for multi-media-type RTP sessions must ensure: (1) all participants are aware of the multi-media-type nature, (2) payload type values are globally unique across all media types in the session, (3) session-level RTCP and transport parameters are consistent, and (4) RTP/RTCP functions are reused rather than duplicated.
When using SDP signalling to establish a single RTP session with multiple media types, the BUNDLE extension (RFC 8843) is the designated mechanism.
RTP provides strong security mechanisms applicable to multi-media-type sessions (RFC 7201, RFC 7202), but implementors must verify that the chosen security mechanism is compatible with all media types present in the session.
Sending multiple media types in a single RTP session generally requires all media to use the same security mechanism. When different media types have different security requirements, separate RTP sessions may be necessary, at the cost of increased resource usage and setup time.