WebRTC Video Processing and Codec Requirements
Extracted elements (31)
Both VP8 and H.264 are mandated for WebRTC because the licensing landscape for video codecs was uncertain at the time of writing; the RTCWEB working group intended to revisit codec requirements for Non-Browsers if one became unambiguously royalty-free, while browser requirements were considered settled.
CVO orientation signaling via RTP header extension is preferred over codec-specific in-band mechanisms because pre-encoding rotation may not be practical when the camera interface returns pre-compressed frames, and swapping mid-stream adds resolution-change complexity already present for screen-source video.
The minimum required decoding capability of 20 fps at 320x240 is selected based on ITU-T H.Sup1 recommendations for sign language and lip-reading real-time video communication.
H.264 optional parameters max-mbps, max-smbps, max-fs, max-cpb, max-dpb, and max-br allow signaling capabilities beyond the profile-level-id level; implementations MAY include them but SHOULD interpret them when received to maximize sent video quality.
The default Y'CbCr 4:2:0 video scan format is known to be less than optimal for screen-sourced content, which generally uses RGB with at least 24 bits per sample; future codecs optimized for screen content may be preferable.
When CVO is not signaled in the SDP, H.264 implementations MAY send and SHOULD support proper interpretation of Display Orientation SEI messages as a fallback orientation mechanism.
All implementations SHOULD support interpretation of the R0 and R1 CVO bits; they MAY support the other CVO bits.
For the H.264 codec, endpoints MUST support the payload formats defined in RFC 6184 and MUST support Constrained Baseline Profile Level 1.2; they SHOULD support H.264 Constrained High Profile Level 1.3.
For the VP8 codec, endpoints MUST support the payload formats defined in RFC 7741.
H.264 implementations MUST be prepared to receive 'User data registered by Rec. ITU-T T.35' and 'User data unregistered' SEI messages without any ill effects, even if they do not act on them.
H.264 implementations MUST include the profile-level-id parameter within SDP and MUST interpret it when receiving it.
H.264 implementations MUST support proper interpretation of SEI 'filler payload' and 'full frame freeze' messages; they MAY also send these messages.
H.264 packetization-mode 1 MUST be supported; other modes MAY be negotiated and used.
If CVO has been negotiated, the sender MUST NOT make use of codec-specific in-band orientation mechanisms such as H.264 SEI Display Orientation messages.
Senders SHOULD interpret and honor the a=imageattr SDP attribute by limiting encoded resolution to the indicated maximum size, as the receiver may not be capable of handling higher resolutions.
Unless otherwise signaled, implementations that use H.264 MUST encode and decode pixels with an implied 1:1 (square) aspect ratio.
Unless otherwise signaled, implementations that use VP8 MUST encode and decode pixels with an implied 1:1 (square) aspect ratio.
Unless otherwise signaled in SDP, recipients of video streams MUST be able to decode video at a rate of at least 20 fps at a resolution of at least 320 pixels by 240 pixels.
Unless specified otherwise by SDP or codec, the color space SHOULD be sRGB (IEC 61966-2-1), indicated by codepoint 1 from ColourPrimaries as defined in ISO/IEC 23001-8.
VP8 encoders MUST limit the streams they send to conform to the values indicated by receivers in the corresponding max-fr and max-fs SDP attributes.
WebRTC Browsers MUST implement the VP8 video codec as described in RFC 6386 and H.264 Constrained Baseline as described in H.264.
WebRTC implementations MUST signal H.264 sequence and picture parameter information in-band and therefore MUST NOT include the sprop-parameter-sets parameter in the SDP they generate.
WebRTC implementations that can generate media in orientations other than the default MUST support generating the R0 and R1 bits of the Coordination of Video Orientation (CVO) mechanism and MUST send them for all orientations when the peer indicates support.
WebRTC Non-Browsers that support transmitting and/or receiving video MUST implement the VP8 video codec as described in RFC 6386 and H.264 Constrained Baseline as described in H.264.
WebRTC-video recipients MUST be prepared to handle midstream resolution changes in a way that preserves their utility, given that screen-sourced video can change resolution due to window resizing and similar operations.
The a=imageattr SDP attribute (RFC 6236) provides codec-independent indication of preferred maximum video resolution; WebRTC endpoints MAY send it and senders SHOULD honor it.
The Coordination of Video Orientation (CVO) RTP header extension mechanism, defined in 3GPP TS 26.114 Section 7.4.5, is used to signal camera orientation changes mid-call via R0 and R1 rotation bits.
The default video scan pattern for WebRTC video codecs is Y'CbCr 4:2:0, unless specified otherwise by SDP or codec.
Implementors should consider whether variable bitrate video codecs are appropriate for their application, as an eavesdropper may deduce the degree of inter-frame change (and infer the amount of motion in the frame) from the video stream's bitrate.
Implementors using H.264 are advised to carefully review the Security Considerations of RFC 6184, with particular attention to the normative requirement pertaining to SEI messages, which can carry arbitrary data and must not cause ill effects.
In WebRTC, video is protected using DTLS/SRTP; a complete discussion of security considerations can be found in the WebRTC Security and WebRTC Security Architecture documents.