Marker PDU Aligned Framing for TCP Specification
updated by
Extracted elements (29)
MPA uses a deterministic framing approach (Markers at fixed, known TCP sequence-number offsets) rather than a probabilistic approach (detectable values anywhere in the stream) because only a deterministic scheme can unambiguously locate ULPDUs in out-of-order TCP packets without false positives from application data.
The active/active (Initiator/Initiator) connection mode was explicitly rejected by the working group because applications requiring it were theoretical, it complicated dual-stack implementations, and handling rejected simultaneous connections would have required additional frame types and recovery logic with no practical motivation.
The maximum ULPDU size of 64768 octets (rather than the theoretical 65535 from the 16-bit ULPDU_Length field) was derived by subtracting the worst-case sum of IPv4 header, TCP header, options, and MPA overhead from the 65535-octet IP datagram limit, then rounding down to a 128-octet boundary to enable FPDU Alignment.
The requirement that the Responder receive at least one FPDU before sending any FPDUs exists to give the Initiator time to transition its receiver into Full Operation, avoiding potential race conditions. The working group debated removing this requirement for faster startup but retained it due to dual-stack implementation concerns.
Appendix C describes interoperability between IETF RNIC implementations (permissive and non-permissive) and RDMA Consortium (RDMAC) RNICs. The key parameter differences involve Marker and CRC negotiation: RDMAC always requires Markers; a non-permissive IETF RNIC that cannot interoperate with an RDMAC RNIC MUST close the connection.
An MPA receiver MUST locate the start of each FPDU unambiguously and verify its CRC (if enabled) before passing the ULPDU to DDP. The ULPDU_Length field MUST be used to determine whether the entire FPDU is present before forwarding.
CRCs MUST be enabled unless it is clear that the end-to-end connection has data integrity at least as good as MPA with CRC enabled (for example, when IPsec is implemented end to end). An administrative decision to request CRC suppression SHOULD NOT be made without such assurance.
MPA implementations MUST use the difference between the MPA Request Frame and MPA Reply Frame key values to detect incorrect Initiator/Initiator startups. Implementations SHOULD put a timeout on waiting for the MPA Request Frame when in Responder mode to detect Responder/Responder startups.
MPA implementations MUST validate the PD_Length field; the Private Data MUST NOT exceed 512 octets, and PD_Length MUST match the actual length of the Private Data field. Any failure MUST cause the startup frame to be treated as improperly formatted and the TCP connection to be closed.
MPA MUST detect the following errors and SHOULD provide error codes to DDP: (1) TCP connection closed/terminated/lost, (2) CRC mismatch, (3) Marker and ULPDU_Length disagreement on FPDU start (when CRC is valid), (4) invalid MPA Request or Reply Frame (which MUST cause immediate TCP connection close).
MPA MUST encapsulate each ULPDU such that there is exactly one ULPDU contained in one FPDU. The ULPDU_Length field MUST accurately reflect the length of the contained ULPDU, excluding the header, PAD, CRC, and any embedded Markers.
MPA receivers MUST preserve ULPDU boundaries when passing data to DDP. Receivers MUST pass the ULPDU data and ULPDU_Length to DDP, and MUST NOT pass Markers, FPDU headers, or the CRC field to DDP.
MPA Responder mode implementations MUST receive and validate at least one FPDU before sending any FPDUs or Markers, ensuring the Initiator has time to enter Full Operation before an FPDU arrives and avoiding race conditions.
The Key field of the MPA Request Frame MUST be validated by the Responder; if the value does not match 'MPA ID Req Frame', the connection MUST be closed and an error reported locally. Likewise, the Initiator MUST validate the Key in the MPA Reply Frame against 'MPA ID Rep Frame'.
The maximum ULPDU length when Markers are present MUST be computed as MULPDU = EMSS − (6 + 4 × Ceiling(EMSS / 512) + EMSS mod 4). When Markers are absent, MULPDU = EMSS − (6 + EMSS mod 4). If EMSS shrinks below 128 octets, MPA MUST NOT shrink MULPDU below 128 octets.
The range of MULPDU values MUST be between 128 octets and 64768 octets, inclusive. The sending DDP MUST NOT post a ULPDU larger than 64768 octets to MPA.
The Rev field MUST be set to 1 by senders. If the receiver cannot interoperate with the received version, it MUST close the connection and report an error locally. MPA MUST provide the negotiated protocol version to DDP.
When CRCs are enabled, the CRC field in the FPDU MUST be computed using the CRC32c polynomial as described in the iSCSI protocol for Header and Data Digests. On receive, the receiver MUST calculate the CRC and compare it against the received value; a mismatch MUST cause the FPDU to be treated as invalid.
When Markers are enabled, MPA senders MUST insert a Marker at a 512-octet periodic interval in the TCP Sequence Number Space, beginning immediately before the first FPDU of Full Operation. All MPA senders MUST be capable of generating Markers when required by the opposing receiver.
When multiple Markers fall within a single large FPDU, they MUST all point to the same location: the first octet of the ULPDU Length field for that FPDU. When a Marker falls exactly between FPDUs, FPDUPTR MUST be 0x0000 and the Marker MUST be included in the CRC of the following FPDU.
FPDU Alignment is the property that a TCP segment begins with an FPDU (Header Alignment) and contains an integer number of complete FPDUs. Aligned delivery allows DDP to place data directly into application buffers without intermediate buffering, which is the primary performance motivation for MPA.
Private Data is a 0–512 octet field carried in MPA Request and Reply Frames, conveying ULP-specific connection parameters (such as Protection Domain identity, RDMA Read capacity, or client authentication data) before a DDP endpoint is bound. Private Data enables immediate-startup RDMA connections without prior TCP streaming-mode negotiation.
CRC suppression is only permissible when end-to-end data integrity equivalent to CRC32c is guaranteed by another mechanism. End-to-end IPsec with cryptographic integrity check is explicitly cited as a qualifying mechanism; use of channel bindings by the ULP can provide assurance that IPsec protection scope is truly end-to-end.
MPA implementations SHOULD implement a reasonable timeout while waiting for startup frames to prevent certain denial-of-service attacks. ULPs SHOULD also implement timeouts while waiting for FPDUs and application-level messages.
MPA over TCP is vulnerable to spoofing attacks including impersonation, stream hijacking, and man-in-the-middle attacks because TCP provides no inherent authentication. IPsec SHOULD be used to protect MPA connections from these threats; the MPA key exchange in the Startup Phase does not authenticate endpoints.
MPA operates in two distinct phases: the Startup Phase (exchange of MPA Request and Reply Frames as plain TCP byte streams without CRCs or Markers) and the Full Operation Phase (exchange of FPDUs with CRC, Markers, and MULPDU rules). Transition from Startup to Full Operation is triggered by the Initiator receiving a valid, accepted MPA Reply Frame; the Responder transitions after sending the Reply Frame and receiving the first FPDU.
The FPDU (Framed Protocol Data Unit) format consists of a 16-bit ULPDU_Length field, the ULPDU payload, 0–3 octets of PAD (set to zero, ignored on receive except for CRC), and a 32-bit CRC field. The total FPDU length is always an integral multiple of four octets.
The Marker format is a 4-octet field consisting of a 16-bit RESERVED field (must be zero on transmit) and a 16-bit FPDUPTR (FPDU Pointer) unsigned integer. FPDUPTR indicates the number of octets in the TCP stream from the start of the preceding ULPDU Length field to the first octet of the Marker; its two least-significant bits MUST always be treated as zero.
The MPA Request/Reply Frame contains a 16-byte Key (ASCII 'MPA ID Req Frame' or 'MPA ID Rep Frame'), a flags byte with M (Markers required), C (CRC preferred), and R (Rejected Connection, Reply only) bits, an 8-bit Rev field, a 16-bit PD_Length, and an optional Private Data field of 0–512 octets.