Enhanced Remote Direct Memory Access (RDMA) Connection Establishment
updates
Extracted elements (25)
ORD and IRD negotiation was left entirely to the ULP by the existing RDDP suite, requiring every application to re-implement encoding of these parameters in Private Data. Standardizing this exchange enables common connection establishment code to configure RDMA endpoints uniformly across InfiniBand and RDDP transports without per-ULP duplication.
The current MPA protocol only supports a client-server connection model, which forces peer-to-peer applications (e.g., MPI cluster nodes) to artificially designate one side as 'client.' This document adds peer-to-peer support because in cluster workloads neither node predictably sends first, and the workaround of injecting a dummy first message is incompatible with transport-neutral APIs, disrupts Work/Completion Queue accounting, and conflicts with host-based MPA Fencing implementations.
The RTR indication type is made flexible (zero-length FULPDU Send, RDMA Write, or RDMA Read) because RDMA implementations are tightly coupled to hardware and application libraries. Allowing multiple RTR message types lets implementations choose the type least disruptive to their structure; the initiator selects among mutually supported types at its sole discretion.
An enhanced initiator sending an enhanced MPA frame to an unenhanced responder will receive a TCP connection close, because the unenhanced responder treats the frame as improperly formatted per RFC 5044. The initiator MAY then retry using unenhanced MPA if the application permits it.
If an enhanced responder receives an unenhanced MPA message ('S' bit is 0), it MUST respond with an unenhanced MPA message, maintaining full backward compatibility with RFC 5044 initiators.
An all-ones IRD or ORD value (0x3FFF) signals that automatic negotiation is not desired and is left to the ULP. The responder MUST respond to an initiator ORD or IRD value of 0x3FFF by leaving the corresponding local endpoint value unchanged and echoing 0x3FFF in the reply.
If the initiator cannot generate any of the responder-supported RTR indications, it MUST send a TERM message (Layer 2, Error Type 0, Error Code 7) and terminate the connection. For any other local catastrophic error lacking a specific code, Error Code 5 MUST be used.
If the 'S' flag is set in the MPA Request/Reply Frame, Private Data MUST begin with enhanced RDMA connection establishment data as defined in Section 9.
Private Data length MUST NOT exceed 512 bytes in any Enhanced SCTP DDP stream session message, including the enhanced RDMA connection establishment data prefix. Private Data MUST NOT be included in the DDP Stream Session TERM message.
The Enhanced DDP Stream Session Reject function code (0x007) MUST be used to indicate rejection of an enhanced DDP stream session that would have been accepted under unenhanced negotiation. The enhanced DDP stream management MUST use the DDP stream session termination function code to terminate streams established using enhanced function codes.
The initiator MUST NOT use enhanced RDMA connection establishment formats or function codes when no enhanced functionality is desired. The responder MUST continue to accept unenhanced connection requests.
The initiator MUST set Control Flag A to 1 for the peer-to-peer model and MUST set each Control Flag B, C, and D to 1 for each RTR indication type it supports. For the client-server model, Control Flag A MUST be 0, and B, C, D MUST also be 0.
The RDMA connection initiator MUST NOT allow any later FULPDUs to be transmitted before the RTR indication. One method is to delay notifying the ULP that the connection has been established until after any required RTR indication has been transmitted.
The responder MUST set its ORD at most to the initiator IRD. The responder MAY reject the connection if the initiator IRD is insufficient for the ULP-required ORD; in that case the TERM message MUST contain Layer 2, Error Type 0, Error Code 6.
The responder MUST support at least one RTR indication option if it supports Enhanced RDMA connection establishment. If Control Flag A is 1 in the MPA Request, the responder MUST set Control Flag A to 1 in the MPA Reply and MUST set at least one RTR indication option flag.
To use any enhanced connection establishment feature, the Rev field in the MPA Request/Reply Frame MUST be set to two or higher. A host accepting MPA connections MUST continue to accept MPA Requests with version one, even if it supports version two.
Upon receiving the MPA Accept Frame, the initiator MUST set its IRD at least to the responder ORD and its ORD at most to the responder IRD. If the initiator lacks sufficient resources for the required IRD, it MUST send a TERM message (Layer 2, Error Type 0, Error Code 6) and terminate the connection.
Three new MPA error codes are defined under Layer 0x2, Error Type 0x0: Error Code 0x05 (local catastrophic), Error Code 0x06 (insufficient IRD resources), and Error Code 0x07 (no matching RTR option). These extend the four codes already defined by RFC 5044.
Three new SCTP Function Codes for DDP Session Control are defined: Enhanced DDP Stream Session Initiate (0x005), Enhanced DDP Stream Session Accept (0x006), and Enhanced DDP Stream Session Reject (0x007). These parallel the unenhanced codes from RFC 5043 and carry IRD/ORD fields in the first 32 bits of Private Data.
IANA added three entries to the 'MPA Errors' registry (created by RFC 6580): Layer/Type/Code 0x2/0x0/0x05 (local catastrophic error), 0x2/0x0/0x06 (insufficient IRD resources), and 0x2/0x0/0x07 (no matching RTR option).
IANA added three entries to the 'SCTP Function Codes for DDP Session Control' registry (created by RFC 6580): 0x0005 Enhanced DDP Stream Session Initiate, 0x0006 Enhanced DDP Stream Session Accept, and 0x0007 Enhanced DDP Stream Session Reject.
The security considerations from RFC 5044 and RFC 5043 apply; the changes in this document do not introduce new security considerations. However, implementations are recommended to perform sanity checking on input parameters including ORD, IRD, and the control flags used for RTR indication option negotiation.
Enhanced connection negotiation proceeds in three steps: (1) Initiator sends MPA Request with Control Flag A, IRD/ORD, and RTR option flags B/C/D; (2) Responder sends MPA Reply echoing Flag A and setting supported RTR flags and actual IRD/ORD values; (3) Initiator sends first RDMA message as RTR indication, or sends TERM (Error Code 7) if no RTR option is mutually supported.
Enhanced MPA Request/Reply Frame uses a 16-byte Key field (ASCII 'MPA ID Req Frame' or 'MPA ID Rep Frame'), followed by a flags byte (M, C, R, S bits), a Rev byte, and a 16-bit PD_Length field. The new 'S' bit in the previously reserved 'Res' field signals that Private Data begins with enhanced RDMA connection establishment data.
Enhanced RDMA Connection Establishment Data is a 32-bit field placed at the start of Private Data, encoded in network byte order. Bits 0–1 carry control flags A and B (IRD high word), bits 2–15 carry a 14-bit IRD; bits 16–17 carry control flags C and D (ORD high word), bits 18–31 carry a 14-bit ORD.