Remote Direct Memory Access Transport for Remote Procedure Call
obsoleted by
- rfc-8166 — Remote Direct Memory Access Transport for Remote Procedure Call Version 1
Extracted elements (30)
Fixed credit limits at the RPC server are rejected because posted receive buffers are dedicated per-connection until consumed, scaling poorly as connections grow. A dynamic request/grant protocol carried in each RPC-over-RDMA header allows limits to be adjusted to server policy at each opportunity.
RDMA Write (reply chunks via rdma_reply) is preferred over server-side read chunks for long replies because RDMA Write avoids an extra network round-trip, does not require the server to wait for read completion, and eliminates the RDMA_DONE message and its associated timer and resource management.
The offset field is unconditionally present in every xdr_rdma_segment, even though zero-based offset schemes exist. This enables pre-registration and region-based registrations across many RDMA implementations, and produces the most general and simplified protocol form.
The XID is placed at the beginning of the RPC-over-RDMA header (before version, credits, and body) so that receivers can quickly establish message context. The receiver MAY process the message based solely on the header XID and ignore the XID in the inner RPC header.
RPC-over-RDMA framing replaces all other RPC framing (e.g., TCP record marking) when operating over an RPC/RDMA association, even when the underlying RDMA protocol is itself layered on TCP. The framing transition MUST occur between distinct RPC messages and in concert with the transport.
A peer that detects an unsupported RPC-over-RDMA header version MUST reply with ERR_VERS, providing the low and high inclusive version numbers it supports. When other chunk or header decoding errors are detected, ERR_CHUNK MUST be returned.
Any byte count in the XDR stream MUST match the sum of the byte counts present in the corresponding read or write chunk list. If they do not agree, an RPC protocol encoding error results.
Chunk eligibility criteria MUST be determined by each upper-layer protocol to provide an interoperable specification; the RPC-over-RDMA protocol itself does not specify which XDR data items may be chunked.
Registered memory for RDMA operations MUST use a steering tag of no more than 32 bits and memory addresses of up to 64 bits in length.
The Connection Configuration Protocol, when used, MUST be protected by an appropriate RPC security flavor to ensure it is not attacked during RPC/RDMA connection initiation.
The granted credit value in RPC reply messages MUST NOT be zero when no in-progress operations are present at the server, as a zero value would result in client deadlock. The server MUST always provide at least one credit to each connected client from which no requests are outstanding.
The RPC client MUST account for its required RDMA_DONE messages to the server in its accounting of available credits. The server SHOULD provide sufficient credits to allow RDMA_DONE messages to be sent without deadlock.
The RPC client MUST NOT send unacknowledged requests in excess of the granted RPC server credit limit. If the limit is exceeded, the RDMA layer may signal an error and possibly terminate the connection.
The use of RDMA by RPC MUST NOT introduce any vulnerabilities to system memory contents or to memory owned by user processes. Any RDMA provider used for RPC transport is REQUIRED to be conformant to RFC 5042.
The XID in the RPC-over-RDMA header MUST be the same as the XID in the RPC header. A header of type RDMA_MSG or RDMA_MSGP MUST be followed by the RPC call or reply body beginning with the XID.
When a write chunk list is provided for RPC call results, the RPC server MUST provide any corresponding data via RDMA Write to the memory referenced in the chunk list entries. The write chunk list mapping onto procedure results MUST be determined for each upper-layer protocol.
When processing an RPC reply that contained read chunks, the RPC client MUST issue an RDMA_DONE message to notify the server (peer) that the source buffers can be freed.
rpc_rdma_error defines two error codes: ERR_VERS=1 (version mismatch, carrying rdma_vers_low and rdma_vers_high indicating the supported range) and ERR_CHUNK=2 (invalid chunk encoding, no additional data).
The Connection Configuration Protocol (RPC program 100417, VERS1) provides a single CONF_RDMA procedure. The client sends maxcall_sendsize, maxreply_sendsize, and maxrdmaread; the server returns its accepted maxcall_sendsize, alignment recommendation, and maxrdmaread. Use is OPTIONAL.
The rdma_proc enum defines five message types: RDMA_MSG=0 (normal RPC with inline body), RDMA_NOMSG=1 (body transferred separately via RDMA Read), RDMA_MSGP=2 (padded RPC message), RDMA_DONE=3 (client signals reply chunk read completion), RDMA_ERROR=4 (encoding error signaling).
The RPC-over-RDMA header contains four fixed 32-bit fields always present: XID (mirroring the RPC header), version (currently 1), credit value (requested in calls, granted in replies), and message type (RDMA_MSG, RDMA_NOMSG, RDMA_MSGP, RDMA_DONE, or RDMA_ERROR). These fields MUST remain at fixed offsets for all versions.
xdr_rdma_segment encodes a registered memory region as three fields: uint32 handle (steering tag), uint32 length (chunk bytes), uint64 offset (virtual address or base offset). The offset field is unconditionally included to support pre-registration and region-based registrations.
xdr_read_chunk adds a uint32 position field (XDR stream offset) to an xdr_rdma_segment, indicating where the chunk data logically belongs in the XDR stream. A position of zero indicates the entire RPC message body is supplied as a chunk (the long-message RDMA_NOMSG case).
NFS/RDMA service port 20049 is assigned for TCP, UDP, and SCTP (service name 'nfsrdma'). RPC program number 100417 is assigned to the rdmaconfig Connection Configuration Protocol ('rpc.rdmaconfig').
Two RPC netid strings are registered: 'rdma' for IPv4 RDMA transports and 'rdma6' for IPv6 RDMA transports. These may be used for any RDMA network satisfying the abstract requirements of Section 2 and able to identify endpoints using IP port addressing.
A client that withholds RDMA_DONE acknowledgments can exhaust server RDMA buffer resources (denial of service). Servers MUST protect against this via time-based expiry windows or connection closure. The rdma_credit mechanism MAY be used to limit per-client outstanding requests, but credit MUST NOT be reduced to zero solely due to outstanding RDMA_DONE messages.
Where efficiency is required alongside integrity and privacy, link-level IPsec (potentially co-located in RDMA hardware) is preferred over RPCSEC_GSS. Use of RPCSEC_GSS Version 2 (RFC 5403) with Channel Binding (RFC 5056) and IPsec Channel Connection Latching (RFC 5660) is REQUIRED where both integrity/privacy and efficiency are needed.
For RPC replies containing server-advertised read chunks (RDMA_NOMSG reply): server sends header, client issues RDMA Read(s) to fetch chunks, client MUST send RDMA_DONE to release server buffers. For write chunks in calls, RDMA ordering guarantees data placement before the reply Send arrives; no RDMA_DONE is required.
rpc_rdma_header_padded inserts two additional uint32 fields—rdma_align (padding alignment value) and rdma_thresh (opaque data size threshold)—before the read/write chunk lists. When an opaque value's length exceeds rdma_thresh, the XDR decoder MUST skip rdma_align padding bytes before the data.
xdr_rdma_segment is encoded as: uint32 handle (4 bytes), uint32 length (4 bytes), uint64 offset (8 bytes), totaling 16 bytes. xdr_read_chunk prepends a uint32 position field (4 bytes) for a total of 20 bytes per read chunk entry.