Remote Direct Memory Access Transport for Remote Procedure Call Version 1

An Offset field is included in every RDMA segment to support pre-registration and region-based memory registration strategies. Without this field, implementations would be forced to register individual memory regions on each RPC, which is expensive on RDMA transports that lack zero-based offset schemes.

RDMA_MSGP was removed because its specification in RFC 5666 was incomplete—missing full descriptions of alignment, threshold fields, peer negotiation, and interaction with Read/Write chunks—and no complete interoperable prototype was ever built. Its benefit is limited to payloads at the very end of an argument or result list, which is atypical for NFSv4 COMPOUND RPCs.

The protocol version number was not incremented when this document obsoleted RFC 5666 because RPC-over-RDMA version 1 as defined here fully interoperates with RFC 5666 implementations; only ambiguous text was clarified and unimplemented elements were removed.

RPC-over-RDMA framing replaces all other RPC framing (e.g., TCP record marking) for an RPC-over-RDMA association, even when the underlying RDMA protocol is itself layered over TCP. A shift from non-RDMA to RDMA framing must occur between distinct RPC messages and in concert with the underlying transport.

tcp

A Requester MUST NOT send unacknowledged requests in excess of the Responder's currently granted credit limit. Exceeding the granted value may cause the RDMA layer to signal an error, possibly terminating the connection.

A Requester MUST provide a Reply chunk whenever the maximum possible size of the RPC Reply's Transport and Payload streams exceeds the reply inline threshold. The Responder MUST use the Reply chunk if the streams exceed that threshold.

For RPC-over-RDMA version 1, the rdma_vers field MUST contain the value one (1). When a Responder detects an unsupported version, it MUST reply with an RDMA_ERROR using ERR_VERS and include the low and high supported version numbers.

If a Requester cannot invalidate a memory region due to an unrecoverable error after receiving a reply, it MUST close the connection to prevent Responder access to that memory. Similarly, a Responder unable to send an RPC Reply or error message MUST close the connection to signal reply loss.

security

If an incoming message is shorter than the 28-byte minimal RPC-over-RDMA header, the receiver MUST silently discard it without parsing. If other header errors are detected in an RPC Call, the Responder MUST send an RDMA_ERROR reply; errors in an RPC Reply MUST be silently discarded by the Requester.

On a newly established connection, a Requester MUST NOT assume more than one credit is available and MUST receive a valid reply carrying the actual granted credit count before sending its next request. Each implementation MUST support inline thresholds of at least 1024 bytes.

Senders MUST NOT send RDMA_DONE messages; receivers MUST silently discard them. RDMA_DONE was associated with the Read-Read transfer model, which was never implemented.

Senders MUST NOT send RDMA_MSGP procedures. Responders receiving RDMA_MSGP SHOULD reply with RDMA_ERROR (ERR_CHUNK); Requesters receiving RDMA_MSGP MUST silently discard the message. RDMA_MSGP was never fully specified or prototyped.

The rdma_xid field in the RPC-over-RDMA header MUST contain the same XID as the encapsulated RPC message. A mismatch is an XDR error requiring an ERR_CHUNK RDMA_ERROR response from the Responder.

When a Requester provides a Write list, the Responder MUST push data corresponding to DDP-eligible result items to Requester memory and MUST copy and update all segment information into the RPC Reply message's Transport header reflecting actual bytes written.

When reducing a counted array XDR data item into a chunk, the array element count MUST remain in the Payload stream while the array elements MUST be moved to the chunk. XDR union data types MUST NOT be made DDP-eligible, though individual union arms MAY be.

Credit-based flow control is implemented via the rdma_credit field present in every RPC-over-RDMA message. Requesters send a requested credit value; Responders reply with a granted credit value indicating how many concurrent outstanding requests they can accept. The granted value must never be zero.

Every RPC-over-RDMA message consists of two XDR streams: a Transport stream containing the fixed header fields and chunk lists, and a Payload stream containing the encapsulated RPC message beginning with its XID field. In its simplest form both streams are conveyed together in a single RDMA Send operation.

Long Messages handle RPC messages larger than the inline threshold. A Long Call uses a Position Zero Read chunk (all segments have Position=0) carrying the entire Payload stream; a Long Reply uses a single Reply chunk (a special Write chunk) pre-provisioned by the Requester to hold the entire reply Payload stream.

Read chunks represent XDR data items that the Responder pulls from the Requester using RDMA Read operations. They are encoded as a linked list (Read list) of xdr_read_chunk entries grouped by Position value; the Responder reconstructs the reduced Payload stream by concatenating each segment's data at the indicated Position offset.

RPC-over-RDMA version 1 defines five procedure types via the rdma_proc enum: RDMA_MSG (0) carries a Payload stream inline; RDMA_NOMSG (1) carries no inline Payload stream; RDMA_MSGP (2) and RDMA_DONE (3) are reserved and must not be used; RDMA_ERROR (4) signals a header encoding error.

The inline threshold is the largest message size (in octets) transferable in one direction via RDMA Send/Receive. Each connection has two independent values: call inline threshold (Requester-to-Responder) and reply inline threshold (Responder-to-Requester). These values are not advertised in the protocol and cannot change during the connection lifetime.

Write chunks represent XDR result data items that the Responder pushes to the Requester using RDMA Write operations. Each Write chunk is a counted array of plain segments provisioned by the Requester in the RPC Call message; the Responder fills them in segment-array order and returns updated segment lengths in the RPC Reply message.

RFC 8166 defines two new IANA netid values for RPC-over-RDMA: one for RPC services operating atop iWARP (layered on TCP or SCTP) and one for RPC services atop InfiniBand (when a suitable port translation service is available). Additional RDMA networks may define their own netids or share these where port translation is applicable.

registry

RDMA transports expose endpoint memory to remote peers via steering tags (STags). A Requester must ensure that memory regions associated with each RPC transaction are protected from Responder access before allowing upper-layer access, and must not access those regions while the Responder retains access rights—including for canceled RPCs.

security

RPCSEC_GSS integrity protection does not extend to data transferred via RDMA Read and Write operations because such data bypasses the normal message path. Implementations requiring end-to-end integrity over RDMA chunks must employ additional protection mechanisms at the ULP level.

security

A plain RDMA segment (xdr_rdma_segment) encodes three fields: a 32-bit handle (steering tag / R_key), a 32-bit length (size of the memory region in bytes), and a 64-bit offset (virtual address or base offset of the region). An empty segment has a zero length field.

An RDMA read segment (xdr_read_chunk) prepends a 32-bit Position field (byte offset in the unreduced Payload stream where the chunk is reinserted) to a plain xdr_rdma_segment. All RDMA read segments belonging to the same Read chunk share the same Position value.

The RPC-over-RDMA version 1 fixed header (rdma_msg) contains four 32-bit fields in fixed order: rdma_xid (transaction ID), rdma_vers (protocol version), rdma_credit (flow-control credit), and the rdma_body union discriminator (rdma_proc). These four fields must remain at the same positions in all subsequent protocol versions.