Remote Direct Memory Access - Connection Manager (RDMA-CM) Private Data for RPC-over-RDMA Version 1
updates
- rfc-8166 — Remote Direct Memory Access Transport for Remote Procedure Call Version 1
Extracted elements (25)
An out-of-band CM Private Data mechanism was chosen because RPC-over-RDMA version 1 has no way to extend its XDR-based Transport Header format while preserving interoperability with existing implementations. The Private Data exchange is independent of and does not alter the XDR definition in RFC 8166.
Remote invalidation is signaled via a simple R bit rather than a per-STag mechanism because some NFS/RDMA client implementations always expose STags that are safe to invalidate remotely. A blanket flag is sufficient for these clients without requiring new per-chunk XDR extensions.
The inline threshold negotiation was motivated by NFS version 4 COMPOUND operations being larger on average than NFS version 3 procedures, and by RPCSEC_GSS increasing RPC header sizes, both of which force costly explicit RDMA operations under the 1024-byte default threshold.
The Size fields encode inline threshold values in units of 1 kilo-octet above the 1024-octet minimum, fitting the full range 1024–262144 octets into 8 bits. This avoids the need for larger fields while covering all practically relevant buffer sizes.
The Version field enables future extensions using the same Format Identifier but a different version number, while the Format Identifier registry enables entirely new Private Data formats that coexist without conflict. Both paths preserve backward interoperability.
After locating the Format Identifier, the receiver should additionally verify that the Version field contains a recognized value and that the declared content length does not overrun the buffer, as additional assurance against misidentification.
If the CM Private Data field capacity is too small to hold the 8-octet message, or the underlying RDMA transport is not managed by a Connection Manager, the CM Private Data mechanism cannot be used for RPC-over-RDMA version 1 property exchange.
Implementation of the CM Private Data exchange is OPTIONAL. Implementations not recognizing this format are fully interoperable because the absence of conforming Private Data causes the receiving peer to fall back to RFC 8166 defaults.
When searching the received Private Data buffer for the Format Identifier, the offset is not restricted to any alignment. This allows coexistence with other protocols (e.g., MPAv2) that also insert data into the CM Private Data field.
An implementation of this extension MUST be prepared for transport properties (inline thresholds, remote invalidation support) to change upon a reconnection, since each new connection presents a fresh CM Private Data exchange.
If a peer does not receive a conforming CM Private Data message, it MUST behave as if it received a message with the R bit cleared and both Size fields set to zero (i.e., default 1024-byte inline thresholds, no remote invalidation).
If the RPC-over-RDMA version 1 CM Private Data Format Identifier is not found in the received Private Data buffer, an RPC-over-RDMA version 1 receiver MUST behave as if no RPC-over-RDMA version 1 CM Private Data has been provided.
Senders MUST set the 7-bit Reserved field to zero; receivers MUST ignore its value.
When both peers have set the R flag, each RDMA Send with Invalidate operation MUST invalidate only an STag associated with the Transaction ID (XID) in the rdma_xid field of the RPC-over-RDMA Transport Header it carries.
When either peer on a connection clears the R flag, the responder MUST use only RDMA Send (not RDMA Send with Invalidate) when transmitting RPC Replies.
The Format Identifier field is a fixed 32-bit value (0xf6ab0e18 in network byte order) that identifies the Private Data content as belonging to RPC-over-RDMA version 1. It is searched at any byte alignment within the received Private Data buffer.
The R (remote invalidation) bit is a 1-bit flag indicating that the sender supports remote invalidation for all STags it exposes on the connection. When set by both peers, the responder may use RDMA Send with Invalidate when transmitting RPC replies.
The Send Size and Receive Size fields are each 8 bits and encode inline threshold values from 1024 to 262144 octets. The encoded value equals (actual_bytes / 1024) - 1, representing additional kilo-octets beyond the minimum 1024-octet baseline.
The Version field is 8 bits. The value 1 indicates exactly eight octets are present in the order and with the meanings defined in Section 4 of this document.
IANA created the 'RDMA-CM Private Data Identifiers' subregistry within the 'Remote Direct Data Placement' protocol category group. It contains 32-bit Format Identifiers, format length in octets, description, and reference. Initial entry: 0xf6ab0e18, length 8, 'RPC-over-RDMA version 1 CM Private Data'. New entries use Specification Required policy.
Exchanging the Private Data defined in this document does not expose upper-layer payloads to an attacker, and the resulting behavior changes (larger inline thresholds, remote invalidation) do not introduce new risk of upper-layer data exposure.
Improperly set fields in version 1 Private Data can cause disconnection (self-imposed DoS). Similarly, non-RPC-over-RDMA CM Private Data that inadvertently contains the Format Identifier 0xf6ab0e18 could be misinterpreted; additional validation per Section 5.2 mitigates this risk.
The integrity and source authenticity of CM Private Data are ensured by the exclusive use of Reliable Connected (RC) queue pairs. Any attempt to interfere with or hijack data on an RC connection causes the RDMA provider to terminate the connection.
When deployed with RDMAP, DDP, and MPA over TCP, IPsec protects the CM Private Data exchange by establishing a protected channel before any operations are exchanged. However, IPsec is not available for InfiniBand or RoCE deployments, which rely on physical security and CRC checks.
The CM Private Data field for RPC-over-RDMA version 1 is exactly 8 octets: a 32-bit Format Identifier, an 8-bit Version, a 7-bit Reserved field, a 1-bit R flag, an 8-bit Send Size, and an 8-bit Receive Size, all in network byte order.