Network File System (NFS) Upper-Layer Binding to RPC-over-RDMA Version 1
obsoletes
- rfc-5667 — Network File System (NFS) Direct Data Placement
Extracted elements (28)
MOUNT, NLM, and NSM auxiliary protocols have historically been conveyed via TCP even when NFS/RDMA is in use; requiring TCP support for these protocols avoids needing separate RDMA upper-layer bindings for protocols with no significant data payloads.
NFS sessions (RFC 5661) have no effect on RPC-over-RDMA version 1 operation; there is no need to match session slot counts with RPC-over-RDMA credit counts. Transport errors during session use require slot recovery to prevent permanently unavailable slots.
NFS version 4.0 clients should avoid combining non-idempotent operations (e.g., RENAME) with GETATTR operations requesting variable-length results in the same COMPOUND, because short Reply chunk retry cannot safely be applied to such combinations. GETATTR with unpredictable-length results should be sent in a COMPOUND by itself.
Reply size estimation is necessary to avoid reply resource overruns, where the RPC Reply Payload stream does not fit into the provided Reply chunk, preventing the responder from returning the upper-layer reply. Clients can use count/offset fields, result limits, cached sizes, or negotiated maximums (CREATE_SESSION) to estimate reply sizes.
RPC-over-RDMA version 1 is built on RDMA Reliable Connections, which are reliable, connection-oriented, and guarantee in-order delivery, satisfying NFS version 4's requirement for congestion-controlled transports without needing additional mechanisms.
Short Reply chunk retry is intended for only a few operations where the size of returned data items cannot be known in advance; in all other cases, reply size underestimation is considered a correctable implementation bug rather than a protocol-level retry mechanism.
The NFSACL RPC program is conveyed on the same connection as the NFS RPC program (100003) to avoid separate rpcbind queries. ACL data items are not DDP-eligible because ACLs must be encoded and decoded regardless of size.
GSS services increase the size of credential material in RPC headers when Kerberos integrity or privacy is in use, potentially requiring more frequent use of Long messages and increasing chunk list complexity independent of the NFS version 4 COMPOUND being conveyed.
NFS version 4.0 callback operations typically use a separate TCP connection even when the forward channel uses RPC-over-RDMA version 1; no data items in the NFS version 4.0 callback RPC program are DDP-eligible.
Some RDMA transports (such as InfiniBand Reliable Connections) have no keep-alive mechanism; without new RPC traffic, connections can remain alive long after a server becomes unresponsive, eventually exhausting all RPC-over-RDMA credits and blocking further requests.
When an NFS session is not present (e.g., NFS version 4.0), a transport error does not indicate whether the server processed the RPC Call arguments or accessed/modified client memory associated with that RPC.
A Legacy NFS server MUST provide support for MOUNT, NLM, and NSM protocols via TCP to enable interoperation when NFS/RDMA is in use.
An NFS version 4 server SHOULD return either an RDMA_ERROR message with ERR_CHUNK or an RPC Reply with GARBAGE_ARGS when an RPC transaction cannot be processed due to chunk list complexity limits. After receiving one of these errors, an NFS version 4 client SHOULD NOT retransmit the failing request.
An NFS version 4 server supporting RPC-over-RDMA version 1 MUST use the alternative well-known port number (20049) for its RPC-over-RDMA service. Clients SHOULD connect to this well-known port without consulting the RPC portmapper.
If a READ operation returns a union arm without a DDP-eligible result and the NFS version 4 client provided a matching non-empty Write chunk, an NFS version 4 server MUST return an empty Write chunk in that Write list position.
If a transport error (RDMA_ERROR) is indicated before or instead of a disconnect, the requester MUST respond to that error as prescribed by the RPC transport specification; then NFS version 4 retransmission rules apply.
If an NFS version 4.1 client cannot support Long Calls in its backchannel (i.e., backchannel ca_maxrequestsize is not larger than inline thresholds), an NFS version 4 server MUST use only Short messages to convey backchannel operations.
If an NFS version 4 client has provided a matching non-empty Write chunk, the corresponding READ operation MUST return its DDP-eligible data item using that chunk. If the client provided an empty matching Write chunk, the READ operation MUST return all result data items inline.
If there are more READ operations than Write chunks, remaining NFS READ operations in an NFS version 4 COMPOUND that have no matching Write chunk MUST return their results inline.
In NFS version 4.1 and newer minor versions, NFS version 4 clients and servers MUST use the approach described in RFC 8167 when backchannel operations are conveyed on RPC-over-RDMA version 1 transports.
NFS version 4 clients SHOULD follow chunk list complexity prescriptions: the Read list can contain either a Position Zero Read chunk, one Read chunk with a non-zero Position, or both; the Write list can contain no more than one Write chunk; any chunk can contain up to sixteen RDMA segments. NFS version 4 servers MUST accept and process such requests.
NFS version 4 clients SHOULD reserve one RPC-over-RDMA credit for periodic server or connection health assessment, to drive an RPC request on an otherwise idle connection and detect unresponsive servers.
The first Write chunk in the Write list MUST be used by the first READ operation in an NFS version 4 COMPOUND procedure; subsequent Write chunks are consumed by subsequent READ operations in order.
An Upper-Layer Binding specifies which XDR data items and RPC procedures are DDP-eligible for one or more versions of an RPC program, along with RPC binding assignments, Write chunk pairing, and reply size estimation.
For NFS version 4 COMPOUND procedures, only four XDR data items are DDP-eligible: the opaque data field in WRITE4args, the linkdata field of NF4LNK in createtype4, the opaque data field in READ4resok, and the linkdata field in READLINK4resok.
For NFS versions 2 and 3, the DDP-eligible XDR data items are: the opaque file data argument in WRITE, the pathname argument in SYMLINK, the opaque file data result in READ, and the pathname result in READLINK. All other argument or result data items are not DDP-eligible.
IANA has assigned port 20049 (nfsrdma) for NFS over RDMA on TCP, UDP, and SCTP in the IANA port registry. This is the well-known port that NFS version 4 servers supporting RPC-over-RDMA version 1 must use.
RPC-over-RDMA version 1 supports all RPC security models including RPCSEC_GSS and transport-level security (RFC 7861). The choice of DDP mechanism to convey RPC arguments and results does not affect security, as it changes only the method of data transfer; all relevant security considerations are addressed at the RFC 8166 layer.