Network File System (NFS) Direct Data Placement

NFSv4.1 callbacks carry no bulk data (no READ or WRITE), so they present no particular issue over RPC/RDMA and MAY be transmitted inline via RDMA_MSG or via RDMA_NOMSG if they overflow negotiated buffer sizes.

The identification of RDMA Read chunks can be implemented entirely within the RPC/XDR layer, transparent to NFS. File data in an NFS WRITE can be selected as an RDMA chunk based on size criteria independently of the NFS protocol layer, with XDR position values in Read chunk entries providing the correspondence.

The NFSv4 COMPOUND maximum size is unbounded even with RDMA chunks, making a configuration protocol exchange unworkable due to layering issues in how COMPOUNDs are built; however, typical NFSv4 clients behave predictably and support traditional rsize/wsize parameters.

Using RDMA Write lists in NFS requests avoids the need for RDMA Read lists in replies, enabling more efficient operation by avoiding server buffer exposure for RDMA and eliminating 'RDMA_DONE' exchanges.

Write padding via 'RDMA_MSGP' is OPTIONAL for non-RDMA WRITE transfers and is likely restricted to COMPOUNDs containing only a single WRITE operation to ensure accurate alignment for all data.

In the NFSv4.1 callback shared-connection case, the XID in the RPC/RDMA header may collide with XIDs used by the client for other requests; both client and server must inspect the RPC component to determine whether the message is an RPC CALL or RPC REPLY before processing the XID.

NFS use of direct data placement requires an additional NFS port assignment for networks sharing traditional UDP/TCP port spaces with RDMA services (e.g., iWARP), unlike InfiniBand which does not share this port space.

registry

The use of a configuration protocol with NFS v2 and v3 is OPTIONAL because practical data sizes (<=8 KB for v2, <=32 KB for v3) allow basic operation without configuration exchange; however, configuration may improve server resource management.

This NFS DDP specification applies to NFSv4.0 and any subsequent minor versions that do not override this mapping. NFSv4.1 introduces a comprehensive upper-layer parameter exchange as part of RFC 5661.

An NFS v2 or v3 server supporting RPC/RDMA MAY choose an arbitrary port or MAY use the alternative well-known port number, and MAY register with the RPC portmapper under the netid assigned by RFC 5666.

registry

An NFSv4 server supporting RPC/RDMA on networks sharing UDP/TCP port space with RDMA services MUST use the alternative well-known port number for its RPC/RDMA service. Clients SHOULD connect to this port without consulting the RPC portmapper.

registry

For NFS v2 and v3, the server MUST post buffers large enough to receive the largest possible incoming message (approximately 12 KB for NFSv2, 36 KB for NFSv3), and MUST NOT return data in excess of those sizes.

For NFSv4, the Write list MUST be considered only for the COMPOUND procedure, and only the opaque file data from READ and the pathname from READLINK MUST utilize Write list entries.

If a Write list entry is presented with non-zero segments, the corresponding READ or READLINK MUST return its data via RDMA Write to that buffer. If the entry has zero segments or total size zero, the operation MUST return its result inline.

If there is no Write list (null), then any READ or READLINK operations in the NFSv4 COMPOUND MUST return their data inline. The NFSv4.0 client MUST ensure results will fit within its receive buffers to avoid RDMA transport errors.

In NFSv4.1 when the callback channel shares a connection with client requests, the client MUST ascertain whether an incoming message is an RPC REPLY before processing it, and the server MUST ascertain whether an incoming message is an RPC CALL. Both MUST NOT reject or accept a message without determining its proper context.

The RDMA Write list buffer total size MUST be large enough to accept the result. If the buffer is too small, the server MUST return an XDR encode error.

security

The server MUST return result data for a posted buffer by progressively filling its segments, leaving trailing segments unfilled or partially full if the result is smaller than the total buffer. Results returned by direct placement MUST NOT be returned by other methods such as Read chunk list or inline.

Write list entries are consumed in order: the first entry MUST be used by the first READ or READLINK in the COMPOUND, the next by the next, and so on. If more READ/READLINK operations exist than Write list entries, the remaining MUST return results inline.

The xdr_rdma_segment structure describes a single memory segment using three fields: handle (uint32, registered memory STag), length (uint32, chunk size in bytes), and offset (uint64, virtual address or offset).

The xdr_write_chunk structure contains an array of xdr_rdma_segment entries (target<>), representing a single client-posted buffer as potentially discontiguous memory segments for scatter-write by the server.

The xdr_write_list structure is a linked list of xdr_write_chunk entries, allowing the client to post multiple result buffers each mapped to a specific result in the reply.

All relevant security considerations for NFS DDP are addressed at the RPC/RDMA transport layer (RFC 5666); this document defines only the NFS protocol binding atop that layer.

security

The RDMA transport for RPC supports all RPC security models including RPCSEC_GSS and link-level security. The use of RDMA Read and Write for argument and result transfer does not introduce new vulnerabilities, as RFC 5666 ensures security requirements are maintained regardless of data transfer method.

security, crypto

xdr_rdma_segment encodes: handle as uint32 (4 bytes, registered memory handle/STag), length as uint32 (4 bytes, chunk length in bytes), offset as uint64 (8 bytes, virtual address or offset); total 16 bytes per segment.