Using the Parallel NFS (pNFS) SCSI Layout to Access Non-Volatile Memory Express (NVMe) Storage Devices
Extracted elements (21)
NGUID is preferred over EUI64 for NVMe namespace identification because it is the larger (16-octet vs 8-octet) identifier, providing a larger namespace for unique identifiers and reducing collision risk. The XDR length field implicitly encodes which identifier type is present.
The NVMe Flush command is the direct NVMe equivalent to SCSI SYNCHRONIZE CACHE commands. The mapping is used for volatile write cache flushing before LAYOUTCOMMIT, preserving the same semantic guarantee across both protocols.
This document does not amend RFC 8154 (pNFS SCSI layout); instead it establishes a mapping from SCSI constructs to NVMe constructs. This approach allows NVMe namespaces to be used within the existing SCSI layout framework without modifying the base specification.
NVMe reservation keys always apply to all controllers used by a host (identified by NVMe Host Identifier), analogous to the SCSI ALL_TG_PT bit. Unlike SCSI where ALL_TG_PT support is inconsistent and unreliable, this behavior is always supported in NVMe Reservations.
PS_DESIGNATOR_T10 and PS_DESIGNATOR_NAME identifiers used in SCSI layouts have no equivalent in NVMe and cannot be used to identify NVMe storage devices in a pNFS SCSI layout.
If a volatile write cache is enabled on an NVMe namespace used as pNFS SCSI storage, the pNFS server (MDS) MUST use the NVMe Flush command to flush the volatile write cache to stable storage before LAYOUTCOMMIT returns.
If an NVMe command returns a non-retryable error (DNR bit set), the client MUST commit all layouts using that storage device through the MDS, return all outstanding layouts, forget the device ID, and unregister its reservation key.
NVMe namespaces used as pNFS SCSI layout storage devices MUST support either the EUI64 or NGUID value reported in a Namespace Identification Descriptor, the I/O Command Set Independent Identify Namespace data structure, and the Identify Namespace data structure.
The 'sbv_designator' XDR opaque<> length MUST be 16 octets for an NVMe NGUID identifier and MUST be 8 octets for an NVMe EUI64 identifier. All other lengths MUST NOT be used with an NVMe namespace.
When constructing pnfs_scsi_base_volume_info4 for an NVMe namespace, the 'sbv_code_set' field SHALL be set to PS_CODE_SET_BINARY and 'pnfs_scsi_designator_type' SHALL be set to PS_DESIGNATOR_EUI64. If both NGUID and EUI64 are available, NGUID SHOULD be used.
A volatile write cache on an NVMe controller is considered enabled when bit 0 of the Volatile Write Cache (VWC) field in the Identify Controller data structure is set AND the Volatile Write Cache Enable (WCE) bit in the Volatile Write Cache Feature (Feature Identifier 06h) is set. The NVMe Flush command (NVME-BASE Section 7.1) is the NVMe equivalent of SCSI SYNCHRONIZE CACHE.
Before returning a PNFS_SCSI_VOLUME_BASE volume to the client, the MDS registers its reservation key and then issues a Reservation Acquire command with RACQA=000b (Acquire) and RTYPE=4h (Exclusive Access - Registrants Only) to establish the reservation.
Client fencing uses NVMe Persistent Reservations (PRs). Both the MDS and clients register a reservation key with the storage device, and the MDS creates an Exclusive Access - Registrants Only Reservation (RTYPE=4h) using the Reservation Acquire command.
Reservation keys are registered on NVMe namespaces using the Reservation Register command (NVME-BASE Section 7.3) with RREGA=000b and the key in NRKEY. Keys are unregistered with RREGA=001b and the key in CRKEY.
The MDS fences a non-responding client by issuing a Reservation Acquire command with RACQA=001b (Preempt) or 010b (Preempt and Abort), CRKEY set to the server's key, PRKEY set to the client's key, and RTYPE=4h. The client can detect fencing via the Reservation Conflict NVMe status code.
Volume identification for NVMe namespaces maps to the SCSI Device Identification VPD page (page code 83h). NVMe namespaces are identified via their EUI64 (8-octet) or NGUID (16-octet) identifiers encoded as XDR variable-length opaque in the sbv_designator field. PS_DESIGNATOR_T10 and PS_DESIGNATOR_NAME have no NVMe equivalent and cannot be used.
In environments where security requirements for the storage protocol cannot be met, pNFS with an NVMe layout SHOULD NOT be deployed. Similarly, where client-side protection from access to storage outside of the layout is insufficient, pNFS with SCSI layout on NVMe SHOULD NOT be deployed.
NFSv4 and RPC security are not directly applicable to I/Os to NVMe data servers. Security for NVMe traffic operates at a different granularity and identity model than NFSv4; the pNFS client is responsible for enforcing appropriate correspondences between these security layers.
pNFS with NVMe layouts may use transports that provide no additional security (e.g., NVMe over PCIe). Physical and/or logical security measures such as air gaps and isolated VLANs may substitute for protocol-level security when appropriate for the sensitivity of the data involved.
When a metadata server revokes a layout, the client's access MUST be terminated at the storage devices via fencing as specified in Section 2.2. The client then has the opportunity to acquire a new layout.
When using IP-based storage protocols such as NVMe over TCP, data confidentiality and integrity SHOULD be provided using TLS (RFC 8446). For NVMe over TCP specifically, TLS SHOULD be used as described in the NVMe TCP Transport Specification to protect traffic between pNFS clients and NVMe namespaces.