Parallel NFS (pNFS) Small Computer System Interface (SCSI) Layout
Extracted elements (34)
Client-side copy-on-write support is explicitly required (via the READ_DATA / INVALID_DATA extent overlap mechanism) because pNFS enables clients to perform direct block access; classic NFS would have the server perform copy-on-write, but doing so through the MDS in a pNFS environment would negate the performance benefits of direct I/O.
The LAYOUTCOMMIT payload for the SCSI layout type uses a commit list of byte ranges (pnfs_scsi_layoutupdate4) rather than sending full extent data. This design reduces the size of the LAYOUTCOMMIT payload compared to the block layout type in RFC 5663, optimizing the I/O path.
This document supersedes the block layout type in RFC 5663 for SCSI-based storage by using SCSI persistent reservations for mandatory fencing (rather than the optional RFC 6688 disk protection mechanism) and SCSI device identifiers (VPD page 0x83) for reliable device discovery instead of probing all attached devices. These changes provide more reliable fencing and more efficient device identification.
The pNFS SCSI layout type is based on RFC 5663 (pNFS Block/Volume Layout) and shares its volume topology data structures above the base-volume level and extent data structures. RFC 5663 remains the defining document for the existing block layout type; the two layout types coexist in the pNFS Layout Types Registry.
A client that detects a RESERVATION CONFLICT SCSI status MUST commit all layouts using the storage device through the MDS, return all outstanding layouts for the device, forget the device ID, and unregister the reservation key.
A pNFS client MUST NOT perform direct I/O operations that are not permitted by an extent held by the client. Layout extents are the sole means of obtaining permission for direct I/O to storage devices.
A pNFS client MUST possess both a valid open stateid and a valid layout stateid before issuing any SCSI I/O operation. I/O using anonymous or write-bypass stateids MUST NOT be done using the pNFS SCSI layout type.
A pNFS server MUST avoid partial-block write conflicts by implementing a concurrency control policy of single writer XOR multiple readers for a given data block, since SCSI storage does not provide byte-granularity access.
A request for a readable layout MUST return only PNFS_SCSI_READ_DATA or PNFS_SCSI_NONE_DATA extents and SHALL NOT return PNFS_SCSI_INVALID_DATA or PNFS_SCSI_READ_WRITE_DATA extents. A request for a writable layout MUST return PNFS_SCSI_READ_WRITE_DATA or PNFS_SCSI_INVALID_DATA extents and SHALL NOT return PNFS_SCSI_NONE_DATA extents.
All SCSI pNFS clients MUST support copy-on-write processing. When writing to a PNFS_SCSI_READ_DATA extent region, the client MUST copy data from the READ_DATA extent for partial blocks, merge in the changes, and write the result to the corresponding PNFS_SCSI_INVALID_DATA extent.
Before performing the first I/O to a device, the client MUST register the sbv_pr_key reservation key via PERSISTENT RESERVE OUT (REGISTER). When a client stops using a device, it MUST unregister the key via PERSISTENT RESERVE OUT (REGISTER) with the earlier key in the RESERVATION KEY field.
Each system (MDS and each client) MUST use a unique persistent reservation key generated by the MDS. The MDS MUST generate a key for itself before exporting a volume and a key for each client that accesses SCSI layout volumes.
Extents in the extent list MUST be logically contiguous for a read-only layout. For a read-write layout, the set of writable extents MUST be logically contiguous, and every PNFS_SCSI_READ_DATA extent in a read-write layout MUST be covered by one or more PNFS_SCSI_INVALID_DATA extents.
Extents included in the LAYOUTCOMMIT slu_commit_list MUST be aligned to the server's file system block granularity and have a size that is a multiple of that granularity. The extents in the commit list MUST be disjoint and MUST be sorted by sr_file_offset.
Extents MUST be aligned to logical block size boundaries of the underlying logical units (typically 512 or 4096 bytes). For complex volume topologies, the server MUST ensure extents are aligned to the largest logical block size in the topology.
In environments where pNFS clients cannot be trusted to enforce file-based security policies, pNFS SCSI layouts MUST NOT be used, because SCSI storage devices generally cannot enforce such file-based access controls.
Logical units used as pNFS SCSI layout devices and the SCSI initiators used for the pNFS metadata server and clients MUST support SCSI persistent reservations as defined in SPC-4.
The layout hint attribute (loh_type LAYOUT4_SCSI_VOLUME) is not supported; the pNFS server MUST reject setting this attribute during OPEN or SETATTR, and MUST NOT report the layout_hint attribute in supported_attrs on a SCSI-only file system.
The LAYOUTRETURN operation for the SCSI layout type requires no layout-specific data; the opaque lrf_body field of the layoutreturn_file4 data structure MUST have length zero.
The NFSv4 server MUST NOT grant layouts that conflict with mandatory locks or share reservations. If a conflicting mandatory lock or OPEN request arrives, the server MUST recall the conflicting portion of the layout before granting the request.
When a client holds a writable layout with data in PNFS_SCSI_INVALID_DATA state after a server crash and the data is no longer in the client's cache, the client MUST NOT source data from the data servers; instead it SHOULD commit blocks to the server during the recovery grace period via LAYOUTCOMMIT with loca_reclaim set to true.
When a file is truncated, the server MUST recall from clients the portions of any pNFS layouts that refer to blocks that will be freed or written before effecting the file truncation.
When open stateids are revoked, the server MUST recall all layouts that allow I/O on any files affected by the revocation. If those layouts are not successfully returned, the client MUST be fenced.
When volatile write caches are present on storage devices (indicated by WCE=1 in the Caching mode page), the pNFS server MUST ensure the volatile write cache has been committed to stable storage before the LAYOUTCOMMIT operation returns, using SYNCHRONIZE CACHE commands.
Client fencing is implemented using SCSI persistent reservations of type 'Exclusive Access - Registrants Only' (type 8h) on each SCSI LU. The MDS registers a key and places the reservation before returning any PNFS_SCSI_VOLUME_BASE volume to clients, enabling it to preempt non-responding clients at any time.
The pnfs_scsi_extent_state4 enumeration defines four extent states: PNFS_SCSI_READ_WRITE_DATA (valid for reading and writing), PNFS_SCSI_READ_DATA (read-only), PNFS_SCSI_INVALID_DATA (allocated but uninitialized; reads return zeros), and PNFS_SCSI_NONE_DATA (hole in the file; invalid location).
The pNFS SCSI layout type assigns layouttype4 value LAYOUT4_SCSI = 5, extending the existing layout type enumeration. This new layout type maps NFS file regions to extents on SCSI block storage devices accessed via protocols such as FCP, iSCSI, or SAS.
The pnfs_scsi_volume_type4 enumeration defines four volume types: PNFS_SCSI_VOLUME_BASE (maps to a single LU), PNFS_SCSI_VOLUME_SLICE (a slice of another volume), PNFS_SCSI_VOLUME_CONCAT (concatenation of multiple volumes), and PNFS_SCSI_VOLUME_STRIPE (striped across multiple volumes).
IANA has assigned a new pNFS layout type in the 'pNFS Layout Types Registry': LAYOUT4_SCSI with value 0x00000005, applicable to NFSv4.1 minor version 1.
NFSv4 security is not directly applicable to pNFS SCSI layouts because SCSI storage access is at a lower layer than NFSv4. The security provided depends on the storage protocol (e.g., SAS provides essentially none; iSCSI can provide significant security via IPsec per RFC 3723/RFC 7146). In environments where security requirements for the storage protocol cannot be met, pNFS SCSI layouts SHOULD NOT be used.
SCSI storage device access control operates at host/LU granularity, not at file or block granularity. The responsibility for enforcing NFSv4 security and locking semantics—including restricting access to extents covered by the layout and preventing access beyond returned layouts—is delegated entirely to the pNFS client.
The pnfs_scsi_base_volume_info4 structure encodes a logical unit name and reservation key, containing sbv_code_set (pnfs_scsi_code_set enum), sbv_designator_type (pnfs_scsi_designator_type enum), sbv_designator (variable-length opaque), and sbv_pr_key (uint64_t).
The pnfs_scsi_extent4 structure contains se_vol_id (deviceid4), se_file_offset (offset4), se_length (length4), se_storage_offset (offset4), and se_state (pnfs_scsi_extent_state4). The pnfs_scsi_layout4 wraps a variable-length array of these extents as the loc_body for SCSI layouts.
The pnfs_scsi_layoutupdate4 structure (used as the SCSI layout-specific argument in LAYOUTCOMMIT) contains slu_commit_list, a variable-length array of pnfs_scsi_range4 structures (each with sr_file_offset and sr_length) covering regions transitioned from PNFS_SCSI_INVALID_DATA to PNFS_SCSI_READ_WRITE_DATA.