Parallel NFS (pNFS) Block/Volume Layout

Client participation in copy-on-write is enabled by the combination of PNFS_BLOCK_READ_DATA and PNFS_BLOCK_INVALID_DATA extent states with their permitted overlap in writable layouts. In classic NFS, copy-on-write is done server-side; since pNFS enables direct client block access, shifting CoW to clients reduces server load and improves parallelism.

Negative bsc_sig_offset values (indicating byte offsets from the end of the volume) are permitted in pnfs_block_sig_component4 to simplify client implementation on systems where device labels are at a fixed offset from the volume end. However, they SHOULD NOT be used on dynamically resizable volumes unless the label is guaranteed to track the end-of-volume offset.

Volume identification uses content-based opaque byte-sequence matching rather than network addresses because the same storage volume may be simultaneously accessible via multiple network technologies (e.g., both iSCSI and Fibre Channel through different ports on different networks), making network addresses unreliable for unique identification across multipath environments.

Unilateral layout revocation (client fencing) for block/volume storage can only operate at host/LUN granularity. When one layout is revoked for a client, all of that client's layouts for files on the same storage units are effectively rendered unusable, which may impact files in other file systems on those storage units.

A pNFS client MUST NOT perform direct I/O operations on storage that are not permitted by a layout extent currently held by the client. Layout extents are the sole authorization mechanism for client direct storage I/O; this places the pNFS server in control of potentially conflicting storage operations.

security

All block/volume pNFS clients MUST support copy-on-write processing. When writing to an area covered by PNFS_BLOCK_READ_DATA extents, the client MUST request a writable layout, merge partial block changes with data fetched from the READ_DATA extent, and write the merged result to the corresponding PNFS_BLOCK_INVALID_DATA extent. After storing data in INVALID_DATA, subsequent reads MUST come from that (now READ_WRITE_DATA) extent; the READ_DATA extent MUST NOT be used.

Any block pNFS system using this layout MUST support content-based unique volume identification via the pnfs_block_sig_component4 data structure. This is necessary because the same volume may be accessible via multiple network technologies (e.g., iSCSI and Fibre Channel), making network addresses unsuitable for volume identification.

Because clients cannot determine whether the server uses LUN masking, clients MUST always implement timed-lease fencing. After a layout lease expires (lease_time seconds since last renewal), the client MUST cease using that layout and SHOULD revert to standard NFSv4.1 READ/WRITE operations. If max I/O time cannot be bounded, the client MUST use all-1s as blh_maximum_io_time.

security

During crash recovery, if a client wrote data to PNFS_BLOCK_INVALID_DATA blocks that are no longer in the client cache, the client MUST NOT attempt to source that data from the data servers. Instead, it should commit the blocks to the server during the recovery grace period via LAYOUTCOMMIT with the loca_reclaim flag set to true.

Extents in the LAYOUTCOMMIT blu_commit_list MUST be disjoint and MUST be sorted by bex_file_offset. They MUST be aligned to the server's file-system block size granularity with sizes that are multiples of that granularity. If the end-of-file falls within a committed block, the client MUST zero-fill from EOF to the end of that block before committing.

Extents MUST be aligned to 512-byte boundaries; writable extents MUST be aligned to the NFSv4 server's file-system block size (available as the layout_blksize attribute). Readable extents SHOULD be aligned to the server block size, but 512-byte alignment is acceptable for legacy file systems with fragments.

For LAYOUTRETURN with LAYOUTRETURN4_FILE_return type, the opaque lrf_body field MUST have length zero — there is no block-layout-specific return payload. Multiple LAYOUTRETURN operations within a single COMPOUND may be used to return disjoint file regions.

If a SEQUENCE response has SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED, SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED, or SEQ4_STATUS_ADMIN_STATE_REVOKED set in sr_status_flag, the client MUST immediately cease using all layouts and device ID to device address mappings associated with the corresponding server.

In a LAYOUTGET extent list: the first extent MUST contain the requested starting offset; the total size MUST cover at least loga_minlength (except at EOF for read-only requests); extents MUST be logically contiguous for read-only layouts; for read-write layouts writable extents MUST be logically contiguous; extents MUST be ordered by bex_file_offset with READ_DATA preceding INVALID_DATA on ties.

In the pnfs_block_deviceaddr4 volume array, concat, slice, and stripe volumes MUST refer to volumes defined by lower-indexed elements of the array. The array is ordered so that the root of the volume hierarchy is the last element.

The server MUST maintain the last blh_maximum_io_time hint per client and MUST apply any updated value to all files for which that client holds a layout. A client that receives NFS4ERR_INVAL in response to the layout-hint SETATTR MUST NOT subsequently issue LAYOUTGET; the server MUST then return NFS4ERR_LAYOUTUNAVAILABLE to all subsequent LAYOUTGET requests from that client.

When a file is truncated via SETATTR, the server MUST recall from clients all pNFS layout portions referring to blocks that will be freed or zeroed by the truncation before processing the request. If unable to complete in reasonable time, the server SHOULD reply with NFS4ERR_DELAY.

When storage devices do not provide atomicity for concurrent read/write or write/write operations, the pNFS server MUST implement a single-writer XOR multiple-readers concurrency control policy via layout grants and recalls. The server MUST NOT grant layouts conflicting with mandatory locks or share reservations, and MUST recall conflicting layout portions before granting such requests.

security

A pNFS block/volume layout maps NFS file regions to extents on a logical storage volume. Extents use 64-bit byte offsets and lengths (offset4, length4 types), and the volume topology is communicated via GETDEVICEINFO using a pnfs_block_deviceaddr4 structure returned as the storage-protocol-specific da_addr_body field.

CB_RECALL_ANY uses RCA4_TYPE_MASK_BLK_LAYOUT = 4 in craa_type_mask to request a client to return block layouts. The client should immediately respond NFS4_OK and then asynchronously return complete file layouts until the number of files with cached layouts falls below craa_objects_to_keep.

Client fencing prevents a client with an expired or revoked layout from continuing I/O to storage. Two approaches are defined: LUN masking/mapping at the storage system or SAN (preferred), and timed-lease fencing where the client ceases I/O after lease_time seconds. The server must wait at least lease_time + blh_maximum_io_time before reallocating layouts from the original client to another.

security

The pnfs_block_extent_state4 enum defines four extent states: PNFS_BLOCK_READ_WRITE_DATA (valid, readable and writable), PNFS_BLOCK_READ_DATA (valid, read-only; write requests prohibited), PNFS_BLOCK_INVALID_DATA (allocated but uninitialized, reads as zero, writes convert to READ_WRITE_DATA), and PNFS_BLOCK_NONE_DATA (hole with no physical storage, reads as zero, never returned for writable layouts).

Access to block/volume storage operates below the NFSv4 security layer; NFSv4 security mechanisms (encryption, cryptographic integrity) may not apply. Security granularities differ: NFSv4 controls user-to-file access, while iSCSI controls initiator-to-volume access. Administrators must ensure appropriate protection for the storage protocol; in environments where requirements cannot be met, block/volume pNFS layouts SHOULD NOT be used.

security

Block/volume storage devices are unable to validate NFS ACLs, mandatory locks, or file open modes. In environments where pNFS clients cannot be trusted to enforce such file-based policies, pNFS block/volume storage layouts SHOULD NOT be used. Virtual block addresses exported by storage devices can enable device-side access checks as an alternative.

security

Block/volume storage systems enforce access control at volume (LUN) granularity, not per-file or per-block. pNFS clients must be trusted to restrict I/O to the layout extents they hold, because the server cannot prevent a client from accessing parts of a physical disk not covered by its layout, nor from accessing blocks covered by a layout it has already returned.

security

The pnfs_block_extent4 structure contains: bex_vol_id (deviceid4, logical volume identifier), bex_file_offset (offset4, starting byte offset in file), bex_length (length4, size in bytes), bex_storage_offset (offset4, starting byte offset in volume, validity depends on bex_state), and bex_state (pnfs_block_extent_state4). The pnfs_block_layout4 wraps an array of these extents as the block-layout-specific loc_body.

The pnfs_block_layouthint4 structure is the block-layout-specific loh_body for LAYOUT4_BLOCK_VOLUME layout hints set via SETATTR. It contains blh_maximum_io_time (uint64_t, maximum I/O time in seconds; all-1s indicates an unbounded I/O time). Clients using block layouts MUST set this hint attribute before issuing the first LAYOUTGET operation.

The pnfs_block_layoutupdate4 structure is the block-layout-specific LAYOUTCOMMIT argument, containing blu_commit_list: an array of pnfs_block_extent4 entries covering file regions previously in PNFS_BLOCK_INVALID_DATA state that have been written. All extents MUST have bex_state set to PNFS_BLOCK_READ_WRITE_DATA; bex_storage_offset is unused.

The pnfs_block_sig_component4 structure identifies a storage volume by content: bsc_sig_offset (int64_t, byte offset from volume start; negative means offset from end) and bsc_contents (opaque, MUST NOT be interpreted as zero-terminated). A signature is an array of up to PNFS_BLOCK_MAX_SIG_COMP (16) components; components need not be co-located in a single sector.

The pnfs_block_volume_type4 enum defines PNFS_BLOCK_VOLUME_SIMPLE (single LU), PNFS_BLOCK_VOLUME_SLICE (portion of another volume with bsv_start, bsv_length, bsv_volume), PNFS_BLOCK_VOLUME_CONCAT (bv_volumes array of indices), and PNFS_BLOCK_VOLUME_STRIPE (bsv_stripe_unit size + bsv_volumes array). These are combined in pnfs_block_deviceaddr4 as an array where the root is the last element.