Parallel NFS (pNFS) Flexible File Layout
Extracted elements (32)
LAYOUTSTATS are defined as cumulative (not reset per operation). When two LAYOUTSTATS for the same file and layout stateid from the same client are processed simultaneously, the one with larger values contains the most recent time-series data.
Loose coupling uses synthetic uids/gids for fencing: the MDS assigns unpredictable numeric uid/gid values to data files on storage devices, distributes them to clients in the layout, and fences clients by changing these values. This approach provides security equivalent to AUTH_SYS but cannot protect against malicious clients.
The flexible file layout type only supports sparse mapping (not dense mapping). With sparse mapping, the logical file offset is also the physical offset on the storage device, resulting in holes across storage devices that do not contain the current stripe index. This simplifies the layout type while supporting striping.
Storage devices using NFSv3 or NFSv4.0 are always treated as loosely coupled. NFSv4.1+ devices are tightly coupled only if they return EXCHGID4_FLAG_USE_PNFS_DS in EXCHANGE_ID and are advertised via a control protocol; otherwise they are treated as loosely coupled like NFSv4.0 devices.
When a client cannot use any configured ffdv_version/ffdv_minorversion combination for a storage device, it reports NFS4ERR_MINOR_VERS_MISMATCH via LAYOUTRETURN or LAYOUTERROR using ff_ioerr4 to identify the specific problematic device, then may retry GETDEVICEINFO or fall back to I/O through the MDS.
After a write error and receipt of a new layout, the client MUST NOT assume the new layout matches the previous one. If updates were not committed to all mirrors, the client MUST resend those updates to all mirrors in the new layout.
For tightly coupled storage devices, if ffdv_tightly_coupled is set, the client MUST ignore both ffds_user and ffds_group, since user and group on the storage device will be the same as on the metadata server.
If ffdv_tightly_coupled is not set, then the client MUST commit writes to the storage devices for the file before sending a LAYOUTCOMMIT to the metadata server — writes MUST be committed via WRITEs with stable_how == FILE_SYNC or by issuing a COMMIT after WRITEs with stable_how != FILE_SYNC.
If ffdv_version equals 3, the server MUST set ffdv_minorversion to 0 and ffdv_tightly_coupled to false, and the client MUST access the storage device using NFSv3. If ffdv_version equals 4, the server MUST set ffdv_minorversion to one of the NFSv4 minor version numbers, and the client MUST access the storage device using NFSv4 with the specified minor version.
If the lora_layout_type is LAYOUT4_FLEX_FILES and lou_type is LAYOUT4_FLEX_FILES, the lou_body field in the LAYOUTCOMMIT loca_layoutupdate argument MUST have zero length.
Once a COMMIT is replied to, all replicas must reflect the written data committed to stable storage. A client with outstanding unstable writes to a single node MUST do all reads from that same node to avoid stale reads.
Synthetic uids and gids of 0 SHOULD be avoided under AUTH_SYS, as uid 0 typically grants super access or maps to an anonymous id, either of which could allow a fenced client to continue accessing the data file.
The over-the-wire transfer of mirrored file contents MUST appear identical across all mirrors. Client-side mirroring requires the client to ensure modifications are made on all copies of layout segments it receives in the layout.
There MUST be exactly as many elements in ffds_fh_vers as there are in ffda_versions. For loose coupling with NFSv4 storage, the server MUST set ffds_stateid to the anonymous stateid since no global stateid model exists.
When FF_FLAGS_WRITE_ONE_MIRROR is not set and the client updates mirrors serially, it SHOULD stop at the first error and report to the MDS. If updating in parallel, it SHOULD wait for all storage devices to respond so it can report all errors encountered.
When FF_FLAGS_WRITE_ONE_MIRROR is set, the storage device MUST ensure all mirror copies are updated when any one mirror is updated; if the storage device gets an error updating a mirror, it MUST inform the client, who MUST then inform the metadata server.
When LAYOUT4_FLEX_FILES is used, the client MUST report I/O errors to the server at LAYOUTRETURN time using the ff_ioerr4 structure. The metadata server MUST recall any outstanding layouts before recovering from errors and MUST complete recovery before issuing new layouts for affected byte ranges.
When resilvering, the metadata server MUST recall writable layout segments being resilvered. If the client requests a writable layout for a segment being resilvered, the MDS may deny with NFS4ERR_LAYOUTUNAVAILABLE, forcing I/O through the metadata server.
CB_RECALL_ANY is extended with flexible file layout mask flags: PNFS_FF_RCA4_TYPE_MASK_READ (value 16) and PNFS_FF_RCA4_TYPE_MASK_RW (value 17), allowing the metadata server to selectively recall read-only or read-write layouts and specify how many the client may keep via craa_layouts_to_keep.
ff_device_addr4 is the layout-type-specific device address structure returned by GETDEVICEINFO. It contains ffda_netaddrs (multipath list of storage device network addresses) and ffda_versions (array of ff_device_versions4 specifying NFS version, minor version, rsize, wsize, and coupling strength).
ff_iostats4 reports I/O statistics per byte range at layout return: ffis_offset, ffis_length, ffis_stateid, ffis_read/write (io_info4 counts and bytes), ffis_deviceid, and ffis_layoutupdate (ff_layoutupdate4 with latency and network path details). Statistics are voluntary and client may report only 'hot' ranges.
ff_layout4 is the core layout structure containing ffl_stripe_unit (stripe unit size), ffl_mirrors (array of mirrored storage device sets), ffl_flags (bitmap of behavioral flags), and ffl_stats_collect_hint (hint for LAYOUTSTATS reporting frequency in seconds).
ffl_flags is a bitmap with four defined flags: FF_FLAGS_NO_LAYOUTCOMMIT (client need not send LAYOUTCOMMIT), FF_FLAGS_NO_IO_THRU_MDS (client should not proxy I/O through MDS), FF_FLAGS_NO_READ_IO (client should not send READs with RW iomode layouts), FF_FLAGS_WRITE_ONE_MIRROR (client need only update one mirror).
The flexible file layout type (LAYOUT4_FLEX_FILES, value 4) is defined as a pNFS layout that allows clients to access storage devices via NFSv3, NFSv4.0, NFSv4.1, or NFSv4.2 protocols. The layout separates metadata (on the MDS) from data (on storage devices) and supports client-side mirroring.
This document adds LAYOUT4_FLEX_FILES (value 4) to the existing pNFS Layout Types Registry and adds RCA4_TYPE_MASK_FF_LAYOUT_MIN (16) and RCA4_TYPE_MASK_FF_LAYOUT_MAX (17) to the NFSv4 Recallable Object Types Registry. No new IANA registries are created.
For loosely coupled storage, security is equivalent to AUTH_SYS: physical security, encrypted communications, or other measures are RECOMMENDED to protect MDS-to-storage communication from eavesdropping and man-in-the-middle attacks. The model cannot protect against malicious clients.
The metadata server enforces access control at LAYOUTGET time, verifying permissions and ACL. When file permissions or ACL change, the MDS SHOULD recall all layouts and MUST fence off clients still holding layouts by invalidating their distributed credentials before committing to new permissions. Clients must re-authenticate via new LAYOUTGET.
Loosely coupled locking model: OPENs, byte-range locks, delegations, TEST_STATEID, and FREE_STATEID are all handled locally by the MDS using anonymous stateids for storage device I/O. Mandatory byte-range locking cannot be supported. In case of stateid revocation, the MDS prevents client access and may use fencing.
Tightly coupled locking model: stateids are generated by the MDS and propagated to storage devices via control protocol. OPENs, OPEN_DOWNGRADE, CLOSE, advisory and mandatory locks, delegations, and FREE_STATEID all require MDS-to-storage-device state synchronization. Lease expiration on storage devices is possible.
ff_data_server4 contains per-storage-device layout state: ffds_deviceid (deviceid4), ffds_efficiency (uint32_t mirror quality hint), ffds_stateid (stateid4), ffds_fh_vers (array of filehandles, one per NFS version combination), ffds_user (fattr4_owner), and ffds_group (fattr4_owner_group) for RPC credentials.
ff_ioerr4 reports I/O errors: ffie_offset (offset4) and ffie_length (length4) identify the byte range, ffie_stateid (stateid4) identifies the state, and ffie_errors (array of device_error4) reports per-device operation and error type. NFSv3 errors must be mapped to NFSv4 status codes.
ff_layoutreturn4 is sent in the lrf_body of LAYOUTRETURN when layout type is LAYOUT4_FLEX_FILES. It contains fflr_ioerr_report (array of ff_ioerr4 for I/O errors) and fflr_iostats_report (array of ff_iostats4 for usage statistics); zero-length array indicates no errors.