Object-Based Parallel NFS (pNFS) Operations

Serialization of overlapping writes in RAID layouts with parity (RAID-4, RAID-5, RAID-PQ) or mirroring is the responsibility of the metadata server rather than the clients. For example, the server enforces this by not granting overlapping write layouts within mirrored objects to different clients.

The object-based layout always uses a 'dense' layout where the second stripe unit starts at offset 0 of the second component rather than at byte offset stripe_unit. This avoids holes in component objects and ensures that after a full stripe, the next stripe unit appends directly to the first component.

The OSD protocol version is explicitly encoded in pnfs_osd_object_cred4 (oc_osd_version), even though it could be inferred from the capability format byte. The explicit field is considered more robust for handling OSD v1 and anticipated v2 capabilities, which have different features (byte-range capabilities, snapshots).

OSD byte-range capabilities (used in OSDv2) cannot be merged or split at the pNFS protocol layer, unlike generic NFSv4.1 layout segments. This means layout segment tracking must be per-capability granularity, and LAYOUTRETURN cannot return arbitrary sub-ranges of a held layout segment.

After recovery from an I/O error, the metadata server MUST complete recovery before handing out any new layouts to the affected byte ranges; it SHOULD recall all outstanding layouts to gain exclusive write access during recovery.

Component objects MUST appear in the olo_components array exactly once. The data placement algorithm that maps file offsets to component objects assumes each component is unique in the array.

In response to CB_LAYOUTRECALL, each LAYOUTRETURN sent MUST completely cover at least one outstanding layout segment; the client MUST NOT return a sub-range of a layout segment it holds, because OSD byte-range capabilities cannot be split.

The capability key SHOULD be transferred securely to prevent eavesdropping. A client SHOULD issue LAYOUTGET or GETDEVICEINFO via RPCSEC_GSS with the privacy service, or previously establish a Secret State Verifier (SSV) via NFSv4.1 SET_SSV to enable SSV-encrypted capability key delivery.

security, crypto

The client MUST report I/O errors to the metadata server at LAYOUTRETURN time using the pnfs_osd_ioerr4 structure whenever errors occur during direct OSD access with the LAYOUT4_OSD2_OBJECTS layout type.

The metadata server MUST revoke outstanding capabilities (change POLICY ACCESS TAG) when any of: file permissions change, a conflicting mandatory byte-range lock is granted, or a layout is revoked and reassigned to another client.

security

The OSD NOSEC security method MUST NOT be used for any I/O operation. Of the four T10 OSD security methods (NOSEC, CAPKEY, CMDRSP, ALLDATA), only non-NOSEC methods allow verification and enforcement of the metadata server's access control policy using layout credentials.

security

When a file's permissions or ACL change, the metadata server MUST change the POLICY ACCESS TAG (capability version attribute) on all objects comprising the file to implicitly invalidate outstanding capabilities before committing to the new access policy.

security

When client layouts are revoked (expired lease or recalled layouts not returned within a lease period), the metadata server MUST fence off the revoked clients from the respective objects by invalidating their OSD capabilities to avoid data corruption.

security

When performing I/O on behalf of local users, the pNFS client MUST authenticate and authorize each user by issuing OPEN and ACCESS calls to the metadata server before using OSD capabilities, analogous to NFSv4 data delegations.

security

pnfs_osd_data_map4 parameterizes data placement with: odm_num_comps (number of component objects), odm_stripe_unit (bytes per stripe unit), odm_group_width and odm_group_depth (nested striping parameters, both MUST be zero if no nesting), odm_mirror_cnt (replication count, MUST be 0 if no mirroring), and odm_raid_algorithm.

pnfs_osd_deviceaddr4 is the complete OSD device address: a discriminated union for the SCSI target (anonymous, SCSI name string, or SCSI device ID opaque), an optional netaddr4 for the network endpoint, oda_lun[8] (SAM-3 64-bit LUN), opaque oda_systemid<>, root object credentials oda_root_obj_cred, and opaque oda_osdname<>.

pnfs_osd_errno4 defines seven OSD I/O error codes for LAYOUTRETURN reporting: EIO (device failure/media error), NOT_FOUND (missing object), NO_SPACE, BAD_CRED (expired or revoked capability), NO_ACCESS (capability lacks permission), UNREACHABLE (communication failure), and RESOURCE (client-side local failure—MUST guarantee OSD command was never dispatched).

security

pnfs_osd_ioerr4 reports a single component I/O error to the metadata server: oer_component (failed object ID), oer_comp_offset and oer_comp_length (byte range within the component covering all failed I/O operations), oer_iswrite (true if the operation was data-modifying), and oer_errno.

pnfs_osd_layout4 is the LAYOUT4_OSD2_OBJECTS opaque body, containing a pnfs_osd_data_map4 (olo_map), an index into the full component array (olo_comps_index), and an array of object credentials (olo_components). The server MAY return a partial component array sufficient for the layout's data range.

pnfs_osd_layoutreturn4 is the LAYOUT4_OSD2_OBJECTS opaque body for LAYOUTRETURN, containing olr_ioerr_report<> (an array of pnfs_osd_ioerr4 elements). If no errors are reported, the array size is set to zero.

pnfs_osd_layoutupdate4 is the LAYOUT4_OSD2_OBJECTS opaque body for LAYOUTCOMMIT, containing olu_delta_space_used (optional int64_t delta of OSD-reported capacity consumed by writes) and olu_ioerr_flag (MUST be set to true if I/O errors were encountered, deferring attribute updates until LAYOUTRETURN error report is received).

pnfs_osd_object_cred4 is the per-component security credential containing the object ID, OSD protocol version, capability key security method (PNFS_OSD_CAP_KEY_SEC_NONE or PNFS_OSD_CAP_KEY_SEC_SSV), an opaque capability key, and an opaque capability. Together with oda_systemid from pnfs_osd_deviceaddr4, these provide OSD access credentials.

security, crypto

pnfs_osd_objid4 identifies an object using a three-level hierarchy: deviceid4 oid_device_id (selects the OSD from GETDEVICELIST/GETDEVICEINFO), uint64_t oid_partition_id (64-bit partition number within the OSD), and uint64_t oid_object_id (64-bit object number within the partition).

pnfs_osd_raid_algorithm4 is an enum of four data redundancy algorithms: PNFS_OSD_RAID_0 (no parity), PNFS_OSD_RAID_4 (fixed parity device with XOR), PNFS_OSD_RAID_5 (rotating parity with XOR), and PNFS_OSD_RAID_PQ (Reed-Solomon double-parity P+Q).

pnfs_osd_version4 is an enum indicating the OSD protocol version (PNFS_OSD_VERSION_1=1, PNFS_OSD_VERSION_2=2) or that the component is unavailable (PNFS_OSD_MISSING=0). RAID algorithms that support redundancy use this to determine which components are missing for reconstruction.

LAYOUT4_OSD2_OBJECTS (value 2) is the NFSv4.1 layout type number for the object-based pNFS layout, already assigned by IANA as part of NFSv4.1 processing. This document defines the associated protocol and requires no new IANA actions.

registry

Capabilities are revoked by changing the OSD POLICY ACCESS TAG attribute. An OSD rejects any request using a capability with a mismatched POLICY ACCESS TAG (ILLEGAL REQUEST sense). The client should issue LAYOUTCOMMIT with PNFS_OSD_ERR_BAD_CRED and optionally a compound LAYOUTRETURN/LAYOUTGET to obtain fresh capabilities.

security

If a LAYOUTGET reply containing CapKey and Cap is intercepted, it can be used to forge valid OSD requests within the capability's access restrictions. Privacy for the capability key can be achieved via RPCSEC_GSS privacy, SSV-encrypted oc_capability_key (GSS_Wrap with conf_req_flag=TRUE), VLAN isolation, or IPsec.

security, crypto, ipsec

The OSD security protocol uses a two-level MAC scheme: the metadata server signs capabilities as CapKey = MAC<SecretKey>(Cap, SystemID); clients sign each OSD request as ReqMAC = MAC<CapKey>(Req, ReqNonce). The OSD validates using its shared SecretKey, enforcing access policy without knowing user IDs or ACLs.

security, crypto

pnfs_osd_deviceaddr4 encodes the SCSI target as a discriminated union on pnfs_osd_targetid_type4 (OBJ_TARGET_ANON=void, OBJ_TARGET_SCSI_NAME=variable-length string, OBJ_TARGET_SCSI_DEVICE_ID=variable-length opaque); oda_lun is a fixed 8-byte opaque field; oda_systemid and oda_osdname are variable-length opaque.

pnfs_osd_objid4 is encoded in XDR as a struct with three fields: deviceid4 oid_device_id (opaque type defined in NFSv4.1 nfs4_prot.x), uint64_t oid_partition_id, and uint64_t oid_object_id.