Remote Direct Memory Access (RDMA) Protocol Extensions
Extracted elements (31)
Atomic Operation Request messages reuse Queue Number 1 (the RDMA Read Request Queue) so that Atomic Operations can share the existing ORD/IRD flow-control infrastructure already defined for RDMA Read Requests, avoiding a new flow-control mechanism.
Because adapter query interfaces return the scope of atomicity guarantees rather than individual Atomic Operation support, an RNIC that supports any Atomic Operation must support all Atomic Operations defined in this specification. This avoids per-operation probing by applications.
The extensions are motivated by feature parity with InfiniBand: RSockets cannot use RDMA Write with Immediate Data over iWARP, and DAT Atomics primitives available for InfiniBand are absent in iWARP RNICs. Adding Atomic Operations and Immediate Data closes this gap so middleware like RSockets and DAT Atomics can operate interchangeably over InfiniBand and iWARP.
A FetchAdd operation with 'Add Data' set to 0 can safely probe whether a remote peer supports Atomic Operations without modifying the contents of its memory. A Remote Operation Error or Unexpected OpCode error from the remote peer indicates Atomic Operations are unsupported.
It is RECOMMENDED that RDMA applications and ULPs negotiate the use of Immediate Data and Atomic Operations at the application or ULP level before using them. Existing applications SHOULD NOT assume these extensions are supported; in the absence of negotiation, errors returned by the remote peer can be used to probe capability.
The Immediate Data operation combined with a preceding RDMA Write produces the equivalent of the RDMA Write with Immediate Data operation found in InfiniBand, allowing applications such as RSockets to use the same code path over both iWARP and InfiniBand.
A valid Atomic Operation Request Message received by the Responder MUST be processed by the RDMAP layer and MUST NOT be delivered to the Responder's ULP. The Responder MUST reply with a valid Atomic Operation Response Message for every valid request received.
All new RDMA Messages defined in this specification MUST set the RDMA Version (RV) field to 01b and the Invalidate STag to zero; receivers MUST ignore the Invalidate STag field.
An Immediate Data or Immediate Data with SE Message MUST reference an Untagged Buffer (QN=0), MUST consume exactly one Untagged Buffer, and the DDP layer MUST transmit exactly 8 bytes of data. The Remote Peer MUST check that exactly 8 bytes are present.
An RNIC that supports Atomic Operations MUST implement both the FetchAdd operation and the CmpSwap operation; partial support is not permitted.
Atomic Operation Response messages MUST set Queue Number (QN) to 3 in the DDP header. Atomic Operation Request messages MUST use Queue Number 1 (the RDMA Read Request Queue).
Atomicity guarantees MUST be provided within the scope of a single RNIC. Atomicity with respect to access via other RNICs or local processor memory accesses to the same Tagged Buffer is OPTIONAL.
CmpSwap Atomic Operations MUST target ULP Buffer addresses that are 64-bit aligned. If misaligned, the operation MUST NOT be performed, the Responder's memory MUST NOT be modified, the result MUST be surfaced as an error, and a Terminate Message MUST be generated.
FetchAdd Atomic Operations MUST target ULP Buffer addresses that are 64-bit aligned. A misaligned target MUST be surfaced as an error, the Responder's memory MUST NOT be modified, and a Terminate Message MUST be generated.
Immediate Data and Immediate Data with SE Messages MUST be delivered to the Remote Peer's ULP in the order they were sent, and the Local Peer MUST issue them in the order submitted by the ULP.
On incoming Atomic requests, the RDMAP layer MUST validate that the Remote Peer's Tagged ULP Buffer address is 64-bit aligned; on failure, a Terminate Message indicating RDMA Layer Remote Operation Error with Error Code 'Catastrophic error, localized to RDMAP Stream' MUST be generated.
The Local Peer MUST send a Terminate Message when errors are detected while creating or receiving Atomic Request, Atomic Response, Immediate Data, or Immediate Data with SE messages. Error type, error code, and terminated DDP header fields are included as appropriate.
The Requester MUST NOT consider the Responder's Tagged Buffer modified by an Atomic Operation until the Atomic Operation Response Message has been Delivered to RDMAP at the Requester. An Atomic Operation MUST NOT be Completed at the Requester until DDP delivers the associated Atomic Operation Response.
The Responder MUST process Atomic Operation Request Messages in the order they were sent, and MUST submit Atomic Operation Response Messages to the DDP layer in the same order. The Requester MUST pass Atomic Operation Request Messages to DDP in the order submitted by the ULP.
The Responder RNIC MUST implement mechanisms to prevent concurrent Atomic Operations to a memory location registered for Atomic Operations while an in-progress Atomic Operation is targeting that location, ensuring the read-modify-write is not interleaved by other RDMAP Streams on the same RNIC.
Four new RDMA Message opcodes are defined: 1000b (Immediate Data), 1001b (Immediate Data with SE), 1010b (Atomic Request), and 1011b (Atomic Response). All use the Untagged Buffer model; STag and Tagged Offset are not applicable. Immediate Data operations use QN=0; Atomic Requests use QN=1; Atomic Responses use QN=3.
The CmpSwap Atomic Operation reads a 64-bit value at a 64-bit-aligned ULP Buffer address, ANDs it with a Compare Mask and compares the result to (Compare Data AND Compare Mask). If equal, masked bits are replaced with Swap Data using Swap Mask; otherwise memory is unchanged. The original value is always returned to the Requester.
The FetchAdd Atomic Operation reads a 64-bit value at a 64-bit-aligned ULP Buffer address on the Responder, adds a 64-bit 'Add Data' value subject to a 64-bit 'Add Mask' that defines independent sub-field boundaries for carry isolation, writes the result back, and returns the original value to the Requester.
A new 'RDMAP DDP Untagged Queue Numbers' registry is created under 'Remote Direct Data Placement (RDDP)'. Values are 32-bit codes assigned by Standards Action. Initial contents: 0x0 (Send operations), 0x1 (RDMA Read Request), 0x2 (Terminate), 0x3 (Atomic Response, this specification). Sequential assignment is recommended for hardware compatibility.
A new 'RDMAP Message Atomic Operation Subcodes' registry is created under 'Remote Direct Data Placement (RDDP)'. Values are 4-bit codes assigned by Standards Action. Initial contents: 0x0 (FetchAdd), 0x1 (Reserved), 0x2 (CmpSwap). New codes should be assigned sequentially to support hardware implementations.
Four new entries are added to the existing 'RDMAP Message Operation Codes' registry under 'Remote Direct Data Placement (RDDP)': 0x8 (Immediate Data), 0x9 (Immediate Data with Solicited Event), 0xA (Atomic Request), and 0xB (Atomic Response), all referencing this specification.
RDMAP traffic from structured HPC applications (e.g., MPI) exhibits distinctive patterns in message timing, source/destination, and size that can reveal sensitive information (application type, dataset size, progress) even when encrypted. ESPv3 Traffic Flow Confidentiality (RFC 4303) should be used to obfuscate these characteristics; ESPv3 requirements for RDMAP are specified in RFC 7146.
The Security Considerations of RFC 5040 apply in full. Atomic Operations use the same ULP Buffer address-based remote buffer addressing as defined in RFC 5040 and are subject to the security model in RFC 5042, which requires that Tagged Buffer access be authorized through the STag mechanism.
The Atomic Operation Request Header (52 bytes, immediately following the DDP header) contains: Reserved/Not Used (28 bits), AOpCode (4 bits), Request Identifier (32 bits), Remote STag (32 bits), Remote Tagged Offset (64 bits), Add or Swap Data (64 bits), Add or Swap Mask (64 bits), Compare Data (64 bits), and Compare Mask (64 bits). All fields are big-endian on the wire.
The Atomic Operation Response Header (32 bytes, immediately following the DDP header) contains: Original Request Identifier (32 bits) echoing the Request Identifier from the corresponding request, and Original Remote Data Value (64 bits) holding the pre-operation value read from the Responder's ULP Buffer.
The Immediate Data Header is 64 bits (8 bytes) of opaque ULP data placed immediately after the DDP header. Both Immediate Data (opcode 1000b) and Immediate Data with SE (opcode 1001b) carry exactly this 8-byte payload and no additional ULP message payload.