A Remote Direct Memory Access Protocol Specification

Buffer Advertisement — informing the Remote Peer of a Tagged Buffer's STag, base address, and length — is deliberately left outside the scope of RDMAP and delegated to the ULP. A typical method embeds the Tagged Buffer identifiers in a Send Message to the peer.

Buffer contents at the Data Sink are indeterminate until the associated Delivery event: for Send until Delivered to ULP; for RDMA Write until a subsequent Send is Delivered; for RDMA Read until the Read Response is Delivered. Applications MUST NOT 'peek' at buffer contents mid-operation since out-of-order Placement can leave intermediate indeterminate state.

RDMAP explicitly separates Placement (writing data to a buffer by DDP, may occur out of order) from Delivery (informing the ULP a Message is available, strictly ordered). This separation allows DDP to directly place out-of-order segments into their final destination without copy operations, while preserving ordered semantics for the application.

The RDMA Read operation sends the Request in the Untagged Buffer direction but returns the Response using the Tagged Buffer Model directly into the Data Sink's pre-Advertised buffer, with no ULP involvement at the Data Source. This enables zero-copy remote memory reads with minimal CPU overhead at the source.

RDMAP version 00b (RDMA Consortium specification) and version 01b (this RFC 5040 specification) are both valid version values in the RDMA Version field. Interoperability between implementations of different versions depends on MPA protocol version negotiation such as MPA marker and MPA CRC settings.

A valid RDMA Write Message MUST NOT be delivered to the Data Sink's ULP; it is placed directly into the Tagged Buffer by the DDP layer. The Data Source RDMAP layer MUST issue RDMA Write Messages to DDP in the order submitted by the ULP, and a single RDMA Write Message MUST only write to one remote Tagged Buffer.

After sending a Terminate Message on an RDMAP Stream, the Local Peer MUST NOT send any more Messages on that stream. The Terminate Message MUST reference Queue Number 2, and MUST be Delivered to the Remote Peer's RDMAP layer, which MUST then Deliver it to the ULP.

All RDMA Messages MUST set the RDMA Version field (bits 24-25) to 01b for compliance with this specification. Version 00b (RDMA Consortium) is also valid; interoperability between implementations of different versions depends on MPA protocol version negotiation.

If the Data Source receives a valid RDMA Read Request, it MUST respond with a valid RDMA Read Response. If the RDMA Read Message Size field is zero, the Data Source MUST NOT validate the STag and Tagged Offset, and MUST respond with a zero-length RDMA Read Response Message.

RDMA Read Request Messages MUST NOT start processing at the Remote Peer until they have been Delivered to RDMAP by DDP, ensuring that a Read following a Send or RDMA Write to the same buffer will return the data written by that prior operation rather than stale data.

RDMAP MUST be layered on top of the Direct Data Placement Protocol (DDP). This is the foundational architectural constraint of the entire iWARP protocol stack.

Send Messages MUST be Delivered to the ULP at the Remote Peer in the order they were submitted at the Local Peer. RDMA Read Response Messages MUST be Delivered to the ULP in the order that RDMA Read Request Messages were received by the Data Source RDMAP layer.

The Remote Peer MUST process RDMA Read Request Messages in the order they were sent. A single RDMA Read Request Message MUST only read from one remote Tagged Buffer. The Data Source MUST ensure sufficient Untagged Buffers on Queue Number 1 to support the ULP-negotiated maximum number of concurrent RDMA Read Requests.

The RsvdULP field provided to RDMAP by DDP MUST be 1 octet for DDP Tagged Messages and 5 octets for DDP Untagged Messages, accommodating the 1-octet RDMAP Control Field and the 4-octet Invalidate STag field present on Untagged Messages carrying Send with Invalidate operations.

When Send with Invalidate or Send with Solicited Event and Invalidate Messages are Delivered, the RDMAP layer MUST verify the STag is associated with the RDMAP Stream and then invalidate it; if the STag is not associated with the RDMAP Stream, RDMAP MUST issue a Terminate Message with the 'STag Cannot be Invalidated' error code.

RDMA OpCodes are 4-bit values assigned per message type: 0000b=RDMA Write (Tagged), 0001b=RDMA Read Request (Untagged/QN1), 0010b=RDMA Read Response (Tagged), 0011b=Send (Untagged/QN0), 0100b=Send with Invalidate, 0101b=Send with SE, 0110b=Send with SE and Invalidate, 0111b=Terminate (Untagged/QN2); 1000b-1111b are Reserved.

RDMAP defines seven RDMA operations: RDMA Write, RDMA Read, Send, Send with Invalidate, Send with Solicited Event, Send with Solicited Event and Invalidate, and Terminate. All except RDMA Read generate exactly one RDMA Message; RDMA Read generates one Request and one Response.

RDMAP uses three DDP Untagged Buffer queues: Queue Number 0 for Send operations, Queue Number 1 for RDMA Read Request operations, and Queue Number 2 for Terminate operations. Tagged operations (RDMA Write, RDMA Read Response) use the DDP Tagged Buffer Model.

The STag Invalidation mechanism prevents a Remote Peer from accessing a previously Advertised Tagged Buffer after it has been invalidated. Invalidation is triggered by Send with Invalidate or Send with Solicited Event and Invalidate Messages, or directly by the Local ULP; the buffer cannot be accessed remotely again until explicitly re-Advertised.

The Terminate Message carries structured error diagnostics including the offending DDP Segment Length, the Terminated DDP Header (if available), and for RDMA Read Request errors, an updated Terminated RDMA Header reflecting the current Data Sink and Data Source Tagged Offsets and remaining data size at the time of termination.

IPsec is identified as the primary security service for RDMAP, providing authentication, integrity, and confidentiality for iWARP traffic. A Privileged Resource Manager must control STag registration and buffer binding to prevent unauthorized remote memory access; kernel bypass implementations must ensure ULP applications cannot subvert STag protection.

security, ipsec

The RNIC (RDMAP layer) MUST enforce STag protection: validating STags, checking base and bounds on Tagged Offsets, enforcing access rights (read/write), verifying STag-to-RDMAP-Stream association, and reporting violations via Terminate Messages with Remote Protection Error type codes including Invalid STag, Base/Bounds Violation, Access Rights Violation, and TO Wrap.

security

RDMAP Stream lifecycle has two teardown modes: graceful termination (RDMAP Graceful Termination, Section 6.2) allows all in-progress and pending RDMA Operations to complete, while abortive termination (Section 6.2.1) closes the RDMAP Stream immediately via a Terminate Message without completing pending operations.

The RDMA Read Request Header immediately follows the DDP header and contains: Data Sink STag (32 bits), Data Sink Tagged Offset (64 bits), RDMA Read Message Size (32 bits), Data Source STag (32 bits), and Data Source Tagged Offset (64 bits). Total: 224 bits.

The RDMAP Control Field occupies the 8-bit RsvdULP field of every DDP message. Bits 24-25 carry the RDMA Version (01b for this spec, 00b for RDMA Consortium), bits 26-27 are reserved (MUST be zero on transmit), and bits 28-31 carry the 4-bit OpCode. For Send with Invalidate messages, bits 32-63 carry the Invalidate STag.

The Terminate Control field is 19 bits structured as: Layer (4 bits; 0000b=RDMA, 0001b=DDP, 0010b=LLP), EType (4 bits; 0000b=Local Catastrophic, 0001b=Remote Protection, 0010b=Remote Operation), Error Code (8 bits), and HdrCt (3 bits: M=DDP Segment Length valid, D=DDP Header included, R=RDMA Header included).

The Terminate Header contains a Terminate Control field (19 bits), Reserved (13 bits, MUST be zero), optional DDP Segment Length (16 bits, valid when M bit set), optional Terminated DDP Header (112 bits for Tagged, 144 bits for Untagged), and optional Terminated RDMA Header (224 bits, present only for RDMA Read Request errors).