Direct Data Placement over Reliable Transports
updated by
- rfc-7146 — Securing Block Storage Protocols over IP: RFC 3723 Requirements Update for IPsec v3
Extracted elements (28)
DDP's IPsec requirements are based on RFC 2401 (older IPsec) profiled by RFC 3723, rather than the newer RFC 4301, because an important early application is iSER (iSCSI over RDMA), and using a common IPsec profile across iSCSI and RDDP simplifies deployment.
DDP separates Data Placement from Data Delivery to allow out-of-order placement of DDP Segments directly into the final ULP Buffer, eliminating the traditional copy from a reassembly buffer. Delivery to the ULP still occurs in order, after all segments of a message and all prior messages are placed.
The Tagged Buffer Model allows a non-zero initial TO and re-use of the same STag across multiple consecutive ULP Messages, enabling ULPs such as RDMAP to implement their own segmentation and reassembly within a large advertised buffer without requiring DDP to recover ULP Message boundaries.
DDP is designed to be LLP-independent but was specifically created for the iWARP protocol suite layered over MPA/TCP or SCTP. When used with MPA over TCP, DDP Segments are framed by MPA; when used with SCTP, each DDP Stream maps directly to one SCTP stream.
A zero-length DDP Message is allowed and MUST consume exactly one DDP Segment. The STag and TO fields MUST NOT be checked for a zero-length Tagged DDP Message.
Any LLP serving as a lower layer to DDP MUST expose MULPDU and MULPDU changes so DDP can perform aligned segmentation. DDP MUST NOT be required to re-segment previously posted DDP Segments when the MULPDU changes.
At the Data Sink, DDP MUST Deliver a DDP Message to the ULP if and only if: the last DDP Segment had L=1, all segments have been placed, all preceding messages have been placed, and each preceding message has already been delivered.
At the Data Source, DDP MUST transmit DDP Messages in the order they were submitted and SHOULD transmit segments within a message in increasing MO order (Untagged) or increasing TO order (Tagged). At the Data Sink, DDP MUST deliver messages in the order they were sent.
At the Data Source, the DDP layer MUST segment ULP messages into DDP Segments no larger than the LLP MULPDU, and the ULP Message Length MUST be less than 2^32 octets.
DDP MUST NOT independently initiate Stream Teardown; it only responds to LLP teardown or ULP requests. After a fatal error is indicated to the ULP, the DDP Stream MUST NOT be terminated until the Local Peer ULP requests abortive teardown.
For non-zero-length Untagged DDP Segments, DDP MUST validate the QN, MSN, MO, and payload range against the associated buffer before Placement. For non-zero-length Tagged segments, it MUST validate STag, TO range, and that the 64-bit sum of payload length and TO does not wrap.
If an LLP does not support independent LLP Stream teardown, and a DDP error occurs on a specific DDP Stream, the LLP MUST label that stream as erroneous and MUST NOT allow further data transfer on it after DDP requests teardown.
IPsec ESP (RFC 2406) MUST be supported with per-packet data origin authentication, integrity, and replay protection. ESP in tunnel mode MUST be supported; IKE (RFC 2409) MUST be supported for peer authentication and key management, following the profile in RFC 3723 Section 2.3.
The DV field MUST be set to one (version 1) and MUST be the same value for all DDP Segments transmitted or received on a DDP Stream.
The L (Last) flag MUST be set to one on the last DDP Segment of every DDP Message and MUST NOT be set on any other DDP Segment. The segment with L=1 MUST be posted to the LLP after all other segments of the message.
The LLP MAY deliver incoming segments out of order for placement, but if it does, it MUST also provide information specifying the sender-specified order. The LLP MUST also provide the length of each DDP Segment received.
The LLP MUST preserve DDP Segment and Message boundaries at the Data Sink, and MUST provide a strong digest (at least equivalent to CRC32-C) covering at least the DDP Segment.
The MSN MUST be initialized to one and MUST increment by one (modulo 2^32) with each DDP Message targeting the specific Queue Number on the DDP Stream. The MSN MUST wrap to 0 after 0xFFFFFFFF.
DDP Error Numbers are structured as a 4-bit Error Type and 8-bit Error Code. Types include 0x0 (Local Catastrophic), 0x1 (Tagged Buffer Errors: invalid STag, base/bounds violation, STag not associated with stream, TO wrap, invalid DDP version), and 0x2 (Untagged Buffer Errors: invalid QN/MSN/MO, message too long, invalid version).
DDP supports two buffer models: the Tagged Buffer Model, where the Data Sink advertises a named buffer (STag) to the Data Source for remote placement at arbitrary offsets, and the Untagged Buffer Model, where the Data Sink queues anonymous receive buffers consumed in order by incoming DDP Messages.
DDP directly exposes memory addresses on the wire via the STag and TO fields. All Placement information MUST be validated (STag validity, octet-level base and bounds checks) before any data is placed to prevent a third party from injecting spoofed DDP Segments that corrupt Data Sink memory.
RNIC security requirements include: ensuring a DDP Stream cannot access STags in a different Protection Domain, enforcing stream-scoped STag restrictions, preventing remote access outside the registered buffer bounds, and providing ULP mechanisms to establish and revoke STag validity and TO ranges.
STag access control requires two mandatory association mechanisms: Protection Domain (PD) association, which restricts STag use to DDP Streams with the same PD ID, and DDP Stream association, which binds an STag to a specific locally-identified DDP Stream. DDP MUST support both mechanisms.
The Privileged Resource Manager (PRM) MUST proxy all non-privileged ULP interactions with the RNIC that could affect other ULPs, MUST verify a ULP's right to access a Data Buffer before associating it with an STag, and SHOULD prevent any Local Peer from consuming more than its fair share of resources.
DDP Stream teardown has two paths: Graceful Teardown (all in-progress DDP Messages complete before the LLP Stream is closed; further ULP requests MUST return error) and Abortive Teardown (triggered by LLP connection/stream loss, LLP fatal error, or invalid DDP Header fields).
The DDP Control Field is an 8-bit field at the start of every DDP Header containing: T (Tagged flag, 1 bit), L (Last flag, 1 bit), Rsvd (4 bits, must be zero on transmit), and DV (DDP version, 2 bits, must be set to 1).
The Tagged Buffer DDP Header contains the 8-bit DDP Control Field (T=1), an 8-bit RsvdULP field, a 32-bit STag (Steering Tag), and a 64-bit TO (Tagged Offset) indicating the byte offset within the destination buffer.
The Untagged Buffer DDP Header contains the 8-bit DDP Control Field (T=0), a 40-bit RsvdULP field, a 32-bit QN (Queue Number), a 32-bit MSN (Message Sequence Number), and a 32-bit MO (Message Offset) in bytes from the start of the DDP Message.