Internet Small Computer System Interface (iSCSI) Extensions for Remote Direct Memory Access (RDMA)
obsoleted by
- rfc-7145 — Internet Small Computer System Interface (iSCSI) Extensions for the Remote Direct Memory Access (RDMA) Specification
updated by
- rfc-7146 — Securing Block Storage Protocols over IP: RFC 3723 Requirements Update for IPsec v3
Extracted elements (32)
HeaderDigest and DataDigest are forced to None for iSER sessions because RCaP (e.g., iWARP's MPA layer with CRC, or InfiniBand Reliable Connection mode) provides equivalent or better error detection for all iSER messages, and iSCSI digests cannot apply to the RDMA Write/Read operations used for SCSI data movement.
iSER is designed to require no changes to the iSCSI ecosystem (MIB, bootstrapping, naming, discovery, security, authentication) and no iSCSI- or iSER-specific hardware in the RDMA controller, so that any generic RDMA-Capable Controller can be used and existing iSCSI infrastructure is fully reused.
The target drives all data transfer (RDMA Write for SCSI reads, RDMA Read for SCSI writes) rather than the initiator, because the target knows when data is ready and can directly address the advertised initiator buffers. The initiator only Advertises its I/O Buffers via STags in the SCSI Command PDU iSER header.
If RDMAExtensions=Yes is not negotiated or negotiation fails, the connection MUST fall back to Traditional iSCSI mode per RFC 3720. For InfiniBand, this may require re-establishing the connection as an IPoIB connection since InfiniBand connections may not natively support TCP byte-stream mode.
iSER requires that OFMarker and IFMarker be negotiated to No (Section 6.6), since iSCSI markers are a TCP-specific mechanism for framing PDUs in a byte stream and have no meaning in the RDMA message-based transport used by iSER.
After the connection transitions to iSER-assisted mode, the first iSER Message sent by the initiator MUST be the iSER Hello Message, and the target MUST return the iSER HelloReply Message as its first iSER Message.
iSER-assisted mode is a session-wide attribute. If RDMAExtensions=Yes is negotiated on the leading connection, all subsequent connections of that session MUST enable iSER-assisted mode without re-exchanging the RDMAExtensions key. Conversely, if it is not negotiated on the leading connection, the key MUST NOT be used on any additional connections.
The iSCSI layer at the initiator MUST NOT issue SNACKs for PDUs when operating in iSER-assisted mode, since the RDMA-based data transfer model does not support the SNACK retransmission mechanism.
The iSCSI layer at the initiator MUST set ExpDataSN to 0 in Task Management Function Requests for Task Allegiance Reassignment for read/bidirectional commands, causing the target to retransmit all unacknowledged read data.
The iSCSI layer at the target MUST always return the SCSI status in a separate SCSI Response PDU for read commands; there MUST NOT be a phase collapse when concluding a SCSI read command in iSER-assisted mode.
The iSER layer at the initiator MUST set IRD (Inbound RDMA Read Queue Depth) to 1 or higher if R2Ts are to be used on the connection. The iSER-IRD value SHOULD be set to the IRD value and MUST NOT exceed the IRD value.
The iSER layer at the target MUST send the final Login Response PDU in native transport mode (byte stream for TCP) before transitioning to iSER-assisted mode; for iWARP, sending the final Login Response PDU and enabling RCaP MUST be performed atomically to avoid race conditions.
The iSER layer at the target MUST set iSER-ORD to the minimum of the target's own iSER-ORD value and the iSER-IRD value declared by the initiator in the iSER Hello Message.
The RDMAExtensions key MUST NOT be negotiated for a Discovery session; Discovery sessions are always conducted using the transport layer as described in RFC 3720.
The same connection MUST be used for both the iSCSI Login Phase and the subsequent iSER-assisted Full Feature Phase; a separate connection cannot be established for the Full Feature Phase.
To enable iSER-assisted mode, both the initiator and target MUST exchange RDMAExtensions=Yes on the leading connection during the LoginOperationalNegotiation stage. If negotiation fails or either party offers No, the connection MUST continue with standard iSCSI semantics per RFC 3720.
To minimize denial-of-service risk, the iSCSI layer MUST NOT request iSER layer connection resource allocation (Allocate_Connection_Resources) until after the SecurityNegotiation stage completes; resource allocation must be deferred to the LoginOperationalNegotiation stage.
Upon connection termination, the iSER layer MUST deallocate all connection and task resources and invalidate Local and Remote Mappings (associating ITTs to local and Advertised STags) before notifying the iSCSI layer via Connection_Terminate_Notify.
When RDMAExtensions=Yes is negotiated, HeaderDigest and DataDigest MUST be negotiated to None on all iSCSI/iSER connections in that session. RCaP provides error detection at least as good as a 32-bit CRC, making iSCSI digests unnecessary and inapplicable to RDMA operations.
A Steering Tag (STag) is the identifier of an I/O Buffer on an RDMA-Capable Controller that the initiator Advertises to the target by embedding it in the iSER header of a SendSE Message containing the SCSI Command PDU. The base Tagged Offset is always implicitly zero; the buffer length is taken from the SCSI Command PDU.
MaxOutstandingUnexpectedPDUs (Section 6.7) limits the number of iSCSI control-type PDUs that can be outstanding in the RDMA Send flow without prior credit grants, providing flow control for the Send Message path.
The Datamover Interface (DI) defines abstract Operational Primitives between the iSCSI and iSER layers: Send_Control, Put_Data, Get_Data, Allocate_Connection_Resources, Deallocate_Connection_Resources, Enable_Datamover, Connection_Terminate, Notice_Key_Values, and Deallocate_Task_Resources are offered by iSER; Control_Notify, Data_Completion_Notify, Data_ACK_Notify, and Connection_Terminate_Notify are used by iSER.
The iSER Hello/HelloReply exchange is the mandatory first exchange after a connection enters iSER-assisted mode. The Hello Message declares the initiator's iSER-IRD; the HelloReply declares the target's iSER-ORD. A negotiation failure occurs if iSER-IRD > 0 but iSER-ORD = 0, or if supported protocol versions do not overlap.
The RDMAExtensions key (Section 6.3) is a session-wide, leading-only boolean key negotiated during LoginOperationalNegotiation with a default of No and AND result function. Both initiator and target must offer it in the first permissible Login PDU of the leading connection's LoginOperationalNegotiation stage.
The TargetRecvDataSegmentLength and InitiatorRecvDataSegmentLength keys replace MaxRecvDataSegmentLength for iSER sessions, defining the maximum RDMA Send message size each side can receive. These govern the size of iSCSI control-type PDU transfers only, since SCSI data is moved via RDMA Write/Read.
An Advertised STag is automatically invalidated upon normal task completion via the SendInvSE Message carrying the SCSI Response PDU. For bidirectional commands and abnormal completions, the initiator is required to explicitly invalidate STags. The initiator must always sanity-check the invalidation to prevent unauthorized remote buffer access.
To prevent denial-of-service attacks, the iSCSI layer at both initiator and target MUST defer RDMA connection resource allocation (Allocate_Connection_Resources) until after the SecurityNegotiation stage of iSCSI Login completes, ensuring resources are only consumed for authenticated peers.
An iSCSI/iSER connection progresses through: (1) TCP/transport connection established; (2) iSCSI Login Phase in byte-stream mode with RDMAExtensions negotiation; (3) Allocate_Connection_Resources invoked; (4) Enable_Datamover triggers RCaP mode; (5) iSER Hello/HelloReply exchange; (6) Full Feature Phase in iSER-assisted mode. Termination reverses resource allocation and invalidates all STags.
The iSER Header for an iSCSI Control-Type PDU (Section 9.2, Figure 3) includes the iSER opcode for control PDUs, a read STag and write STag advertising the initiator's I/O buffers for SCSI read and write data respectively, and is followed immediately by the iSCSI PDU BHS.
The iSER Header (Section 9.1, Figure 2) is an 8-byte structure prepended to iSCSI PDUs sent over RDMA Send messages. It carries an opcode field identifying the message type (control PDU, Hello, or HelloReply) and STag fields advertising I/O buffers. No length field is present in the iSER header because the RCaP layer supplies the message length.
The iSER Hello Message header (Section 9.3, Figure 4) carries the iSER protocol version range supported by the initiator and the iSER-IRD value. It is encapsulated in a SendSE RDMA message and is the mandatory first message sent by the initiator after transitioning to iSER-assisted mode.
The iSER HelloReply Message header (Section 9.4, Figure 5) carries the negotiated iSER protocol version and the target's iSER-ORD value. It is encapsulated in a SendSE RDMA message and is the mandatory first message sent by the target after receiving the Hello.