IP over InfiniBand: Connected Mode
Extracted elements (25)
Because interfaces on the same subnet may support different link MTUs (UD vs. connected mode), implementations may define a large IP MTU reduced per-destination, storing the relevant MTU in a route cache or neighbor cache entry.
IPoIB-CM uses an existing IPoIB-UD QP for multicast/broadcast rather than emulating multicast over connected mode, because every IPoIB implementation is required to support UD mode, avoiding the need for a separate multicast emulation mechanism.
RC mode is designed for fabrics with short latencies. Its use over fabrics with long or highly variable latency is a concern because RC lacks measurement-based reliable transmission, potentially causing problems for congestion-aware traffic.
The RC mode of InfiniBand guarantees in-order delivery but may cause TCP to enter slow-start: an upper transport layer can time out while the RC layer is still retransmitting a complete message, making the RC layer's retransmission invisible to TCP's congestion control.
All address resolution responses (ARP or Neighbor Discovery) MUST always be encapsulated in a UD mode packet, regardless of whether connected modes are also supported.
Every CM message exchanged during IB connection setup MUST include the sender's IPoIB-UD QPN in the 'private data' field, including all REJ messages, so the receiving node can determine the initiating node's link address.
Every IPoIB-CM interface MUST have two sets of QPs: one unreliable datagram QP and one or more connected mode QPs.
Every IPoIB implementation compliant with this document MUST accept all IP unicast transmissions over any of the IPoIB modes it supports (RC, UC, or UD).
Every IPoIB implementation MUST support RFC4391 (IPoIB-UD) and MAY support the connected-mode extensions described in this document. IPoIB-CM is an OPTIONAL extension to IPoIB-UD.
Future uses of the reserved bits and octets in the link-layer address, Service-ID, and Private-Data Format MUST be published as RFCs. Reserved bits MUST be set to zero on sends.
IB connections created through IPoIB-CM SHOULD be torn down when the IPoIB interfaces they are associated with are torn down. Connections MAY also be torn down when the address resolution entry expires.
If an implementation only supports IPoIB-UD (not IPoIB-CM), it MUST ignore the Flags field on reception and MUST set the Flags octet to all zeros on transmission.
Multicast mapping, transmission, reception of multicast packets, and multicast routing MUST use the UD QP associated with the IPoIB interface, since connected modes do not support multicasting.
The link MTU MUST be able to support the minimum MTU required by the protocols using the IB connection. The default IPoIB-CM MTU is 2044 octets (2048-octet IPoIB-link MTU minus the 4-octet encapsulation header).
The node MUST keep a record of IB connections it is participating in. A node MAY attempt another connection to a remote peer using the same Service-ID as an existing connection; the receiver MAY drop such a request.
The private data field in the CM connection setup message (REQ) MUST include the Receive MTU, indicating the maximum packet size the requester can accept. The requester MUST also be able to accept smaller MTU sizes.
To prevent duplicate connections when REQ messages cross, MUST compare link-layer addresses (zeroing out connection mode flags): if the local address is numerically smaller, accept the connection; otherwise reject it with a 'Consumer Reject' error code.
IPoIB-CM extends the IPoIB link-layer address format from '<1 octet reserved>:QP:GID' to '<Flags>:QPN:GID', where the Flags octet indicates connected modes supported (RC in bit 0, UC in bit 1).
The IPoIB Service-ID is a 64-bit value with the format: 0x01 | Type (0x00) | Reserved (zeros) | UD QPN. The Type field MUST be 0, Reserved MUST be zeros, and QPN MUST be the UD QP exchanged during address resolution.
The InfiniBand specification defines a 64-bit block of Service-IDs (starting with 0x01) for IETF use. The definition and management of this block is delegated to the IETF. IPoIB Service-IDs use a specific format within this block.
An impostor may return a false set of flags to an IPoIB interface, causing unnecessary connection attempts and delay or disruption in IPoIB communication. Similarly, wrong or spurious QPN values provided during address resolution broadcast/multicast can cause disruption.
IB connection setup follows a three-way handshake: initiator sends REQ, peer responds with REP (or REJ to reject), and initiator completes with RTA (Ready To Acknowledge) or REJ. Either side may reject at any step.
All IP datagrams over IPoIB-CM are prefixed with a 4-octet encapsulation header consisting of a 16-bit Type field (0x0800 for IPv4, 0x86DD for IPv6) and a 16-bit Reserved field, as defined in RFC4391.
The Flags field is a single octet where bit 0 indicates RC support and bit 1 indicates UC support; all other bits are reserved and MUST be set to 0 on transmit and ignored on receive. Format: +RC|UC|0|0|0|0|0|0+.
The private data field in every CM connection establishment message is 8 octets: 1 octet Reserved (zero), 3 octets UD QPN, and 4 octets Receive MTU. Reserved MUST be set to zero on transmit and ignored on receive.