Bidirectional Remote Procedure Call on RPC-over-RDMA Transports
Extracted elements (23)
Bidirectional operation is described as an extension of RPC transport connection sharing, enabling two endpoints to exchange independent RPC messages in opposite directions over a shared connection without modifying the RPC-over-RDMA version 1 protocol. This avoids the need for a separate backchannel connection.
Forward- and reverse-direction credits are accounted separately to prevent head-of-line blocking: if one direction stalls, the other can continue unimpeded. This is a deliberate choice over a unified credit pool.
Reverse-direction chunk support is made optional because some ULPs have no DDP-eligible data items or can restrict message sizes natively, and omitting chunk support avoids the added complexity of performing RDMA Reads and Writes in the reverse direction.
NFS version 4.0 over RDMA forechannel typically uses a separate TCP-based backchannel for callbacks because the RDMA transport itself does not inherently support reverse-direction operation. The client advertises a 'tcp' or 'tcp6' netid for its callback service via SETCLIENTID.
NFS version 4.1 clients and servers signal backchannel capability via the CREATE_SESSION or BIND_CONN_TO_SESSION operations. Without a backchannel, NFS version 4.1 servers do not grant delegations and pNFS operation is not possible.
Without a reverse-direction RPC-over-RDMA capability, NFS version 4.1 clients must open an independent TCP socket for the backchannel. With bidirectional RPC-over-RDMA, forechannel and backchannel can share a single transport connection, simplifying connection management and improving scalability.
A ULP MUST NOT perform reverse-direction ONC RPC operations until the peer has indicated it is prepared to handle them, to prevent connection loss from sending reverse-direction messages to an unprepared endpoint.
Forward-direction Requesters SHOULD maintain a transport connection as long as there is the possibility that the connection peer can send reverse-direction requests, to avoid leaving the reverse-direction Requester unable to retransmit.
If a reverse-direction requester provides a non-empty chunk list to a Responder that does not support chunks, the Responder MUST reply with an RDMA_ERROR message with the rdma_err field set to ERR_CHUNK.
The rdma_vers field MUST contain the same value in a reverse-direction Reply message as in the matching Call message.
When message direction is not fully determined by context or by an accompanying RPC message payload with a call direction field, the receiver MUST ignore the header's credit value.
Whether presented inline or as a separate chunk, the ONC RPC Reply header MUST start with the same XID value present in the RPC-over-RDMA header, and the RPC header's msg_type field MUST contain the value REPLY.
Bidirectional RPC-over-RDMA operation allows both endpoints to act as Requester and Responder concurrently on a single transport connection, without changes to the RPC-over-RDMA version 1 protocol. Forward-direction and reverse-direction RPC transactions can coexist on the same connection.
Chunks MAY be used in the reverse direction and operate the same way as in the forward direction. Implementations supporting only ULPs without DDP-eligible data may choose not to implement chunk support in the reverse direction, in which case reverse-direction messages are always Short Messages.
Forward- and reverse-direction operations on the same connection share the same Receive buffers, so inline threshold values are identical in both directions. The call inline threshold for the reverse direction equals the reply inline threshold for the forward direction, and vice versa.
Forward-direction and reverse-direction credits on the same connection are accounted separately to prevent head-of-line blocking in one direction from impacting the other. The reverse-direction credit value is the number of RPC requests the reverse-direction Responder (the ONC RPC client) is prepared to receive concurrently.
Forward-direction operation refers to the traditional model where the ONC RPC client acts as Requester and the ONC RPC service acts as Responder. Reverse-direction operation inverts this: the ONC RPC service acts as Requester and the ONC RPC client acts as Responder.
Reverse-direction retransmission is asymmetric: since the ONC RPC server (reverse-direction Requester) cannot initiate transport connections, it must wait for the client to re-establish a connection before retransmitting. A waiting reverse-direction Call may time out to avoid indefinite blocking.
To receive incoming reverse-direction Calls, an RPC-over-RDMA client endpoint posts enough additional Receive buffers to match its advertised reverse-direction credit value, plus one per outstanding forward-direction RPC. On reconnect, it re-posts buffers for retransmitted forward Calls and for reverse-direction Calls.
To receive incoming reverse-direction Replies, an RPC-over-RDMA server endpoint posts enough additional Receive buffers to handle replies for each reverse-direction Call it sends. On reconnect, it re-posts buffers for retransmitted reverse Calls and for incoming forward-direction Calls.
XID values in bidirectional operation are generated independently on each host. A forward-direction Requester MAY use the same XID value simultaneously as a reverse-direction Requester on the same connection; such concurrent requests represent distinct ONC RPC transactions.
If an endpoint is not prepared to receive reverse-direction messages, loss of the RDMA connection may result. A sender that continues to send reverse-direction messages after every transport reconnect to an unprepared endpoint could cause a denial-of-service condition.
RPC security is handled at the RPC layer above the transport layer where RPC-over-RDMA operates. Reverse-direction operations use an authentication mechanism and credentials independent of forward-direction operation, but otherwise operate in the same fashion as described in RFC 8166 Section 8.2.