High Availability within a Forwarding and Control Element Separation (ForCES) Network Element
updated by
- rfc-7391 — Forwarding and Control Element Separation (ForCES) Protocol Extensions
updates
- rfc-5810 — Forwarding and Control Element Separation (ForCES) Protocol Specification
Extracted elements (25)
Backup CEs are permitted to issue queries and receive heartbeats from the FE (even though SET/DEL config messages are dropped) because in practice it has been found useful to allow backup CEs to monitor FE state before taking over as master.
Hot standby improves over cold standby by having the FE pre-associate with all backup CEs while the master is active. On master failure, there is no association setup cost — the FE simply promotes an already-associated backup, reducing failover time to near-zero for steps 2 and 3 of the NE recovery metric.
FEPO v1.1 retains backward compatibility with CEs that implement only v1.0 because no existing components were changed — only new datatypes and components were appended. However, v1.0-only CEs cannot use the AllCEs, HAMode, or PrimaryCEChanged features introduced in v1.1.
A communication failure between an FE and CE, regardless of whether detected by the TML or by Heartbeat messages, MUST be considered to be a loss of association between the CE and the corresponding FE.
Asynchronous events that the master CE has subscribed to, as well as heartbeats, MUST be sent to all associated CEs. Packet redirects continue to be sent only to the master CE.
Configuration messages (SET/DEL) from backup CEs MUST be dropped by the FE and logged as received errors. Backup CEs are permitted only to issue queries and receive heartbeats and async events.
FEs and CEs requiring HA MUST interoperate per the definition in RFC 5810 cold standby setup. This default setup MUST be implemented by CEs and FEs requiring HA; the Fr (CE-CE) plane is out of scope and, if available, is proprietary.
If CE failover policy is 1 and HAMode is 2 (hot standby), after successfully associating with the master CE, the FE MUST attempt to connect and associate with all the CEs of which it is aware. The FE MAY flag unreachable CEs to avoid continuous reconnection attempts.
In the pre-association phase, the FEM MUST configure the FE to make it aware of which CE is the master and MAY specify any backup CE(s). This information is static and provisioned before association.
In the pre-association phase, the first CE (lowest table index) in the AllCEs table MUST be the first CE with which the FE will attempt to connect and associate. The FE MUST associate with at least one CE.
When CEFailoverPolicy is mode 1 and CEFTI expires without successful re-association, the FE transitions to the pre-association state and MUST operationally bring down its forwarding path by setting the RFC 5812 FEObject FEState component to OperDisable.
When the master CE is considered down in hot standby mode, the FE MUST try to find the first already-associated CE from the AllCEs list in round-robin fashion. If none is found, it MUST attempt to connect and associate with the first CE in the list, cycling round-robin until success or CEFTI expiry.
AllCEType struct combines CEID (uint32), Statistics (StatisticsType), and CEStatus (CEStatusType) to represent one CE's full connection information. Used as the element type in the read-only AllCEs array component of FEPO v1.1.
CEStatusType (unsigned char) tracks per-CE connection state with six values: 0=Disconnected (no attempt yet), 1=Connected (TML connection complete), 2=Associated, 3=IsMaster (associated and master), 4=LostConnection, 5=Unreachable (repeated failed attempts).
FEPO v1.1 adds two events: PrimaryCEDown (fires when LastCEID changes, reports the previous master CE ID) and PrimaryCEChanged (fires when CEID changes, reports the new master CE ID). Both are sent to all associated CEs on mastership transition.
HAModeValues (unsigned char) specifies the FE's operating HA mode: 0=No HA Mode, 1=Cold Standby, 2=Hot Standby. Stored in the read-write HAMode component of FEPO v1.1.
StatisticsType is a struct with eight uint64 counters per CE: RecvPackets, RecvErrPackets (includes badly formatted packets and packets from non-master CEs), RecvBytes, RecvErrBytes, TxmitPackets, TxmitErrPackets, TxmitBytes, TxmitErrBytes.
The FE Protocol Object (FEPO) LFB carries HA parameters including CEID (current master CE), BackupCEs (table of backup CEs), FEHI (FE Heartbeat Interval), CEHDI (CE Heartbeat Dead Interval), CEFailoverPolicy, and CEFTI (CE Failover Timeout Interval). These components govern CE failure detection and failover behavior.
FEPO LFB version 1.1 is registered: LFB Class Identifier 2, LFB Class Name 'FE Protocol Object', LFB Version '1.1', with RFC 7121 as the reference. It defines parameters for hot standby HA operation within ForCES.
The 'Logical Functional Block (LFB) Class Names and Class Identifiers' IANA registry is updated to add an LFB Version column. All existing entries receive version 1.0. New versions of already-defined LFBs MUST NOT remove previous version entries; the table SHOULD be sorted by Class ID then version.
Secure communication channels SHOULD be used between CEs to prevent connection of malicious CEs. The master CE should account for DoS and DDoS attacks from malicious or malfunctioning CEs on the CE-CE plane.
Split-brain risk is mitigated by two FE fail-safes: the CEID component designates exactly one master CE, and backup CEs are prohibited from sending configuration messages to the FE. Backup CEs that believe the master has dropped SHOULD query the FE CEID component before acting as master.
The FE initiates all associations with CEs; a CE cannot initiate association with an FE and such inbound messages are dropped. This protects the FE from rogue CEs attempting to claim control.
FE cold standby state machine has three states: Pre-association, Associated, and Not Associated. Triggers include CE mastership change, CE-issued Teardown, or lost association. CEFailoverPolicy=0 causes immediate return to Pre-association; CEFailoverPolicy=1 transitions to Not Associated and starts the CEFTI timer.
The hot standby FE state machine (Figure 4) extends cold standby by: in the Not Associated state, the FE first searches for an already-associated CE before attempting new associations, and on successful recovery it issues both PrimaryCEDown and PrimaryCEChanged events.