An Experiment: Network Anomaly Detection Lifecycle

Network Anomaly Detection is the act of detecting problems in the network. Accurately detecting problems is very challenging for network operators in production networks. Good results require a lot of expertise and knowledge around both the implied network technologies and the connectivity services provided to customers, apart from a proper monitoring infrastructure. In order to facilitate network anomaly detection, novel techniques are being introduced, including programmatical, rule-based and AI-based, with the promise of improving scalability and the hope to keep a high detection accuracy. To guarantee acceptable results, the process needs to be properly designed, adopting well-defined stages to accurately collect evidence of anomalies, validate their relevancy and improve the detection systems over time, iteratively. This document describes a well-defined approach on managing the lifecycle process of a network anomaly detection system, spanning across the recording of its output and its iterative refinement, in order to facilitate network engineers to interact with the network anomaly detection system, enable the "human-in-the-loop" paradigm and refine the detection abilities over time. The major contributions of this document are: the definition of three key stages of the lifecycle process, the definition of a state machine for each anomaly annotation on the system and the definition of YANG data models describing a comprehensive format for the anomaly labels, allowing a well-structured exchange of those between all the interested actors.