Flow Aggregation for the IP Flow Information Export (IPFIX) Protocol
Extracted elements (30)
Aggregate combination is idempotent, meaning any number of Intermediate Aggregation Processes operating in series can be modeled as a single process. This allows aggregation operations to be flexibly distributed across multiple processes without altering the semantics of the result.
External interval distribution—configuring Original Exporters to synchronize their Flow intervals with the desired aggregation interval—avoids most IAP interval distribution complexity and allows simpler Start Interval or End Interval accounting, at the cost of reducing the usefulness of Original Flows for non-aggregation purposes.
No new aggregation-specific timestamp Information Elements are introduced because Aggregated Flows are simply IPFIX Flows and existing timestamp IEs (e.g., flowStartMilliseconds, flowEndNanoseconds) are fully sufficient for expressing aggregation intervals.
The document defines both conservative and non-conservative Original Flow counting because each serves distinct analytical purposes: conservative counts allow accurate total flow reconstruction across re-aggregation, while non-conservative counts reflect how many Contributing Flows are actually present within each Aggregated Flow.
Aggregation of sampled Flows should only be applied to Original Flows sampled using the same sampling rate and algorithm, or Flows that have been normalized to an equivalent sampling rate before aggregation, to maximize comparability of the resulting Aggregated Flows.
When aggregating Flows from multiple Metering Processes with different active timeouts, the IAP delay is determined by the maximum active timeout among all contributing Metering Processes, as the IAP must wait for all Original Flows that may contribute to a given interval.
When aggregating heterogeneous Original Flows from different Observation Points into a single Aggregated Flow stream, the correlation and normalization operation must ensure each packet is represented in only one Aggregated Flow or set of Aggregated Flows differing only by aggregation interval, to prevent double-counting.
All counters in a given Aggregated Flow MUST be distributed via the same method to avoid confusion in interpretation of the aggregated data.
Counters MUST be distributed across multiple Aggregated Flows such that the total count is preserved within the limits of implementation accuracy, allowing data to be aggregated and re-aggregated with negligible loss of original count information.
Each Aggregated Flow carrying timing information SHOULD contain both an interval start and interval end timestamp; no new aggregation-specific timestamp IEs are necessary as existing IPFIX IEs are sufficient.
In the Aggregate Counter Distribution Options Template, the templateId Information Element MUST be defined as a Scope field, as its uniqueness is local to the Transport Session and Observation Domain.
Original Exporters SHOULD be configured to export Flows with equal or similar activeTimeout and inactiveTimeout values and the same cacheMode to ensure meaningful and comparable Original Flow counts during aggregation.
The interval distribution process SHOULD accept Flow start and end times in Original Flows in any reasonable order. The expiration of intervals MUST be made configurable in contexts where 'reasonable order' is not obvious at implementation time.
Aggregation operations that eliminate IP addresses make it impossible to later directly identify nodes using those addresses, providing a potential anonymizing effect on Aggregated Flows; however, this effect must be explicitly evaluated against RFC 6235 guidelines and is not a guaranteed privacy protection.
Aggregate combination merges multiple Partially Aggregated Flows sharing the same Flow Keys and Aggregation Interval into a single Aggregated Flow, combining non-key values using per-element operations: delta counters are summed, flags are unioned, etc. This operation is idempotent, so any series of IAPs can be modeled as a single process.
An Aggregated Flow is an IPFIX Flow derived from zero or more Original Flows within a defined Aggregation Interval. Once exported, an Aggregated Flow is treated as a standard IPFIX Flow per RFC 7011 and requires no protocol changes for export or storage.
deltaFlowCount (IE 3, unsigned64, deltaCounter) is the conservative count of Original Flows contributing to an Aggregated Flow, distributable via any method expressed by valueDistributionMethod.
Interval distribution imposes an Aggregation Interval on Partially Aggregated Flows. Intervals may be regular, irregular, or derived from Original Flow timing. Choosing an interval longer than the longest Original Flow limits each Original Flow to contributing to at most two Aggregated Flows, greatly simplifying counter distribution.
Key aggregation generates new Aggregated Flow Key values by reduction (dropping fields or precision, e.g., masking an IP address to a /24) or replacement (substituting derived values, e.g., IP-to-ASN lookup). Both operations may be applied any number of times in any order and may also add new non-key fields such as Original Flow counters or distinct key counters.
originalFlowsInitiated (IE 376, unsigned64, deltaCounter) is the conservative count of Original Flows whose first packet is represented within an Aggregated Flow. originalFlowsCompleted (IE 377, unsigned64, deltaCounter) is the conservative count of Original Flows whose last packet is represented. Both must sum to the original count on re-aggregation.
originalFlowsPresent (IE 375, unsigned64, deltaCounter) is the non-conservative count of Original Flows contributing to an Aggregated Flow. Non-conservative counts need not sum to the original count on re-aggregation.
Seven counter distribution methods are defined for distributing Original Flow counters across Aggregated Flows when an Original Flow spans multiple intervals: End Interval, Start Interval (default), Mid Interval, Simple Uniform Distribution, Proportional Uniform Distribution, Simulated Process, and Direct.
Six distinct host count Information Elements are defined to export distinct address counts reduced away during key aggregation: distinctCountOfSourceIPAddress (IE 378, unsigned64), distinctCountOfDestinationIPAddress (IE 379, unsigned64), distinctCountOfSourceIPv4Address (IE 380, unsigned32), distinctCountOfDestinationIPv4Address (IE 381, unsigned32), distinctCountOfSourceIPv6Address (IE 382, unsigned64), distinctCountOfDestinationIPv6Address (IE 383, unsigned64); all use totalCounter semantics.
The Aggregate Counter Distribution Options Template binds a valueDistributionMethod to a Template ID scope field, allowing an Exporting Process to signal to Collecting Processes which counter distribution method was applied to the Aggregated Flows defined by that Template.
The IAP may be deployed at three locations within the IPFIX architecture: integrated with a Metering/Exporting Process before initial export, within a Mediator that collects Original Flows and exports Aggregated Flows, or within a Collecting Process for data reduction prior to storage.
The Intermediate Aggregation Process (IAP) architecture is composed of four operations on Partially Aggregated Flows: interval distribution (temporal), key aggregation (spatial), value aggregation (spatial), and aggregate combination. The first three may be applied in any order and any number of times; at least one interval distribution step is required.
Two Original Flow counting modes are defined: conservative counting (each Original Flow contributes exactly one to the total across all Aggregated Flows, with fractional distribution as needed) and non-conservative counting (each Contributing Flow presence in an Aggregated Flow adds one, so the total sum may exceed the Original Flow count).
valueDistributionMethod (IE 384, unsigned8) encodes the counter distribution method applied to non-key fields in associated Aggregated Flows: 0=Unspecified, 1=Start Interval (assumed default if absent), 2=End Interval, 3=Mid Interval, 4=Simple Uniform Distribution, 5=Proportional Uniform Distribution, 6=Simulated Process, 7=Direct. The originalFlowsInitiated and originalFlowsCompleted IEs are exempt from this method as they imply their own distribution.
RFC 7015 registers ten new IPFIX Information Elements with IANA: originalFlowsPresent (375), originalFlowsInitiated (376), originalFlowsCompleted (377), distinctCountOfSourceIPAddress (378), distinctCountOfDestinationIPAddress (379), distinctCountOfSourceIPv4Address (380), distinctCountOfDestinationIPv4Address (381), distinctCountOfSourceIPv6Address (382), distinctCountOfDestinationIPv6Address (383), and valueDistributionMethod (384).
Any application of aggregation as part of a data protection or anonymization scheme must address all issues raised in RFC 6235, specifically Sections 4 (Anonymization of IP Flow Data), 7.2 (IPFIX-Specific Anonymization Guidelines), and 9 (Security Considerations), as aggregation alone does not guarantee adequate anonymization.