Specification of the IP Flow Information Export (IPFIX) File Format
Extracted elements (29)
All file metadata (checksums, time windows, session details) is stored as IPFIX Options within the message stream rather than in a separate file header, making IPFIX Files fully wire-protocol-compatible and requiring no extensions to the IPFIX protocol itself.
Anonymization support was omitted from the IPFIX File format because anonymization algorithms remain an open research area and no standardized methods existed at time of publication; the IPFIX requirements document had already qualified this requirement as MAY rather than MUST.
Flat file format was chosen over RDBMS-based storage because RDBMS internal data structures are generally private to a single implementation and unsuitable for standardization, while flat files support a broad ecosystem of external tools and standards. Flow data's simple, repetitive structure also does not benefit significantly from relational technology.
IPFIX Message format was chosen as the file format basis because it is extensible, compact, and purpose-built for flow data export. It allows implementations to reuse substantially the same information model and data formatting code for both transport and file storage, unlike ad hoc serializations which impose a fixed data model.
XML was rejected for flow storage because its start/end-tag overhead and plaintext value encoding are severely inefficient for high-volume, repetitive flow records (millions to billions per hour). XML parsers are also complex and resource-intensive compared to the simple linear parsers possible for binary flow formats.
IPFIX Files containing enterprise-specific Information Elements are not fully self-describing unless those IEs are additionally described via the Information Element Type Options Template (RFC 5610). A File Reader encountering unknown enterprise IEs can only treat them as opaque octet arrays.
NetFlow V9 flow data can be stored in IPFIX Files because the NetFlow V9 information model is a compatible subset of the IPFIX Information Model. Appendix B specifies a transformation method for converting NetFlow V9 Messages to IPFIX format for multi-vendor collection environments.
The supported compression formats for IPFIX Files are bzip2 and gzip, specified to maximize interoperability. Block compression algorithms SHOULD be preferred over stream compressors because they isolate errors to a single block, whereas stream compressors may fail to resynchronize after a single bit error and invalidate the entire stream.
A File Reader MUST NOT treat a single IPFIX File as containing multiple Transport Sessions; in the general case a File SHOULD be treated as a single Transport Session, though a File Reader MAY be configurable to treat a collection of Files as one session.
A File Writer MUST NOT write more than one time window record to an IPFIX File, and MUST NOT write any Flow with a start time before the declared window beginning or an end time after the declared window end.
Any valid serialized IPFIX Message stream MUST be accepted by a File Reader as a valid IPFIX File; the filesystem is treated as another IPFIX transport alongside SCTP, TCP, and UDP.
File Readers MUST support the union of IPFIX Template management features across SCTP, TCP, and UDP: accepting Templates, Options Template Sets, and Data Sets in the same message; accepting redefined Templates (new replaces old); and accepting Template Withdrawals.
File Writers configurable to store a single Transport Session across multiple IPFIX Files MUST also provide a configuration option supporting one-file-per-session behavior for interoperability purposes.
IPFIX File Writers MUST behave as IPFIX Exporting Processes and IPFIX File Readers MUST behave as IPFIX Collecting Processes as specified in RFC 5101, except as modified by RFC 5655.
IPFIX Files intentionally containing non-compliant messages for Collector error-handling tests MUST NOT be transmitted to Collecting Processes or given as input to File Readers that are not under test.
When a File Writer reorders messages from SCTP-transported sessions, Template reordering MUST preserve the sequence of Template Sets with Template Withdrawals to ensure Template consistency.
When IPFIX Files are used as documents, each File MUST be readable as a single Transport Session, self-contained aside from any detached signature, and making no reference to metadata stored in separate Files, to ensure interoperability.
An IPFIX File is a serialized sequence of IPFIX Messages stored to a filesystem. A typical file begins with one IPFIX Message containing Options Template Sets and Template Sets, followed by one or more messages containing Data Sets with the actual flow records.
The Export Session Details Options Template records IPFIX Transport Session context—exporter and collector IPv4/IPv6 addresses, ports, transport protocol, TLS/DTLS certificates, and min/max export timestamps—all scoped via sessionScope. It SHOULD appear only once per File.
The File Time Window Options Template uses sessionScope (scope), minFlowStart* (earliest flow start time), and maxFlowEnd* (latest flow end time), where the precision of min and max timestamps SHOULD match. At most one time window record MAY appear per File.
The Message Checksum Options Template uses messageScope (scope field, value written as 0 and ignored on read) and messageMD5Checksum (16-byte MD5 of the Message with this field zeroed). The checksum Data Record MUST appear at most once per IPFIX Message.
The Message Details Options Template attaches per-message metadata including collectionTimeMilliseconds, exporter/collector addresses and ports, transport protocol, collectorProtocolVersion, and exportSctpStreamId, scoped per message via messageScope. It enables storage of multiple Transport Sessions in one File.
The messageMD5Checksum Information Element (ElementId 262, octetArray, 16 bytes) carries the MD5 digest of the containing IPFIX Message computed with this field set to all zeros. It is explicitly intended as a simple error-detection checksum; collision resistance and algorithm agility are not required.
The opaqueOctets Information Element (ElementId 266, octetArray) encapsulates arbitrary non-IPFIX data inline within an IPFIX File to allow non-IPFIX data processors to embed a data stream. Collecting Processes and File Writers MUST NOT interpret its contents, distinguishing it from paddingOctets (which must be 0x00).
The sessionScope Information Element (ElementId referenced in Section 8.2.18) provides session-level scoping for Options Templates; its presence as scope signifies that options apply to the containing Transport Session. Its value MUST be written as 0 and MUST be ignored by readers.
Section 13 registers the application/ipfix media type for IPFIX Files per RFC 4288, and registers new IPFIX Information Elements with ElementIds 258–275, including collectionTimeMilliseconds (258), messageMD5Checksum (262), messageScope (263), opaqueOctets (266), sessionScope, collectorCertificate (274), exporterCertificate (275), and a family of time-window IEs for file metadata.
A File Writer collocated with a Collecting Process SHOULD NOT sign an IPFIX File unless the originating Transport Session was protected by TLS or DTLS and the Exporting Process was positively identified by certificate; signing data from an unauthenticated source provides no meaningful end-to-end data provenance guarantee.
Collocating a File Writer with a Metering Process is the only way to provide positive cryptographic authentication of a Metering Process through CMS signatures. When the File Writer is collocated only with a Collecting Process, the chain of custody from the Metering Process cannot be cryptographically verified.
IPFIX Files may be signed using CMS (RFC 3852) detached signatures stored separately from the File to provide integrity and authentication. The CMS SignedData structure with detached EncapsulatedContentInfo is the specified signing method.