Compacted-DNS (C-DNS): A Format for DNS Packet Capture
Extracted elements (27)
CBOR was chosen as the encoding format because it is a binary IETF specification that is economical in storage, simple enough to implement from scratch, easily converted to JSON for debugging, and supports schema description via CDDL. No other binary format investigated had a significant advantage over CBOR.
Malformed DNS messages may optionally be stored as raw wire-format bytes. Without this, a malformed Query that cannot be represented structurally causes its FORMERR Response to appear as 'unmatched', which can produce misleading statistics in downstream analysis.
Multiple Q/R data items are collected into blocks, with common data abstracted into shared Block Tables referenced by index. This blocking and indexing provides significant file size reduction by exploiting DNS message structure knowledge, and also substantially reduces CPU needed for subsequent general-purpose compression.
StorageHints uses a negative/omit-oriented flag model: an unset bit means the field is omitted. A positive/include flag would not guarantee presence, only possible presence. This lets decoders quickly determine whether input data is insufficient for their needs (e.g., traffic regeneration) without scanning the data.
The basic unit of C-DNS data is a combined DNS Query and its associated Response (a Q/R data item). This design was chosen because a Query and Response represent the basic client interaction, and combining them reduces storage requirements due to data commonality between the two messages.
C-DNS can be converted back to PCAP format, but this requires a sufficient set of stored fields. If so few fields are stored that a decoder cannot determine whether a Q/R data item originated from a Query only, a Response only, or a Query/Response pair, DNS traffic cannot be reconstructed even with sensible defaults for missing fields.
Different DNS implementations use different name compression algorithms when generating packets (e.g., NSD vs. Knot Authoritative). C-DNS stores names uncompressed; when converting back to PCAP, the reconstructed compression will generally differ from the original, producing valid but byte-for-byte non-identical packets.
If IP address prefixes are stored (client or server prefix lengths specified in StorageParameters), then 'qr-transport-flags' in QueryResponseSignature, 'ae-transport-flags' in AddressEventCount, and 'mm-transport-flags' in MalformedMessageData MUST be present, as they carry the IP version bit needed to interpret the truncated address.
In BlockTables, if 'qlist' is present then 'qrr' must also be present; if 'rrlist' is present then 'rr' must also be present, because qlist entries are indexes into qrr and rrlist entries are indexes into rr.
It is RECOMMENDED that the 'sampling-method' and 'anonymization-method' fields in StorageParameters contain URIs pointing to resources describing the methods used, to enable downstream consumers to understand how data was processed.
Positive integer map keys not currently assigned are reserved for future standard extensions. Negative integer map keys (from -1 to -24, which also have single-byte CBOR representation) are reserved for implementation-specific entries, ensuring private extensions are not at a space efficiency disadvantage.
C-DNS captures client IP addresses and query names, which are personal data in many jurisdictions. The format provides mechanisms to address this: IP address prefix storage (truncating client addresses), storage-flags to signal anonymization or sampling, and the 'anonymization-method' field to document the technique used. Operators should consider omitting or anonymizing client address fields at capture time rather than post-processing.
A Block is a CBOR map containing 'block-preamble' (mandatory), and optional 'block-statistics', 'block-tables', 'query-responses', 'address-event-counts', and 'malformed-messages'. Each array, if present, must not be empty. Blocks share common BlockParameters identified by an index into the file-level array.
AddressEventCount records per-client counts of network events (ICMP messages, TCP resets) received at the server. It is stored as a block-level array to capture traffic metadata not captured in Q/R items.
BlockTables is a map of shared arrays referenced by index from QueryResponse and MalformedMessage items: 'ip-address' (byte strings), 'classtype' (ClassType maps), 'name-rdata' (byte strings), 'qr-sig' (QueryResponseSignature maps), 'qlist'/'qrr' (question lists and questions), 'rrlist'/'rr' (RR lists and RRs), and 'malformed-message-data'.
QueryResponseSignature captures fields common across multiple Q/R data items, including server address/port, transport flags, QR type (Stub/Client/Resolver/Authoritative/Forwarder/Tool), QR signature flags, query OPCODE, DNS header flags, RCODEs, section counts, and EDNS parameters. These are stored once in a block table and indexed.
IANA created a 'C-DNS Storage Flags' registry for bit flags in the 'storage-flags' field of StorageParameters. Initial values: bit 0=anonymized data, bit 1=sampled data, bit 2=names normalized to uniform case.
IANA created a 'C-DNS Transport Types' registry. Initial values: 0=UDP, 1=TCP, 2=TLS, 3=DTLS, 4=HTTPS, 15=Non-standard transport. Values 5-14 are reserved. These values are used in the 'qr-transport-flags', 'ae-transport-flags', and 'mm-transport-flags' fields (bits 1-4).
IANA created 'C-DNS Response-Processing Flags' and 'C-DNS AddressEvent Types' registries for the response-processing-data field and address-event-count type field respectively, enabling future extensibility of these classifications.
C-DNS captures full DNS message content including client IP addresses and query names, which can reveal sensitive information about users' browsing and online activities. Operators must consider access controls, secure storage, and transmission security for C-DNS files, as they represent a comprehensive log of DNS client behavior.
A C-DNS File is a CBOR array containing three mandatory fields: 'file-type-id' (text string 'C-DNS'), 'file-preamble' (a FilePreamble map), and 'file-blocks' (array of Block maps, which may be empty).
FilePreamble is a CBOR map with mandatory fields 'major-format-version' (unsigned integer 1), 'minor-format-version' (unsigned integer 0), and 'block-parameters' (array of BlockParameters, minimum one entry). An optional 'private-version' is available for implementation-specific use.
'qr-transport-flags' in QueryResponseSignature is a conditionally mandatory unsigned integer: bit 0 encodes IP version (0=IPv4, 1=IPv6); bits 1-4 encode transport (0=UDP, 1=TCP, 2=TLS, 3=DTLS, 4=HTTPS, 15=non-standard, 5-14 reserved); bit 5 indicates trailing bytes in the Query packet.
StorageHints is a map of four mandatory unsigned integers encoding which optional fields are omitted: 'query-response-hints' (18 bits, covering time-offset through response-additional-sections), 'query-response-signature-hints' (17 bits), 'rr-hints' (2 bits for ttl and rdata-index), and 'other-data-hints' (2 bits for malformed-messages and address-event-counts). An unset bit means that field is omitted.
StorageParameters is a mandatory map in BlockParameters containing 'ticks-per-second' (sub-second timing resolution), 'max-block-items' (maximum array sizes, for decoder resource planning), 'storage-hints' (StorageHints map), 'opcodes' (array of recorded OPCODEs 0-15), and 'rr-types' (array of recorded RR TYPEs 0-65535).
The 'earliest-time' field in BlockPreamble is a two-element CBOR array (Timestamp): the first unsigned integer is seconds since the POSIX epoch (excluding leap seconds), the second is sub-second ticks since the start of the second, where tick resolution is given by 'ticks-per-second' in StorageParameters.
The 'name-rdata' BlockTable array stores each DNS NAME or RDATA in wire format as a byte string. Names and labels within RDATA are stored as full domain names with no DNS name compression (per RFC 1035); compression is only applied at the C-DNS block level through indexing.