HMAC-MD5 IP Authentication with Replay Prevention
Extracted elements (19)
ipad (0x36 repeated 64 times) and opad (0x5C repeated 64 times) are fixed, distinct strings used to domain-separate the inner and outer MD5 computations in HMAC, ensuring the two hashes use different key material.
The advised key length is 128 bits, equal to the MD5 hash output length. Keys shorter than 128 bits are strongly discouraged as they decrease security strength; keys longer than 128 bits are acceptable but extra length may not significantly increase strength unless key randomness is suspect.
The AH key is described as a shared secret rather than a traditional cryptographic key: it is hashed with the transmitted data so an intervening party cannot duplicate the authentication data. The strength of the transform lies in the singular mapping of key and IP datagram to authentication data.
This transform is defined for use with the IP Authentication Header [RFC-1826]. The Next Header, RESERVED, and SPI fields follow RFC-1826 definitions; this document adds the HMAC-MD5-specific Length interpretation and the optional Replay Prevention field.
A key length of zero is prohibited and implementations MUST prevent key lengths of zero from being used with this transform, since no effective authentication could be provided by a zero-length key.
All conforming and compliant implementations MUST support a key length of 128 bits or less. Implementations SHOULD support longer key lengths as well.
All implementations that claim conformance or compliance with the IP Authentication Header specification [RFC-1826] MUST implement this HMAC-MD5 transform.
If an out-of-order window is supported, the implementation shall ensure that any and all packets accepted out of order are guaranteed not to have arrived before — each packet is accepted at most once.
Keys longer than 64 bytes MUST first be hashed using MD5; the resulting hash is then used to calculate the authentication data. This applies both to AH key setup and during the HMAC-MD5 computation.
The shared secret key must not be used for a period of time that allows the 64-bit replay counter to wrap, i.e., no more than 2^64 packets may be transmitted using a single key.
When the destination is a multicast address with more than one sender sharing the same IPsec SA, Replay Protection SHOULD NOT be enabled. For multicast replay protection with multiple senders, each sender SHOULD have its own IPsec Security Association.
Authentication data is computed over the entire IP datagram. Fields that are variant during transit and the authentication data field itself must contain all zeros prior to computation. The Replay Prevention field, if present, is included in the calculation.
HMAC-MD5 is computed as MD5(K XOR opad, MD5(K XOR ipad, text)), where ipad is 0x36 repeated 64 times and opad is 0x5C repeated 64 times, and K is zero-padded to 64 bytes (or pre-hashed with MD5 if longer than 64 bytes).
The Security Parameters Index (SPI) is used to determine whether the Replay Prevention option is included in the AH, giving per-SA control over whether replay protection is active.
The security of HMAC-MD5 IP authentication depends on the strength of MD5, correctness of the algorithm implementation, security and correctness of the key management mechanism, strength of the associated secret key, and correctness of all participating system implementations. Implementations should consult [HMAC-MD5] for a detailed discussion of MD5 strengths and weaknesses.
Replay Prevention state: the 64-bit counter starts at 1 and increments per packet. Upon receipt, the replay value is assured to be increasing. An implementation may optionally maintain an out-of-order acceptance window but must ensure no packet is accepted more than once.
The AH packet format contains: Next Header (8 bits), Length (8 bits), RESERVED (16 bits), SPI (32 bits), optional Replay Prevention (64 bits), and Authentication Data (128 bits). The Length field counts the Replay Prevention field and Authentication Data in 32-bit words.
The Authentication Data field is 128 bits (the MD5 output), naturally 64-bit aligned and requiring no padding. It is computed over the entire IP datagram with variant-in-transit fields and the authentication data field itself zeroed.
The Replay Prevention field is a 64-bit up-counter starting at 1, present only when replay prevention is enabled for the Security Association. If not in use, the Authentication Data field directly follows the SPI field.