Sieve Email Filtering: Detecting Duplicate Deliveries
Extracted elements (24)
The ':handle' argument was introduced to prevent interference between unrelated 'duplicate' tests that may encounter similar or overlapping ID values (e.g., two tests using different headers whose values happen to collide). Without ':handle', all duplicate tests share the same tracking namespace.
The mechanism to track duplicate messages is very similar to the mechanism needed for tracking duplicate responses for the 'vacation' action. One recommended implementation strategy is to store a hash of the tracked unique ID combined with the ':handle' argument, leveraging the existing vacation-tracking infrastructure.
The ':uniqueid' argument combined with the 'variables' [VARIABLES] and 'foreverypart'/'extracttext' [SIEVE-MIME] extensions provides detailed control over unique ID creation from arbitrary message parts, enabling advanced duplicate detection beyond header fields.
The 'duplicate' test does not support the 'index' [DATE-INDEX] or 'mime' [SIEVE-MIME] extensions directly; none of the ':index', ':mime', or associated arguments are added to the test. The ':uniqueid' argument combined with the 'variables' extension can achieve equivalent behavior indirectly.
When a Sieve script includes other scripts via the 'include' extension, all 'duplicate' tests across included scripts within a single execution with identical arguments MUST yield the same result, since the tracking list is only updated at the end of the outermost successful execution.
If a header field specified via ':header' exists multiple times, extraction MUST use only the first occurrence. If the specified header field is absent, the 'duplicate' test MUST yield 'false' unconditionally and the tracking list is left unmodified.
If a site's maximum expiration limit is exceeded by the ':seconds' argument, the maximum value MUST be silently substituted; exceeding the limit MUST NOT produce an error. If ':seconds' is zero, the 'duplicate' test MUST yield 'false' unconditionally.
Implementations MUST only update the internal duplicate-tracking list when the Sieve script execution finishes successfully. Failing script executions MUST NOT add the unique ID to the list, as doing so would cause all 'duplicate' tests to erroneously yield 'true' for the next delivery attempt.
Implementations SHOULD let entries in the tracking list expire after a short period of time. If the ':seconds' argument is omitted, an appropriate default value MUST be used; a default of around 7 days is usually appropriate. Sites SHOULD impose a maximum limit on expiration time.
Situations in which the 'duplicate' test erroneously yields 'true' MUST be prevented, even though deferring tracking-list updates to script completion can create a race condition when a duplicate is delivered in parallel before the list is updated.
Specifying both ':header' and ':uniqueid' arguments for a single 'duplicate' test MUST trigger an error. Implementations MUST NOT trigger an error for an invalid header name; instead, the 'duplicate' test MUST yield 'false' unconditionally in that case.
The 'duplicate' test is devised for use at final delivery; its semantics in the 'imapsieve' context are left undefined. Implementations SHOULD NOT allow the 'duplicate' test to be used in the context of 'imapsieve'.
The 'duplicate' test MUST only check for duplicates among unique ID values from previous Sieve script executions; it MUST NOT treat ID values encountered earlier in the current execution as potential duplicates. All 'duplicate' tests within one execution with identical arguments MUST always yield the same result.
The unique ID MUST be matched case-sensitively against the duplicate-tracking list, regardless of how the unique ID was determined. Case-insensitive behavior when using ':uniqueid' must be achieved by normalizing the value in the Sieve script using the 'variables' extension.
When a unique ID is extracted from a message header field (not via ':uniqueid'), implementations MUST: unfold the header line per RFC 5322 Section 2.2.3, convert the value to Unicode/UTF-8 if possible, and trim leading and trailing whitespace.
When ':last' is omitted, the expiration time MUST be measured from when the entry was first created. When ':last' is specified, the expiration time MUST be measured relative to the last script execution that checked that unique ID value, effectively keeping the entry alive while duplicates keep arriving.
A user who deletes a message from the server may expect no record remains, but the message ID persists in the duplicate-tracking list for the duration of the expiry interval. Users of services that intentionally limit traceability should be informed of this, and a shorter default expiry may be appropriate in those situations.
The duplicate-tracking list can include privacy-sensitive information such as message ID values, subject lines, and content extracted from message bodies. Implementations SHOULD protect this information by obscuring it through hashing and/or storing it with access control equivalent to that of the messages themselves.
The 'duplicate' test identifies a message by a unique ID and evaluates to 'true' if that ID was seen in a previous Sieve execution, 'false' otherwise. As a side effect, the unique ID is added to an internal duplicate-tracking list after the Sieve execution finishes successfully.
The ':handle' argument separates a 'duplicate' test from other 'duplicate' tests with a different or omitted ':handle' argument, preventing unrelated tests from interfering with each other. A message is recognized as a duplicate only when the tracked unique ID was seen before by a 'duplicate' test with the same ':handle' argument.
The unique ID for a message is determined by: the Message-ID header field (default), the content of a specific header field when ':header' is used, or an explicit string value when ':uniqueid' is used. The ':header' and ':uniqueid' arguments are mutually exclusive.
This document registers the Sieve capability string 'duplicate' in the IANA Sieve Extensions registry. The capability indicates support for the 'duplicate' test command defined in RFC 7352.
A flood of unique messages could cause the duplicate-tracking list to grow indefinitely. Implementations SHOULD limit the number of entries in the list and SHOULD discard the oldest entries first when the limit is reached.
Message IDs are not necessarily unique, either through benign generator faults or attacker injection of messages with crafted properties. Scripts should be conservative with actions taken when duplicates are identified only by message ID, as false duplicate detection may result.