11 min read

Deduplicating Repeat Alerts with Fingerprint Keys

An audit pipeline that runs hourly and finds the same problem every hour will page somebody twenty-four times a day unless something tells the alerting system that these are all one incident. That something is a fingerprint: a stable identifier derived from what is broken rather than from when it was observed. This page is part of Configuring Alert Thresholds and Routing and covers building one that actually repeats.

What goes in the fingerprint Two columns. The include column lists the rule identifier, the scope the finding applies to, the environment and the severity tier — all properties of what is broken. The exclude column lists the measured value, the timestamp, the run identifier and the human-readable message — all properties of when it was observed. The verdicts note that including anything from the second column means the key never repeats and the cooldown never matches. INCLUDE — what is broken rule_id which check produced this scope the URL segment or template environment production and staging are different severity tier an escalation is a new incident All stable across runs EXCLUDE — when it was seen the measured value changes every single run the timestamp obviously, and yet run_id or build number unique by construction the rendered message contains the value already Any of these breaks dedup

Environment isolation and dependency declaration 

set -euo pipefail
export FP_FIELDS="rule_id,scope_id,environment"   # the identity, in order
export FP_INCLUDE_SEVERITY=0        # 1 forks the incident on escalation
export FP_COOLDOWN_S=3600
export FP_STORE_URL="redis://alerts.internal:6379/2"
export FP_VERSION=2                 # bumped when the field list changes

FP_VERSION matters more than it looks. Changing the field list changes every fingerprint, so every open incident is orphaned and every finding re-alerts once. Versioning the key makes that a deliberate migration with a known one-time cost rather than a surprise burst nobody can explain.

Implementation 

#!/usr/bin/env python3
# /opt/audit/alerts/fingerprint.py
# A fingerprint identifies what is broken. Nothing that varies per run may
# enter it, or the key never repeats and deduplication silently stops.
from __future__ import annotations
import hashlib
import os
import time
from dataclasses import dataclass

import redis

FIELDS = os.environ.get("FP_FIELDS", "rule_id,scope_id,environment").split(",")
WITH_SEVERITY = os.environ.get("FP_INCLUDE_SEVERITY", "0") == "1"
VERSION = os.environ.get("FP_VERSION", "1")
COOLDOWN = int(os.environ.get("FP_COOLDOWN_S", "3600"))

_r = redis.from_url(os.environ["FP_STORE_URL"])


@dataclass(frozen=True)
class Finding:
    rule_id: str
    scope_id: str        # a stable identifier, not a display name
    environment: str
    severity: str
    value: float         # deliberately never part of the key
    run_id: str          # deliberately never part of the key


def fingerprint(f: Finding) -> str:
    parts = [getattr(f, name) for name in FIELDS]
    if WITH_SEVERITY:
        parts.append(f.severity)
    raw = "|".join(str(p) for p in parts)
    digest = hashlib.sha256(raw.encode()).hexdigest()[:24]
    return f"fp{VERSION}:{digest}"


def should_notify(f: Finding) -> tuple[bool, str]:
    """SET NX EX in one atomic command. A get-then-set pair would let two
    concurrent evaluations both decide to notify."""
    key = fingerprint(f)
    first = _r.set(key, int(time.time()), nx=True, ex=COOLDOWN)
    return (bool(first), key)

The decisions that make the key work:

  • The field list is configuration, and the version travels in the key. Changing which fields participate is a migration with a visible cost, and prefixing the version means old and new keys cannot collide during a rollout.
  • scope_id rather than a display name. A section renamed from "Checkout" to "Purchase flow" must not orphan its open incident, so the key is built from an identifier that outlives the label.
  • value and run_id are on the dataclass and never in the key. Keeping them on the model is deliberate — they belong in the alert payload, where a responder needs them, and not in the identity.
  • SET NX EX is one command. A read followed by a write leaves a window in which two concurrent evaluations both see no key and both notify, which is exactly the burst deduplication exists to prevent.
Too coarse, and too fine Two columns describing the same regression affecting four sections. A fingerprint of the rule id alone produces one incident covering all four, so fixing one section does not close anything and two teams share an incident neither fully owns. A fingerprint including the URL produces one incident per affected page, which is nine hundred pages of pager traffic for a single template fault. TOO COARSE — rule_id only One incident for four sections nobody fully owns it Fixing one closes nothing the incident stays open Ownership is ambiguous two teams, one incident Recovery is all-or-nothing it closes when everything is fixed TOO FINE — includes the URL One incident per page 900 pages, 900 incidents The pager is unusable in a burst, within minutes The pattern is invisible nobody sees it is one template Rate limits are hit the platform starts refusing The workable grain is the unit somebody can own and fix: rule plus scope plus environment. That is one incident per section per rule, which is how the work is actually assigned.

Verification and smoke test 

/opt/audit/.venv/bin/python3 - <<'PY'
from fingerprint import Finding, fingerprint, should_notify

base = dict(rule_id="lcp_p75_breach", scope_id="sec_checkout",
            environment="production", severity="critical")

# 1. The key is stable across runs with different values and run ids.
a = fingerprint(Finding(**base, value=3120.0, run_id="r-1"))
b = fingerprint(Finding(**base, value=4880.0, run_id="r-2"))
assert a == b, (a, b)
print(f"PASS: key stable across runs -> {a}")

# 2. A different scope produces a different key.
c = fingerprint(Finding(**{**base, "scope_id": "sec_blog"}, value=3120.0, run_id="r-1"))
assert c != a
print("PASS: a different scope is a different incident")

# 3. Exactly one of many concurrent evaluations notifies.
import concurrent.futures as cf
f = Finding(**base, value=3120.0, run_id="r-3")
with cf.ThreadPoolExecutor(16) as ex:
    results = list(ex.map(lambda _: should_notify(f)[0], range(16)))
assert sum(results) == 1, results
print(f"PASS: {sum(results)} of 16 concurrent evaluations notified")
PY

Expected output is three PASS lines, ending with exactly one of sixteen. The first assertion is the one that fails the moment somebody adds the measured value to the key "so the alert message is more specific" — a change that looks harmless and turns every cooldown into a no-op.

Failure modes

Three fingerprint defects Three rows. A value or timestamp inside the key means it never repeats, so every run opens a new incident and the cooldown never matches anything. A key that changes when severity escalates means a warning that becomes critical opens a second incident while the first stays open. And a key built from a mutable scope string breaks whenever a section is renamed, orphaning the open incident. DEFECT WHAT HAPPENS FIX A value is in the key a new incident opens on every scoring run The key never repeats so the cooldown has nothing to match against Hash identity only then flush the stale keys from the cooldown store Severity is in the key and a warning that becomes critical opens a second Escalation forks the incident the first stays open alongside the new one Decide deliberately either update severity in place, or close the first explicitly A section rename orphans every open incident for it The scope string is mutable the key changed because a label changed, not the finding Key on a stable id a section id, not its display name The second row is a genuine design choice rather than a defect — including severity is defensible, provided the escalation path explicitly closes the lower-severity incident rather than leaving both open.

Every run opens a new incident despite an active cooldown

Something that varies per run is in the key. Print the pre-hash string during a debugging run and compare two consecutive evaluations of the same finding — the differing component is always immediately obvious once the raw string is visible, and is almost never obvious from the hash.

A resolved incident immediately reopens

The cooldown expired before the underlying condition did, so the next evaluation is a genuine first notification. That is correct behaviour, and the fix is a longer cooldown or a confirmed-recovery requirement rather than a change to the key. The incident-management integration covers requiring a confirmed healthy run before auto-resolve for exactly this reason.

Fingerprints changed and every finding alerted at once

The field list was edited without bumping the version, so every open incident was orphaned and every finding re-alerted as new. Bump FP_VERSION on any change to the field list, expect one burst, and announce it — the alternative is a burst nobody can attribute to anything, which is far more expensive to investigate than to schedule.

FAQ

What must never be included in an alert fingerprint?

Anything that varies between runs: the measured value, the timestamp, the run or build identifier, and the rendered human-readable message, which usually contains the value. Including any of them means the key is unique to a single evaluation, so it never repeats, the cooldown has nothing to match against, and every scoring run opens a fresh incident. The rule is that the key describes what is broken and the payload describes what was observed.

How coarse should the fingerprint be?

At the unit somebody can own and fix. Hashing the rule alone produces one incident spanning every affected section, so fixing one section closes nothing and two teams share an incident neither fully owns. Including the URL produces one incident per page, which turns a single template fault into hundreds of pages and hits the platform rate limit. Rule plus scope plus environment lands where work is actually assigned.

Should severity be part of the fingerprint?

It is a genuine choice rather than a rule. Including it means an escalation from warning to critical opens a new incident, which is useful when the two route to different responders and confusing when the first is left open alongside it. Excluding it means the severity is updated in place on one incident, which is tidier and loses the escalation as a distinct event. Either works, provided the escalation path is explicit about closing or updating rather than leaving both open.