15 min read

Routing Audit Alerts to PagerDuty

A threshold breach detected by the scoring pipeline is only useful to on-call if it becomes exactly one incident — not a new page every time the next scoring run re-confirms the same problem, and not silence when the problem clears. This page is part of Integrating Audit Alerts with Incident Management and walks through the specific mechanics of the PagerDuty Events API v2: how to build a stable dedup_key, when to fire a trigger event versus a resolve event, and how to verify the pairing behaves correctly before it sits on the critical path for paging a human at 3 a.m.

The pattern assumes alert evaluation already happened upstream — this page does not decide whether a breach is real. That decision belongs to the threshold logic described in Configuring Alert Thresholds and Routing; by the time code here runs, a confirmed regression has already cleared cooldowns and severity checks and needs to become a page.

PagerDuty Trigger/Resolve Flow with Shared dedup_key Diagram showing two parallel flows. Top: a confirmed breach builds a dedup_key from metric and section, then POSTs a trigger event to the Events API v2, opening one PagerDuty incident. Bottom: a later confirmed recovery builds the same dedup_key and POSTs a resolve event, closing that same incident. Both flows converge on one incident box. Breach confirmed score < threshold Build dedup_key hash(metric, section) POST trigger event_action=trigger /v2/enqueue Recovery confirmed score >= threshold Same dedup_key hash(metric, section) POST resolve event_action=resolve /v2/enqueue One incident opened, then closed same dedup_key

Environment Isolation and Dependency Declaration

Keep the routing key and the HTTP client version pinned before any trigger logic runs. A floating requests version is low-risk here, but an accidentally hardcoded routing key in the script body is not — it belongs only in the environment.

# /opt/audit/pd_alerts — absolute working directory
export PYTHONPATH=/opt/audit/pd_alerts
export PD_ROUTING_KEY="R0XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"   # PagerDuty integration key, per-service
# requirements.txt
requests==2.32.3

set -euo pipefail
cd /opt/audit/pd_alerts
python3 -m venv .venv
.venv/bin/pip install --quiet -r requirements.txt

PD_ROUTING_KEY is an integration key, scoped to one PagerDuty service, not the account-wide REST API token. Events API v2 rejects the REST token outright, and mixing the two credentials in one script is the single most common setup mistake teams hit when they wire this in for the first time.

Implementation

The core problem is state without a database: the audit pipeline runs on a schedule, has no persistent record of "is this incident already open," and still needs trigger/resolve semantics to line up correctly across runs. PagerDuty solves this by tracking incident identity entirely by dedup_key on its side — the client only needs to compute the same key every time it re-encounters the same failure.

One incident, two calls, one key Two columns describing the two Events API calls that make up an incident lifecycle. The trigger carries event_action trigger, the dedup key hashed from metric, section and environment, the severity, and the evidence in custom details. The resolve carries event_action resolve and must reuse the identical dedup key; its severity and payload are irrelevant. Both verdicts note that the dedup key is the only field that must match exactly. TRIGGER event_action: trigger opens the incident, or updates it dedup_key: hash(metric, section, env) never the score, never a timestamp severity: critical | error | warning selects the escalation layer custom_details: values and run id the evidence a responder reads Identity decides the incident RESOLVE event_action: resolve closes that specific incident dedup_key: byte-identical the only field that has to match severity: ignored the platform does not read it here Never sent automatically PagerDuty cannot see your metric The same key, or nothing closes 
#!/usr/bin/env python3
# /opt/audit/pd_alerts/route_to_pagerduty.py
"""
Send a confirmed site-health breach to PagerDuty as a trigger event, then
send a matching resolve event once the same metric/section recovers. The
dedup_key ties the pair together so PagerDuty collapses repeated triggers
for the same underlying failure into one open incident.
"""

import hashlib
import os
import requests

# ── 1. routing key selects the PagerDuty service; never hardcode this ───────
ROUTING_KEY = os.environ["PD_ROUTING_KEY"]
EVENTS_URL = "https://events.pagerduty.com/v2/enqueue"


def build_dedup_key(metric: str, section: str) -> str:
    """
    Deterministic key: the SAME (metric, section) pair always hashes to the
    SAME dedup_key, regardless of which scoring run detects it or what the
    score value happened to be. This is what lets PagerDuty recognize a
    re-triggered breach as "still the same incident" instead of a new one.
    """
    raw = f"{metric}:{section}".encode("utf-8")
    return hashlib.sha256(raw).hexdigest()[:32]


def send_trigger(metric: str, section: str, score: float, threshold: float) -> str:
    """Fire once per scoring run while the breach remains confirmed."""
    dedup_key = build_dedup_key(metric, section)
    severity = "critical" if score < threshold * 0.8 else "warning"
    payload = {
        "routing_key": ROUTING_KEY,
        "event_action": "trigger",
        "dedup_key": dedup_key,
        "payload": {
            "summary": f"{metric} breached threshold in {section}: {score:.1f} < {threshold:.1f}",
            "source": f"audit-pipeline/{section}",
            "severity": severity,
            "component": metric,
            "group": section,
            # custom_details renders on the incident page — give on-call
            # enough to act without re-running the audit themselves.
            "custom_details": {
                "score": score,
                "threshold": threshold,
                "section": section,
            },
        },
    }
    resp = requests.post(EVENTS_URL, json=payload, timeout=10)
    resp.raise_for_status()
    print(f"trigger sent dedup_key={dedup_key} status={resp.status_code}")
    return dedup_key


def send_resolve(metric: str, section: str) -> str:
    """
    Fire once a LATER scoring run confirms the same metric/section is back
    within threshold. No payload block is required for a resolve event —
    only routing_key, event_action, and the matching dedup_key.
    """
    dedup_key = build_dedup_key(metric, section)
    payload = {
        "routing_key": ROUTING_KEY,
        "event_action": "resolve",
        "dedup_key": dedup_key,
    }
    resp = requests.post(EVENTS_URL, json=payload, timeout=10)
    resp.raise_for_status()
    print(f"resolve sent dedup_key={dedup_key} status={resp.status_code}")
    return dedup_key


if __name__ == "__main__":
    # a scoring run confirms a regression on checkout LCP
    send_trigger(metric="lcp_p75", section="checkout", score=42.0, threshold=70.0)
    # a later scoring run confirms recovery for the SAME metric/section
    send_resolve(metric="lcp_p75", section="checkout")

Two details carry the whole design. First, build_dedup_key never touches the score value or a timestamp — only the identity of what's broken. If it hashed the score too, every run with a slightly different number would mint a new key and a new incident, defeating the entire point of deduplication. Second, send_trigger is safe to call on every scoring run for as long as the breach persists: PagerDuty treats a repeated trigger with an already-open dedup_key as an update to the existing incident, not a new page, so calling it repeatedly does not flood on-call.

Verification and Smoke Test

Confirm the pairing works against the live endpoint before wiring it into the scheduled pipeline. A 202 response means PagerDuty accepted the event into its processing queue — it does not by itself confirm the incident opened, so the script also checks status in the response body.

set -euo pipefail

# 1. fire a synthetic trigger with a throwaway dedup_key
curl -sS -o /tmp/pd_trigger.json -w "%{http_code}" \
  -X POST https://events.pagerduty.com/v2/enqueue \
  -H "Content-Type: application/json" \
  -d '{
        "routing_key": "'"$PD_ROUTING_KEY"'",
        "event_action": "trigger",
        "dedup_key": "smoke-test-lcp-checkout",
        "payload": {
          "summary": "smoke test trigger",
          "source": "audit-pipeline/smoke-test",
          "severity": "warning"
        }
      }' > /tmp/pd_status.txt

STATUS=$(cat /tmp/pd_status.txt)
[ "$STATUS" = "202" ] && echo "PASS: trigger accepted (202)" || echo "FAIL: got $STATUS"
python3 -c "
import json
d = json.load(open('/tmp/pd_trigger.json'))
assert d['status'] == 'success', d
print('PASS: dedup_key =', d['dedup_key'])
"

# 2. resolve the SAME dedup_key and confirm it is accepted too
curl -sS -o /dev/null -w "resolve status: %{http_code}\n" \
  -X POST https://events.pagerduty.com/v2/enqueue \
  -H "Content-Type: application/json" \
  -d '{
        "routing_key": "'"$PD_ROUTING_KEY"'",
        "event_action": "resolve",
        "dedup_key": "smoke-test-lcp-checkout"
      }'

Expected output:

PASS: trigger accepted (202)
PASS: dedup_key = smoke-test-lcp-checkout
resolve status: 202

After the smoke test, check the PagerDuty service's incident list directly in the UI — the synthetic incident should show as triggered and then resolved within seconds, with the custom_details you sent visible on the incident timeline. If the incident never appears at all, the routing key points at the wrong service rather than the request being malformed.

Failure Modes

Routing key is valid but points at the wrong service

What a 202 does not tell you Three rows. A valid routing key pointing at the wrong service still returns 202, so the smoke test must be verified in the service incident list rather than by status code alone. A trigger with no matching resolve leaves an incident open indefinitely, because PagerDuty has no visibility into the underlying metric. A dedup key hashed from the metric name alone collides across sections, merging a checkout regression and a blog regression into one incident that neither team fully owns. SYMPTOM ROOT CAUSE FIX 202 accepted, nobody paged the incident is nowhere in the service you expected Right key, wrong service an integration key is scoped to one service and 202 says nothing Verify in the UI check the service incident list, not the HTTP status Incidents never close even long after the metric has fully recovered No resolve is ever sent PagerDuty cannot see the metric and will wait indefinitely Wire the resolve path the same run that confirms recovery must emit event_action resolve Two teams, one incident a checkout and a blog regression merged together The dedup key is too coarse hashing the metric name alone drops the section from identity Include the section identity is metric plus section plus environment, always A 202 from the enqueue endpoint means the event was accepted for processing — nothing more. It does not confirm the service, the escalation policy, or that a human will ever see it.

A 202 response from /v2/enqueue only confirms the event was accepted for processing — it does not confirm which service received it. If PD_ROUTING_KEY was copied from the wrong integration, events queue silently against a service nobody watches.

# diagnostic: list the integration's recent events via the REST API
# (requires the separate account API token, not the routing key)
curl -sS -H "Authorization: Token token=$PD_API_TOKEN" \
  "https://api.pagerduty.com/services/$PD_SERVICE_ID/integrations" | \
  python3 -c "import json,sys; [print(i['id'], i['name']) for i in json.load(sys.stdin)['integrations']]"
# fix: regenerate the integration key for the correct service and re-export
export PD_ROUTING_KEY="<correct-integration-key>"

A trigger is sent but the matching resolve never fires

If the code path that calls send_resolve never runs — a deploy that only wires send_trigger, or an exception thrown before recovery logic executes — the incident stays open indefinitely. PagerDuty has no way to know the underlying metric recovered unless told explicitly.

# diagnostic: find dedup_keys with a trigger but no matching resolve
# in your own audit-alert log (Events API itself does not expose this list)
grep "trigger sent" /var/log/audit/pd_alerts.log | awk '{print $NF}' | sort -u > /tmp/triggered.txt
grep "resolve sent" /var/log/audit/pd_alerts.log | awk '{print $NF}' | sort -u > /tmp/resolved.txt
comm -23 /tmp/triggered.txt /tmp/resolved.txt
# fix: manually resolve the stuck dedup_key once the metric is confirmed healthy
curl -sS -X POST https://events.pagerduty.com/v2/enqueue \
  -H "Content-Type: application/json" \
  -d '{"routing_key":"'"$PD_ROUTING_KEY"'","event_action":"resolve","dedup_key":"<stuck-key>"}'

dedup_key collisions merge unrelated failures into one incident

If build_dedup_key is simplified to hash only the metric name — dropping section — then a checkout LCP regression and a blog LCP regression collapse into the same incident. The on-call responder resolves the checkout issue, and the resolve event closes the incident even though the blog page is still failing.

# diagnostic: confirm the dedup_key includes both dimensions
python3 -c "
from route_to_pagerduty import build_dedup_key
print(build_dedup_key('lcp_p75', 'checkout'))
print(build_dedup_key('lcp_p75', 'blog'))
"
# these two keys must differ — if they match, the hash input is missing 'section'

FAQ

What is the difference between the routing_key and the PagerDuty API token?

The routing_key (also called an integration key) is a per-service key that only the Events API v2 accepts, and it can only trigger, acknowledge, or resolve events. The account-level API token is a separate credential used for the REST API — listing incidents, managing schedules, editing services. Never use the REST API token in the events payload; PagerDuty will reject it, and using it there also grants the script far more scope than it needs.

How do I choose a dedup_key so retries don't create duplicate incidents?

Derive the dedup_key from the identity of the failure, not from the event's timestamp or payload contents. Hashing metric name plus section, as shown in build_dedup_key, means every scoring run that re-detects the same broken pair sends the same key, and PagerDuty collapses repeat triggers into the existing open incident instead of opening a new one each run. This mirrors the deterministic keying already used upstream in Configuring Alert Thresholds and Routing.

Does PagerDuty auto-resolve an incident if my pipeline never sends a resolve event?

No. PagerDuty has no visibility into your underlying metric and will leave the incident open indefinitely unless you explicitly send an event_action of resolve with the matching dedup_key, or an on-call responder resolves it manually in the UI. Treat resolve as a mandatory second half of every trigger you send, not an optional cleanup step — and see Writing Remediation Playbooks for Audit Failures for wiring the resolve call to the same run that verifies a fix.

Can I attach custom details so on-call has enough context to act without re-running the audit?

Yes. The payload.custom_details object accepts arbitrary key-value pairs, and PagerDuty renders them on the incident detail page. Include the observed score, the threshold it breached, the affected section, and a pointer to the relevant remediation procedure so the responder can act immediately instead of digging through dashboards first.