13 min read

Routing Audit Alerts to PagerDuty

A threshold breach detected by the scoring pipeline is only useful to on-call if it becomes exactly one incident — not a new page every time the next scoring run re-confirms the same problem, and not silence when the problem clears. This page is part of Integrating Audit Alerts with Incident Management and walks through the specific mechanics of the PagerDuty Events API v2: how to build a stable dedup_key, when to fire a trigger event versus a resolve event, and how to verify the pairing behaves correctly before it sits on the critical path for paging a human at 3 a.m.

The pattern assumes alert evaluation already happened upstream — this page does not decide whether a breach is real. That decision belongs to the threshold logic described in Configuring Alert Thresholds and Routing; by the time code here runs, a confirmed regression has already cleared cooldowns and severity checks and needs to become a page.

PagerDuty Trigger/Resolve Flow with Shared dedup_key Diagram showing two parallel flows. Top: a confirmed breach builds a dedup_key from metric and section, then POSTs a trigger event to the Events API v2, opening one PagerDuty incident. Bottom: a later confirmed recovery builds the same dedup_key and POSTs a resolve event, closing that same incident. Both flows converge on one incident box. Breach confirmed score < threshold Build dedup_key hash(metric, section) POST trigger event_action=trigger /v2/enqueue Recovery confirmed score >= threshold Same dedup_key hash(metric, section) POST resolve event_action=resolve /v2/enqueue One incident opened, then closed same dedup_key

Environment Isolation and Dependency Declaration

Keep the routing key and the HTTP client version pinned before any trigger logic runs. A floating requests version is low-risk here, but an accidentally hardcoded routing key in the script body is not — it belongs only in the environment.

# /opt/audit/pd_alerts — absolute working directory
export PYTHONPATH=/opt/audit/pd_alerts
export PD_ROUTING_KEY="R0XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"   # PagerDuty integration key, per-service
# requirements.txt
requests==2.32.3

set -euo pipefail
cd /opt/audit/pd_alerts
python3 -m venv .venv
.venv/bin/pip install --quiet -r requirements.txt

PD_ROUTING_KEY is an integration key, scoped to one PagerDuty service, not the account-wide REST API token. Events API v2 rejects the REST token outright, and mixing the two credentials in one script is the single most common setup mistake teams hit when they wire this in for the first time.

Implementation

The core problem is state without a database: the audit pipeline runs on a schedule, has no persistent record of "is this incident already open," and still needs trigger/resolve semantics to line up correctly across runs. PagerDuty solves this by tracking incident identity entirely by dedup_key on its side — the client only needs to compute the same key every time it re-encounters the same failure.

#!/usr/bin/env python3
# /opt/audit/pd_alerts/route_to_pagerduty.py
"""
Send a confirmed site-health breach to PagerDuty as a trigger event, then
send a matching resolve event once the same metric/section recovers. The
dedup_key ties the pair together so PagerDuty collapses repeated triggers
for the same underlying failure into one open incident.
"""

import hashlib
import os
import requests

# ── 1. routing key selects the PagerDuty service; never hardcode this ───────
ROUTING_KEY = os.environ["PD_ROUTING_KEY"]
EVENTS_URL = "https://events.pagerduty.com/v2/enqueue"


def build_dedup_key(metric: str, section: str) -> str:
    """
    Deterministic key: the SAME (metric, section) pair always hashes to the
    SAME dedup_key, regardless of which scoring run detects it or what the
    score value happened to be. This is what lets PagerDuty recognize a
    re-triggered breach as "still the same incident" instead of a new one.
    """
    raw = f"{metric}:{section}".encode("utf-8")
    return hashlib.sha256(raw).hexdigest()[:32]


def send_trigger(metric: str, section: str, score: float, threshold: float) -> str:
    """Fire once per scoring run while the breach remains confirmed."""
    dedup_key = build_dedup_key(metric, section)
    severity = "critical" if score < threshold * 0.8 else "warning"
    payload = {
        "routing_key": ROUTING_KEY,
        "event_action": "trigger",
        "dedup_key": dedup_key,
        "payload": {
            "summary": f"{metric} breached threshold in {section}: {score:.1f} < {threshold:.1f}",
            "source": f"audit-pipeline/{section}",
            "severity": severity,
            "component": metric,
            "group": section,
            # custom_details renders on the incident page — give on-call
            # enough to act without re-running the audit themselves.
            "custom_details": {
                "score": score,
                "threshold": threshold,
                "section": section,
            },
        },
    }
    resp = requests.post(EVENTS_URL, json=payload, timeout=10)
    resp.raise_for_status()
    print(f"trigger sent dedup_key={dedup_key} status={resp.status_code}")
    return dedup_key


def send_resolve(metric: str, section: str) -> str:
    """
    Fire once a LATER scoring run confirms the same metric/section is back
    within threshold. No payload block is required for a resolve event —
    only routing_key, event_action, and the matching dedup_key.
    """
    dedup_key = build_dedup_key(metric, section)
    payload = {
        "routing_key": ROUTING_KEY,
        "event_action": "resolve",
        "dedup_key": dedup_key,
    }
    resp = requests.post(EVENTS_URL, json=payload, timeout=10)
    resp.raise_for_status()
    print(f"resolve sent dedup_key={dedup_key} status={resp.status_code}")
    return dedup_key


if __name__ == "__main__":
    # a scoring run confirms a regression on checkout LCP
    send_trigger(metric="lcp_p75", section="checkout", score=42.0, threshold=70.0)
    # a later scoring run confirms recovery for the SAME metric/section
    send_resolve(metric="lcp_p75", section="checkout")

Two details carry the whole design. First, build_dedup_key never touches the score value or a timestamp — only the identity of what's broken. If it hashed the score too, every run with a slightly different number would mint a new key and a new incident, defeating the entire point of deduplication. Second, send_trigger is safe to call on every scoring run for as long as the breach persists: PagerDuty treats a repeated trigger with an already-open dedup_key as an update to the existing incident, not a new page, so calling it repeatedly does not flood on-call.

Verification and Smoke Test

Confirm the pairing works against the live endpoint before wiring it into the scheduled pipeline. A 202 response means PagerDuty accepted the event into its processing queue — it does not by itself confirm the incident opened, so the script also checks status in the response body.

set -euo pipefail

# 1. fire a synthetic trigger with a throwaway dedup_key
curl -sS -o /tmp/pd_trigger.json -w "%{http_code}" \
  -X POST https://events.pagerduty.com/v2/enqueue \
  -H "Content-Type: application/json" \
  -d '{
        "routing_key": "'"$PD_ROUTING_KEY"'",
        "event_action": "trigger",
        "dedup_key": "smoke-test-lcp-checkout",
        "payload": {
          "summary": "smoke test trigger",
          "source": "audit-pipeline/smoke-test",
          "severity": "warning"
        }
      }' > /tmp/pd_status.txt

STATUS=$(cat /tmp/pd_status.txt)
[ "$STATUS" = "202" ] && echo "PASS: trigger accepted (202)" || echo "FAIL: got $STATUS"
python3 -c "
import json
d = json.load(open('/tmp/pd_trigger.json'))
assert d['status'] == 'success', d
print('PASS: dedup_key =', d['dedup_key'])
"

# 2. resolve the SAME dedup_key and confirm it is accepted too
curl -sS -o /dev/null -w "resolve status: %{http_code}\n" \
  -X POST https://events.pagerduty.com/v2/enqueue \
  -H "Content-Type: application/json" \
  -d '{
        "routing_key": "'"$PD_ROUTING_KEY"'",
        "event_action": "resolve",
        "dedup_key": "smoke-test-lcp-checkout"
      }'

Expected output:

PASS: trigger accepted (202)
PASS: dedup_key = smoke-test-lcp-checkout
resolve status: 202

After the smoke test, check the PagerDuty service's incident list directly in the UI — the synthetic incident should show as triggered and then resolved within seconds, with the custom_details you sent visible on the incident timeline. If the incident never appears at all, the routing key points at the wrong service rather than the request being malformed.

Failure Modes

Routing key is valid but points at the wrong service

A 202 response from /v2/enqueue only confirms the event was accepted for processing — it does not confirm which service received it. If PD_ROUTING_KEY was copied from the wrong integration, events queue silently against a service nobody watches.

# diagnostic: list the integration's recent events via the REST API
# (requires the separate account API token, not the routing key)
curl -sS -H "Authorization: Token token=$PD_API_TOKEN" \
  "https://api.pagerduty.com/services/$PD_SERVICE_ID/integrations" | \
  python3 -c "import json,sys; [print(i['id'], i['name']) for i in json.load(sys.stdin)['integrations']]"
# fix: regenerate the integration key for the correct service and re-export
export PD_ROUTING_KEY="<correct-integration-key>"

A trigger is sent but the matching resolve never fires

If the code path that calls send_resolve never runs — a deploy that only wires send_trigger, or an exception thrown before recovery logic executes — the incident stays open indefinitely. PagerDuty has no way to know the underlying metric recovered unless told explicitly.

# diagnostic: find dedup_keys with a trigger but no matching resolve
# in your own audit-alert log (Events API itself does not expose this list)
grep "trigger sent" /var/log/audit/pd_alerts.log | awk '{print $NF}' | sort -u > /tmp/triggered.txt
grep "resolve sent" /var/log/audit/pd_alerts.log | awk '{print $NF}' | sort -u > /tmp/resolved.txt
comm -23 /tmp/triggered.txt /tmp/resolved.txt
# fix: manually resolve the stuck dedup_key once the metric is confirmed healthy
curl -sS -X POST https://events.pagerduty.com/v2/enqueue \
  -H "Content-Type: application/json" \
  -d '{"routing_key":"'"$PD_ROUTING_KEY"'","event_action":"resolve","dedup_key":"<stuck-key>"}'

dedup_key collisions merge unrelated failures into one incident

If build_dedup_key is simplified to hash only the metric name — dropping section — then a checkout LCP regression and a blog LCP regression collapse into the same incident. The on-call responder resolves the checkout issue, and the resolve event closes the incident even though the blog page is still failing.

# diagnostic: confirm the dedup_key includes both dimensions
python3 -c "
from route_to_pagerduty import build_dedup_key
print(build_dedup_key('lcp_p75', 'checkout'))
print(build_dedup_key('lcp_p75', 'blog'))
"
# these two keys must differ — if they match, the hash input is missing 'section'

FAQ

What is the difference between the routing_key and the PagerDuty API token?

The routing_key (also called an integration key) is a per-service key that only the Events API v2 accepts, and it can only trigger, acknowledge, or resolve events. The account-level API token is a separate credential used for the REST API — listing incidents, managing schedules, editing services. Never use the REST API token in the events payload; PagerDuty will reject it, and using it there also grants the script far more scope than it needs.

How do I choose a dedup_key so retries don't create duplicate incidents?

Derive the dedup_key from the identity of the failure, not from the event's timestamp or payload contents. Hashing metric name plus section, as shown in build_dedup_key, means every scoring run that re-detects the same broken pair sends the same key, and PagerDuty collapses repeat triggers into the existing open incident instead of opening a new one each run. This mirrors the deterministic keying already used upstream in Configuring Alert Thresholds and Routing.

Does PagerDuty auto-resolve an incident if my pipeline never sends a resolve event?

No. PagerDuty has no visibility into your underlying metric and will leave the incident open indefinitely unless you explicitly send an event_action of resolve with the matching dedup_key, or an on-call responder resolves it manually in the UI. Treat resolve as a mandatory second half of every trigger you send, not an optional cleanup step — and see Writing Remediation Playbooks for Audit Failures for wiring the resolve call to the same run that verifies a fix.

Can I attach custom details so on-call has enough context to act without re-running the audit?

Yes. The payload.custom_details object accepts arbitrary key-value pairs, and PagerDuty renders them on the incident detail page. Include the observed score, the threshold it breached, the affected section, and a pointer to the relevant remediation procedure so the responder can act immediately instead of digging through dashboards first.