Monitoring, Alerting & Remediation #

A scoring pipeline that only produces numbers is a reporting tool, not an operations tool. This section covers the layer that turns a health-score regression into a closed incident: evaluating scores against thresholds, routing the confirmed breach to the right owner, running a versioned remediation playbook against the actual finding, and verifying the fix before anything gets marked resolved. The audience is the team that carries the pager for site health — SREs who own the alerting infrastructure, SEO engineers who define what "broken" means for a given section, and agency teams who need a defensible, repeatable response when a client site regresses between audits. None of this replaces the scoring work covered in Metric Scoring & Data Normalization; it consumes that output and decides what to do about it.

The failure mode this section exists to prevent is not "we didn't catch the regression." Most audit pipelines catch regressions fine. The failure mode is catching the same regression five times a day, paging the wrong person, running a fix that doesn't address the root cause, and never confirming whether the fix actually worked. Each of those is a distinct engineering problem — threshold design, routing configuration, playbook authoring, and verification — and each gets its own guide in this section.

---

Architecture: Ingest Scores → Evaluate Thresholds → Route Alert → Execute Playbook → Verify Fix → Close #

The diagram below is the dependency chain every regression travels through, drawn as a loop because a closed incident feeds directly back into the next scoring run rather than terminating the pipeline. Each stage has a corresponding guide covering its configuration.

Three guides configure the three stages that require the most judgment: configuring alert thresholds and routing covers the evaluate-and-route boundary, writing remediation playbooks for audit failures covers the execute stage, and integrating audit alerts with incident management covers wiring the router into an external on-call platform.

---

Phase 1 — Threshold Evaluation #

Problem framing #

A threshold evaluator has one job: decide, for a given metric on a given section, whether the current value represents a real regression or normal variance. Get this wrong in either direction and everything downstream breaks. Set thresholds too tight and every deploy trips a false alarm, which trains the on-call rotation to ignore pages. Set them too loose and a genuine regression sits unaddressed for days because nothing ever crossed the line. Configuring alert thresholds and routing covers the full design space, including rolling-percentile baselines; this phase covers the minimum viable static configuration and the guardrails every threshold set needs regardless of how the boundary is computed.

Thresholds should never live inline in alerting code. Store them as versioned configuration, reviewed the same way you review a schema migration, because a threshold change has the same blast radius as a code change — it silently redefines what "broken" means for every page in a section.

{
  "version": "1.4.0",
  "generated_from": "metric-scoring-data-normalization/thresholds.json",
  "sections": {
    "checkout": {
      "composite_score": {"warning": 80, "critical": 60},
      "lcp_ms":          {"warning": 2500, "critical": 4000},
      "inp_ms":          {"warning": 200,  "critical": 500}
    },
    "content": {
      "composite_score": {"warning": 70, "critical": 50},
      "lcp_ms":          {"warning": 3000, "critical": 5000},
      "inp_ms":          {"warning": 300,  "critical": 600}
    }
  },
  "flap_guard": {
    "min_consecutive_runs": 2,
    "reset_after_hours": 6
  }
}

The flap_guard block is the single highest-leverage setting in this file. A metric that crosses warning on one run and recovers on the next is noise, not a regression, and should never reach the routing stage. Requiring two consecutive breaching runs before evaluation marks an alert eligible eliminates the majority of single-run false positives without meaningfully delaying detection of a real, sustained regression.

Common mistakes:

• Copying thresholds from a different section instead of deriving them from that section's own baseline — a checkout page and a blog post do not share a failure tolerance.
• Setting min_consecutive_runs to 1, which defeats the flap guard entirely and reintroduces single-spike noise.
• Storing thresholds in the alerting service's database instead of version control, which makes a threshold regression invisible to code review.

---

Phase 2 — Alert Routing #

Problem framing #

Once a breach clears the flap guard, routing decides who finds out and how urgently. The two things that go wrong here are duplicate alerts — the same regression paging on-call three times because three metrics on the same page all crossed thresholds in the same run — and routing to the wrong severity of channel, which either buries a real critical breach in a low-priority Slack channel or pages someone at 3 a.m. for a warning-tier issue. Integrating audit alerts with incident management covers wiring this router into PagerDuty or Opsgenie directly; this phase covers the routing configuration those integrations consume.

# routing.yml — reviewed alongside thresholds.json
routes:
  - severity: critical
    channel: "pagerduty:site-health-critical"
    dedup_key_template: "{{section}}:{{metric}}:{{deployment_tag}}"
    cooldown_minutes: 30
    escalation_after_minutes: 15
  - severity: warning
    channel: "slack:#site-health-warnings"
    dedup_key_template: "{{section}}:{{metric}}:{{date}}"
    cooldown_minutes: 120
    escalation_after_minutes: null
defaults:
  timezone: "UTC"
  quiet_hours_paging_override: true   # critical always pages, even in quiet hours

#!/usr/bin/env bash
# evaluate_and_route.sh — reads scores + thresholds, emits routed alerts.
# Pin: jq 1.7.1, curl 8.x. Run once per completed scoring pass.
set -euo pipefail

SCORES_FILE="/var/lib/site-health/scores/latest.json"
THRESHOLDS_FILE="/etc/site-health/thresholds.json"
ROUTING_FILE="/etc/site-health/routing.yml"
DEDUP_STORE="/var/lib/site-health/dedup"

mkdir -p "$DEDUP_STORE"

section="$(jq -r '.section' "$SCORES_FILE")"
metric="$(jq -r '.metric' "$SCORES_FILE")"
value="$(jq -r '.value' "$SCORES_FILE")"
critical="$(jq -r --arg s "$section" --arg m "$metric" '.sections[$s][$m].critical' "$THRESHOLDS_FILE")"

if awk -v v="$value" -v c="$critical" 'BEGIN{exit !(v>=c)}'; then
  dedup_key="${section}:${metric}:$(date -u +%Y%m%d)"
  dedup_file="${DEDUP_STORE}/${dedup_key//[:\/]/_}"

  if [[ -f "$dedup_file" ]] && [[ $(( $(date +%s) - $(stat -c %Y "$dedup_file") )) -lt 1800 ]]; then
    echo "[SKIP] ${dedup_key} within cooldown window"
    exit 0
  fi

  touch "$dedup_file"
  echo "[ROUTE] critical breach ${section}/${metric}=${value} -> pagerduty:site-health-critical"
fi

Common mistakes:

• Building the dedup_key from a timestamp instead of the deployment tag or calendar date, which produces a unique key on every run and defeats deduplication entirely.
• Setting cooldown_minutes shorter than the audit cadence, so every single run re-fires the same alert.
• Forgetting a quiet_hours_paging_override — critical site-health breaches should still page during off-hours; only warning-tier noise should respect quiet hours.

---

Phase 3 — Remediation Playbook Execution #

Problem framing #

A routed critical alert should trigger a specific, versioned fix procedure, not a manual scramble to remember what fixed this last time. Writing remediation playbooks for audit failures covers authoring the playbooks themselves in depth; this phase covers the runner that executes them safely once triggered. Every playbook run must be idempotent — running it twice on an already-fixed target should be a no-op, not a second mutation — and every secret it needs must arrive as an environment variable injected at execution time, never hardcoded in the playbook file, since playbooks are checked into version control alongside application code.

#!/usr/bin/env bash
# run_playbook.sh — executes a matched remediation playbook with guards.
# Pin: flock (util-linux 2.39+). Absolute paths only; no relative cwd assumptions.
set -euo pipefail

FINDING_TYPE="${1:?usage: run_playbook.sh <finding_type> <target_scope>}"
TARGET_SCOPE="${2:?usage: run_playbook.sh <finding_type> <target_scope>}"
DRY_RUN="${DRY_RUN:-false}"
PLAYBOOK_DIR="/opt/site-health/playbooks"
LOCK_DIR="/var/lock/site-health"
STATE_DIR="/var/lib/site-health/playbook-state"

mkdir -p "$LOCK_DIR" "$STATE_DIR"

PLAYBOOK_PATH="${PLAYBOOK_DIR}/${FINDING_TYPE}.sh"
LOCK_FILE="${LOCK_DIR}/${FINDING_TYPE}.lock"
STATE_KEY="${STATE_DIR}/${FINDING_TYPE}__${TARGET_SCOPE//\//_}.done"

if [[ ! -x "$PLAYBOOK_PATH" ]]; then
  echo "[ERROR] no playbook for finding type: ${FINDING_TYPE}" >&2
  exit 1
fi

# Idempotency guard: skip if this exact scope was already remediated this run cycle
if [[ -f "$STATE_KEY" ]]; then
  echo "[SKIP] ${FINDING_TYPE} already remediated for ${TARGET_SCOPE}"
  exit 0
fi

exec 9>"$LOCK_FILE"
flock -n 9 || { echo "[SKIP] another run of ${FINDING_TYPE} is in progress" >&2; exit 0; }

echo "[RUN] ${FINDING_TYPE} on ${TARGET_SCOPE} (dry_run=${DRY_RUN})"
DRY_RUN="$DRY_RUN" TARGET_SCOPE="$TARGET_SCOPE" \
  SITE_API_TOKEN="${SITE_API_TOKEN:?SITE_API_TOKEN must be set}" \
  "$PLAYBOOK_PATH"

if [[ "$DRY_RUN" != "true" ]]; then
  touch "$STATE_KEY"
fi

Idempotency guard: the STATE_KEY file, keyed on finding type and target scope, prevents a duplicate incident notification from triggering the same fix twice in the same cycle. The flock wrapper prevents two concurrent triggers — a manual re-run and an automated retry — from executing the same mutation in parallel.

Common mistakes:

• Writing a playbook that assumes it starts from a known-good state instead of checking current state first — a redirect-collapse script that doesn't verify the current chain length before rewriting can create a new loop.
• Passing secrets as command-line arguments, which leaks them into process listings and shell history; use environment variables exported into the child process only.
• Omitting the DRY_RUN path entirely, which means the first time a new playbook runs against production is also the first time anyone sees what it actually does.

---

Phase 4 — Verification and Auto-Close #

Problem framing #

A playbook that exits 0 has not necessarily fixed anything — it has only completed without an unhandled error. The verification stage is what actually confirms a fix by re-measuring the thing that regressed and comparing it against the pre-incident state. Auto-closing an incident without this check turns the whole loop into theater: a playbook runs, the on-call page stops firing because the alert cooled down, and the underlying finding is still live. This is also where the trend context from tracking metric trends across release cycles becomes operationally useful — a verified fix should show up as a visible recovery in the same trend line that showed the regression.

#!/usr/bin/env bash
# verify_and_close.sh — re-crawls the affected scope and diffs the score.
# Pin: python 3.11+, requests 2.31+ (see requirements.txt).
set -euo pipefail

TARGET_SCOPE="${1:?usage: verify_and_close.sh <target_scope> <incident_id>}"
INCIDENT_ID="${2:?usage: verify_and_close.sh <target_scope> <incident_id>}"
BASELINE_SCORE_FILE="/var/lib/site-health/baselines/${TARGET_SCOPE//\//_}.json"
INCIDENT_API="${INCIDENT_API_BASE:?INCIDENT_API_BASE must be set}"

echo "[VERIFY] re-crawling ${TARGET_SCOPE}"
python3 /opt/site-health/crawl/run_targeted_crawl.py --scope "$TARGET_SCOPE" \
  --output "/tmp/verify_${INCIDENT_ID}.json"

CURRENT_SCORE="$(python3 -c "import json; print(json.load(open('/tmp/verify_${INCIDENT_ID}.json'))['composite_score'])")"
BASELINE_SCORE="$(python3 -c "import json; print(json.load(open('${BASELINE_SCORE_FILE}'))['composite_score'])")"

if awk -v c="$CURRENT_SCORE" -v b="$BASELINE_SCORE" 'BEGIN{exit !(c>=b-2)}'; then
  echo "[CLOSE] ${TARGET_SCOPE} recovered: ${CURRENT_SCORE} vs baseline ${BASELINE_SCORE}"
  curl -sf -X PUT "${INCIDENT_API}/incidents/${INCIDENT_ID}/resolve" \
    -H "Authorization: Bearer ${INCIDENT_API_TOKEN:?INCIDENT_API_TOKEN must be set}" \
    -d "{\"resolution\":\"verified\",\"verified_score\":${CURRENT_SCORE}}"
else
  echo "[HOLD] ${TARGET_SCOPE} not recovered: ${CURRENT_SCORE} vs baseline ${BASELINE_SCORE}" >&2
  exit 1
fi

The -2 tolerance in the comparison exists because a re-crawl is a fresh measurement, not a replay of the original run, and a small amount of run-to-run noise is expected even on a fully fixed page. Requiring the recovered score to be strictly better than that noise band avoids closing an incident on a coincidental partial recovery.

Verification checklist:

1. Confirm the re-crawl actually targeted the affected scope — a typo'd TARGET_SCOPE will silently verify the wrong section.
2. Confirm BASELINE_SCORE_FILE reflects the pre-incident state, not a stale baseline from before the last legitimate threshold change.
3. Confirm the incident record shows resolution: verified, not just resolution: closed — a manually closed incident with no verification tag should not count as a confirmed fix in your incident metrics.

---

Cross-Cutting Concerns #

Retention of alert and incident history #

Retention shorter than a full release cycle makes threshold tuning close to impossible — you cannot tell whether a new warning value reduced noise if the alert history from before the change has already expired.

Versioning playbooks #

Treat every playbook the same way you treat the threshold and severity scoring configuration from the technical debt risk framework: as reviewed, version-controlled artifacts, not ad hoc scripts living on an operator's laptop. Tag each playbook with a semantic version, and log the playbook version alongside every run so a regression introduced by a playbook change is traceable back to the exact commit that shipped it. Never edit a playbook in place while an incident referencing it is still open — branch the fix, verify it against a dry run, and merge before the next trigger.

Environment parity #

Run the same playbook binary and the same threshold configuration against staging and production; a playbook that has only ever executed against staging is unverified in the environment that actually matters. Keep SITE_API_TOKEN, INCIDENT_API_BASE, and every other environment-specific value out of the playbook body entirely — inject them at the runner level so the same playbook file is provably identical across environments, and any divergence in behavior is attributable to configuration, not code drift.

---

Failure Modes and Rollback #

1. Alert storm from a shared root cause Root cause: a single infrastructure event (a CDN outage, a bad deploy) breaches thresholds on dozens of sections simultaneously, and each one routes independently. Diagnostic: grep -c "\[ROUTE\]" /var/log/site-health/router.log for the last 15 minutes — a count far above the historical per-cycle average confirms a storm. Fix: touch /var/lib/site-health/dedup/GLOBAL_SUPPRESS to activate a global suppression window, then fire one aggregated incident manually referencing the root cause instead of the per-section alerts.

2. Flapping metric that never stabilizes Root cause: a metric oscillates around its threshold boundary every run, repeatedly re-opening and auto-closing the same incident. Diagnostic: query the incident store for the same dedup_key prefix resolved and reopened more than twice within 24 hours. Fix: curl -X PATCH "$INCIDENT_API/thresholds/{section}/{metric}" -d '{"min_consecutive_runs": 4}' to temporarily widen the flap guard for that specific metric while the underlying variance is investigated.

3. Stuck incident with no verification result Root cause: the playbook ran and exited 0, but the verification job never ran — a scheduling gap, a crashed worker, or a missing INCIDENT_API_TOKEN. Diagnostic: curl -s "$INCIDENT_API/incidents?resolution=none&age_hours_gt=6" — any result here is a stuck incident. Fix: re-run verify_and_close.sh manually with the incident's original TARGET_SCOPE; if verification still fails, escalate to a human rather than letting the incident age silently.

4. Playbook that worsens the underlying state Root cause: the playbook's fix logic had an edge case — for example a redirect-collapse script that created a new loop instead of removing one. Diagnostic: the verification diff shows CURRENT_SCORE below BASELINE_SCORE, not above it — this should never pass the -2 tolerance check in Phase 4. Fix: invoke the playbook's own rollback path (every playbook must define one) — e.g. PLAYBOOK_ROLLBACK=true /opt/site-health/playbooks/redirect_chain.sh "$TARGET_SCOPE" — then set that finding type to require manual approval before its next automated trigger.

5. Dedup key collision across unrelated findings Root cause: the dedup_key_template omits the metric name, so two different regressions on the same section collapse into a single alert and one gets silently swallowed. Diagnostic: compare the count of distinct (section, metric) breach pairs in the raw score feed against the count of distinct incidents opened in the same window — a mismatch confirms collision. Fix: update routing.yml to include {{metric}} in every dedup_key_template, then redis-cli DEL "dedup:*" (or the equivalent key-namespace flush) to clear stale keys built from the old template.

---

FAQ #

What is the difference between an alert and an incident?

An alert is a threshold evaluation result: a metric crossed a configured boundary on a given run. An incident is what you get after routing decides the alert is real and worth a responder's attention — it carries a dedup key, an owner, a timeline, and a resolution state. Most alerts should never become incidents; that filtering is the job of the threshold and routing stages, not the on-call engineer.

Should every threshold breach open an incident?

No. Require a minimum number of consecutive breaching runs before an alert is even eligible to route, and reserve incident creation for the severity tiers that map to a paging channel. A single-run spike on a noisy metric like CLS is usually measurement noise, not a regression, and should log to the alert history without waking anyone.

How long should alert and incident history be retained?

Keep raw alert events for at least 90 days so you can compute flap rates and dedup effectiveness, and keep resolved incident records for a year or more so severity-scoring changes can be validated against real outcomes. Retention shorter than one full release cycle makes it impossible to tell whether a new threshold actually reduced noise.

What happens if a remediation playbook makes the problem worse?

A correctly written playbook captures a before-state snapshot and a rollback command before it touches anything, so reverting is a single documented step rather than a manual recovery effort. Treat any playbook run that increases the health-score delta as a stop condition: halt automatic re-runs on that finding type, roll back, and require a human sign-off before the playbook is allowed to fire again.

---

Related #

• Configuring Alert Thresholds and Routing — static severity tiers, rolling-percentile baselines, and channel routing
• Writing Remediation Playbooks for Audit Failures — parameterized, idempotent fix procedures with dry-run and rollback
• Integrating Audit Alerts with Incident Management — wiring the router into PagerDuty, Opsgenie, and Slack with auto-resolve
• Metric Scoring & Data Normalization — the upstream scoring pipeline that feeds every threshold evaluation in this section
• Automated Crawling Pipeline & Tooling — the crawl infrastructure that verification re-crawls depend on