17 min read

Automating Remediation with Self-Healing Jobs

The last stage of an audit pipeline is the one most teams never build. Findings are detected, scored, routed and ticketed — and then a human reads a runbook and types the same three commands they typed last month. Automating that step is worth doing and is also the single most dangerous thing in the whole pipeline, because it is the only component with write access to production. This page is part of Monitoring, Alerting & Remediation and covers building jobs that fix findings unattended without ever being the reason for an outage.

What a job may fix on its own A question card asks two things about the finding: whether the fix is fully reversible and whether the blast radius is bounded and known in advance. A reversible fix with a bounded radius runs unattended. A reversible fix with an unbounded radius is queued for approval, because the reversal is only cheap if somebody notices it is needed. An irreversible fix never runs unattended regardless of radius, because there is no rollback to fall back on. Is the fix reversible, and is the blast radius bounded in advance? reversible, bounded Run unattended the whole point of automation — small, known, undoable reversible, unbounded Queue for approval a rollback only helps if someone notices in time irreversible Never unattended there is nothing to fall back to when it is wrong Reversibility is the load-bearing property. A wrong fix that can be undone in one command is an incident; a wrong fix that cannot is a migration nobody planned.

Prerequisites & environment setup

Tool / library Pinned version Purpose
Python 3.11.9 Job runtime and dry-run diffing
PyYAML 6.0.1 Job definitions, version-controlled
flock (util-linux) 2.39 Per-finding-type and per-target concurrency guards
jq 1.7.x Snapshot and audit-trail inspection 
export SH_JOB_DIR="/opt/audit/self-healing/jobs"
export SH_SNAPSHOT_DIR="/data/self-healing/snapshots"
export SH_AUDIT_TRAIL="/data/self-healing/trail"
export SH_DRY_RUN=1                    # the default, always
export SH_MAX_TARGETS=25               # hard cap per invocation
export SH_DEPLOY_STATE_URL="https://deploy.internal/api/in-flight"
export SH_REQUIRE_SCORE_DELTA=1        # verify the score moved, not just the finding

SH_DRY_RUN=1 as the exported default is not a convenience. It means that every way of invoking a job that does not deliberately override it produces a diff rather than a change, so a misconfigured job, a stray cron entry or a copy-pasted command previews instead of mutating.

Step 1 — The job definition

A job is a YAML document plus a thin runner. Everything about its behaviour is in the document, so a change to what a job does is a reviewable diff rather than a change to a script somebody has to read.

# /opt/audit/self-healing/jobs/collapse-redirect-chain.yaml
name: collapse-redirect-chain
finding_code: REDIRECT_CHAIN_DEPTH
version: 3
reversible: true
blast_radius: bounded          # bounded | unbounded
max_targets: 25
target_scope: "path_prefix:/products/"
stages:
  pre_check:  "audit.heal.checks:finding_still_present"
  snapshot:   "audit.heal.snap:capture_redirect_rules"
  dry_run:    "audit.heal.redirect:plan_collapse"
  apply:      "audit.heal.redirect:apply_collapse"
  verify:     "audit.heal.checks:single_hop_and_score_moved"
  rollback:   "audit.heal.redirect:restore_snapshot"
guards:
  deploy_in_flight: block
  lock: "per_finding_type_and_target_prefix"

reversible and blast_radius are declared rather than inferred, and the runner refuses to run unattended unless both are favourable. Declaring them makes the decision reviewable: someone changing blast_radius from unbounded to bounded is making a claim in a pull request, not adjusting a flag.

Six stages, three ways to stop Six stacked stages. Pre-check confirms the finding is still present and no conflicting deploy is in flight. Snapshot records the exact state before anything changes. Dry run produces a diff that is compared against the expected blast radius. Apply makes the change only if the diff is within the cap. Verify re-fetches and confirms the finding is gone and the score moved. Close or roll back is the terminal stage, and rolling back is a first-class outcome rather than an error path. Pre-check finding still present, no deploy in flight stops here if the finding self-resolved Snapshot exact before-state, checksummed without it, rollback is guesswork Dry run produce the diff, compare to the cap stops here if the radius exceeds the cap Apply only within the cap, only once idempotent, so a retry is harmless Verify re-fetch, confirm gone, confirm moved stops here if the score did not move Close or roll back both are successful outcomes rollback is not an error path Three of the six stages can terminate the run without a fix being applied, and that ratio is deliberate: a self-healing job that always reaches the apply stage is not checking anything.

Step 2 — Core configuration parameters

Parameter Type Default Purpose
dry_run bool true Preview only; overridden solely by an explicit --live
max_targets int 25 Hard cap; the run aborts rather than truncating
target_scope string — The blast radius, resolved before any command runs
reversible bool — Declared per job; gates unattended execution
require_score_delta bool true Verify the score moved, not only that the finding is gone
deploy_in_flight enum block What to do when a release is running
lock enum — Concurrency guard scope: finding type, target prefix, or both

require_score_delta deserves its default. A verification that only checks the finding is gone will happily close a run that fixed a symptom — the redirect chain is collapsed and the page still takes four seconds, because the chain was never the reason. Requiring a measurable score movement turns that into a held-open incident instead of a closed ticket.

Step 3 — Execution: the runner 

#!/usr/bin/env bash
# /opt/audit/self-healing/run.sh — invoked by the alert router, not by cron.
set -euo pipefail
JOB="${1:?job name required}"
LIVE="${2:-}"

DEF="${SH_JOB_DIR}/${JOB}.yaml"
LOCK="/var/run/heal-$(yq -r '.finding_code' "$DEF").lock"

exec 9>"$LOCK"
flock --nonblock 9 || { echo "another run of this finding type is active"; exit 0; }

# A release in flight is a hard block: a self-healing job must never fight a deploy.
if curl -fsS "$SH_DEPLOY_STATE_URL" | jq -e '.in_flight == true' >/dev/null; then
  echo "deploy in flight — deferring"; exit 0
fi

MODE="--dry-run"
[ "$LIVE" = "--live" ] && MODE="--live"

python3 -m audit.heal.runner --job "$DEF" "$MODE"   --snapshot-dir "$SH_SNAPSHOT_DIR"   --trail "$SH_AUDIT_TRAIL/$(date -u +%F)/${JOB}.jsonl"

The runner is invoked by the alert router after a finding has survived confirmation and cooldown, never on a schedule of its own. A scheduled self-healing job runs whether or not there is anything to fix, which means its most common execution path is the one nobody tests.

Step 4 — Artifact capture: the audit trail

Every run writes one record whether or not it changed anything, carrying the job version, the resolved target list, the before and after checksums, the dry-run diff, the verification result and the terminal state. Retention is 180 days minimum — longer than the crawl artifacts themselves, because this is the record a post-incident review reads, and the question it answers is not "what does the site look like now" but "what did we change, when, and on what evidence".

Idempotency is the property that makes retries free

Every stage in the sequence above assumes it can be run again. A network blip between apply and verify, an orchestrator that retries a failed step, an operator who reruns a job because they were not sure it took — all of these are ordinary, and all of them re-enter a stage that has already run. If the fix is idempotent, none of them matter. If it is not, each one is a fresh way to produce a state nobody designed.

Idempotency is not a property of the change itself but of how the change is applied. The mechanical rule is: read the target's current state, compare it against the desired state, and skip the target if they already match.

# /opt/audit/heal/redirect.py — the shape every apply stage should have.
from __future__ import annotations

def apply_collapse(targets: list[str], desired: dict[str, str],
                   *, live: bool) -> dict[str, int]:
    applied = skipped = failed = 0
    for url in targets:
        current = read_current_rule(url)          # live read, never cached
        want = desired[url]
        if current == want:
            skipped += 1                          # already in the desired state
            continue
        if not live:
            applied += 1                          # counted, not written
            continue
        try:
            write_rule(url, want)
            applied += 1
        except Exception:
            failed += 1
    return {"applied": applied, "skipped": skipped, "failed": failed}

The skipped counter is more useful than it looks. A rerun of a successful job should report every target as skipped and none as applied — which is a positive, checkable assertion that the previous run actually took effect, obtained for free from a mechanism that already had to exist.

The same discipline applies to the rollback path. Restoring a snapshot has to be safe to run twice, because the most likely moment for a rollback to be interrupted is exactly when the environment is already unhealthy.

Where automated remediation should stop

Not every recurring finding should end in a job, and deciding which ones should is a judgement worth making explicitly rather than by accumulation. Three tests are useful:

  • Does it recur? A finding that has appeared three times in a quarter is a candidate. One that has appeared once is a fix, not an automation problem — building a job for it costs more than the job will ever save.
  • Is the fix mechanical? A remediation that is the same three commands every time automates well. One where the operator reads the situation and chooses between options does not, and encoding that judgement into a job usually means encoding the most common case and silently mishandling the rest.
  • Is the finding trustworthy? Automating a fix for a finding that produces false positives means automating a change to pages that were never broken. Get the finding's false-positive rate down first, using the diagnostics in identifying false positives in automated audits, and only then attach a job to it.

A finding that fails any of these three is better served by a well-written playbook that a human runs. That is not a lesser outcome — a playbook with a dry-run diff and a rollback path already removes most of the risk, and it keeps the judgement where the judgement belongs.

Verification checklist

  1. A dry run produces a non-empty diff for a finding that is genuinely active. An empty diff means the pre-fix read came from a cache rather than from live state.
  2. The resolved target count is inside max_targets. A run that hit the cap must be treated as a partial fix and re-scheduled, never as a completed one.
  3. The snapshot exists and differs from the after-state. Identical checksums mean the apply stage did nothing.
  4. The score moved. Compare the composite before and after; a finding that disappeared without a score change is a symptom fix.
  5. Rollback works from the snapshot alone. Exercise it deliberately in staging — a rollback path that has never run is a rollback path that does not work.
  6. The audit-trail write succeeded. A run that cannot record what it did must exit non-zero, regardless of whether the fix itself worked.

Troubleshooting

Four ways automation makes it worse Four rows. A non-idempotent fix retried after a network blip applies its change twice and produces a state nobody designed. An unbounded target scope turns a ten-URL fix into an estate-wide rewrite. A rollback that itself fails leaves the site in a half-changed state worse than either endpoint. And a job that fires while a deploy is in flight fights the deployment and loses in a way that looks like a deploy failure. SYMPTOM ROOT CAUSE FIX A change applied twice producing a state nobody wrote a rule for The fix is not idempotent a retry after a blip re-applied it on top of itself Read before writing skip targets already in the desired state Ten URLs were expected and forty thousand were rewritten Unbounded target scope a glob matched far more than the finding covered Cap the run refuse to apply when the dry-run diff exceeds the cap A half-changed state after a rollback that itself failed midway Rollback was an afterthought no pre-checks, no snapshot, no verification of its own Treat it as a stage with the same rigour as the forward fix The job fought a deploy and the deploy is blamed for the outcome No in-flight check the pre-check never asked whether a release was running Gate on deploy state the same suppression signal the alert router already reads Every row is a property of the harness rather than of the fix. A perfectly correct redirect rule applied twice, or to forty thousand URLs, or during a deploy, is still an incident.

A job fires repeatedly for the same finding

The verification stage is passing but the finding is regenerating, which means the underlying cause is upstream — a CMS template or a CDN rule producing the bad state again. Treat a repeat within 24 hours as an escalation past the playbook rather than as an occasion to run it again, exactly as writing remediation playbooks for audit failures describes.

The dry-run diff is enormous and nobody noticed until apply

The cap was checked after the apply stage rather than before it. The comparison belongs between dry run and apply, and exceeding it must abort rather than truncate — truncating produces a half-applied change across an arbitrary subset, which is worse than either applying all of it or none.

Two jobs collided on overlapping targets

The lock is scoped per finding type, and two different finding types both touch the same URL prefix. Add the target-prefix lock alongside the finding-type one so overlapping scopes serialise, and accept the throughput cost.

A run reports success with no audit-trail record

The trail write failed and its exit code was swallowed. Make the write a hard requirement of a successful exit: a job that fixed something and cannot say what it fixed should be treated as failed, because from a reviewer's perspective it is indistinguishable from one that did nothing.

FAQ

Which findings are safe to fix unattended?

Ones where the fix is fully reversible and the blast radius is bounded and known before the run starts. Reversibility is the load-bearing property: a wrong change that can be undone with one command is an incident, and a wrong change that cannot is an unplanned migration. A reversible fix with an unbounded radius should be queued for approval instead, because a rollback only helps if somebody notices in time to use it.

Why verify a score change rather than just that the finding is gone?

Because the finding disappearing proves the specific condition was addressed, not that anything got better. A collapsed redirect chain on a page that still takes four seconds satisfies the finding and changes nothing a user experiences, which means the chain was never the reason. Requiring a measurable score movement turns that case into a held-open incident with evidence attached, instead of a closed ticket and a problem that resurfaces under a different finding code.

Should self-healing jobs run on a schedule?

No. They should be invoked by the alert router after a finding has survived confirmation and its cooldown window, so a run only ever happens when there is something to fix. A scheduled job runs whether or not a finding exists, which means its most frequently exercised code path is the one where there is nothing to do — the path nobody writes tests for and nobody watches.

What should happen when the target count exceeds the cap?

The run must abort, not truncate. Truncating applies the change to an arbitrary subset of targets, which leaves the site in a state nobody designed and is harder to reason about than either applying everything or applying nothing. Aborting surfaces the scope problem — which is almost always a glob matching more than the finding covered — and lets a human decide whether the cap or the scope is wrong.