12 min read

Writing Idempotent Remediation Jobs That Are Safe to Retry

Any job with write access to production will eventually be run twice. An orchestrator retries a step that timed out after the write landed; a network blip loses the response to a request that succeeded; an operator reruns something because they could not tell whether it worked. None of these are unusual, and all of them re-enter a stage that already ran. This page is part of Automating Remediation with Self-Healing Jobs and covers the property that makes all of them harmless.

Run it twice and read the counters Two columns describing a second invocation of a job that already succeeded. Under an idempotent apply, every target reads as already in the desired state, so the run reports twenty-five skipped, zero applied and no writes issued — which doubles as proof the first run took effect. Under a non-idempotent apply, the same second run rewrites all twenty-five targets, stacking a second change on top of the first and producing a state the rule was never written for. IDEMPOTENT APPLY — second run skipped: 25 already in the desired state applied: 0 nothing was written No origin writes issued the run is free and harmless Proves the first run took a positive, checkable assertion A retry is a no-op NON-IDEMPOTENT APPLY — second run applied: 25 again, on top of itself skipped: 0 current state was never read 25 origin writes issued each one stacking on the last Produces an undesigned state a rule appended to a rule A retry is a second incident

Environment isolation and dependency declaration 

set -euo pipefail
export IDEM_STATE_READ_MODE="live"        # live | cached — live, always
export IDEM_MARKER="# managed-by: audit-heal"   # the upsert key in config files
export IDEM_MAX_TARGETS=25
export IDEM_ASSERT_RERUN=1                # rerun after apply and assert all-skipped

IDEM_ASSERT_RERUN turns the property from an intention into a test. With it set, a live run applies its changes and then immediately re-runs its own apply stage, asserting that the second pass reports every target as skipped. It costs one extra pass and it is the only way to know idempotency holds for the fix you actually shipped rather than the one you reasoned about.

Implementation 

#!/usr/bin/env python3
# /opt/audit/heal/idempotent.py
# The shape every apply stage should have: read, compare, skip or write, count.
from __future__ import annotations
import os
import re

MARKER = os.environ.get("IDEM_MARKER", "# managed-by: audit-heal")
BLOCK_RE = re.compile(
    rf"^{re.escape(MARKER)} (?P<key>\S+)\n(?:.*?\n)*?# end-managed \1\n",
    re.M)


def upsert_block(config: str, key: str, body: str) -> str:
    """Replace a managed block, or append one. Running this twice with the
    same arguments produces a byte-identical result — which is the point."""
    block = f"{MARKER} {key}\n{body.rstrip()}\n# end-managed {key}\n"
    existing = [m for m in BLOCK_RE.finditer(config) if m.group("key") == key]
    if existing:
        m = existing[0]
        return config[:m.start()] + block + config[m.end():]
    sep = "" if config.endswith("\n") or not config else "\n"
    return config + sep + block


def apply_targets(targets: dict[str, str], read, write, *, live: bool) -> dict:
    """targets: {target_id: desired_state}. read/write are injected so the
    same function is testable without touching an origin."""
    applied = skipped = failed = 0
    for target, desired in targets.items():
        current = read(target)              # live read, never cached
        if current == desired:
            skipped += 1
            continue
        if not live:
            applied += 1                    # counted for the dry-run diff
            continue
        try:
            write(target, desired)
            applied += 1
        except Exception:
            failed += 1
    return {"applied": applied, "skipped": skipped, "failed": failed}

The parts that matter:

  • upsert_block replaces rather than appends. Appending is the single most common way a remediation job stops being idempotent: the second run adds a second rule matching the same request, and which one wins now depends on file order rather than on intent. The marker comment is what makes replacement possible, and it has to be written by the first run so the second can find it.
  • read and write are injected. This is not only for testing — it is what lets the dry-run path use the same comparison logic as the live path, so the diff a reviewer sees is produced by the code that will actually run.
  • skipped is counted and returned. A rerun of a successful job should report every target skipped, which is a checkable assertion that the first run took effect, obtained from a mechanism that had to exist anyway.
  • A failure increments a counter rather than aborting the loop. One unwritable target should not prevent the other twenty-four from being fixed, and the counter surfaces the partial outcome to the verification stage.
The shape of every safe apply Four steps left to right. Read the current state from the origin live, never from a cache. Compare it against the desired state for that target. Skip the target when they already match, or write when they do not. Count each outcome separately so a rerun can be asserted on rather than assumed. Read live current state, never cached Compare against the desired state Skip or write never write unconditionally Count applied, skipped, failed Every non-idempotent apply skips the first two steps. That is the whole difference: it writes because it was asked to, rather than because the target needed it.

Verification and smoke test 

/opt/audit/.venv/bin/python3 - <<'PY'
from idempotent import upsert_block, apply_targets

# 1. Upserting twice is byte-identical.
cfg = "server { listen 80; }\n"
once  = upsert_block(cfg, "redirect-products", "location = /old { return 301 /new; }")
twice = upsert_block(once, "redirect-products", "location = /old { return 301 /new; }")
assert once == twice, "second upsert changed the file"
assert once.count("managed-by") == 1, "block was appended rather than replaced"
print("PASS: upsert is idempotent")

# 2. A second apply over already-correct state writes nothing.
state = {"a": "old", "b": "old"}
desired = {"a": "new", "b": "new"}
r1 = apply_targets(desired, state.get, lambda k, v: state.__setitem__(k, v), live=True)
r2 = apply_targets(desired, state.get, lambda k, v: state.__setitem__(k, v), live=True)
assert r1["applied"] == 2 and r1["skipped"] == 0, r1
assert r2["applied"] == 0 and r2["skipped"] == 2, r2
print(f"PASS: first run {r1}, second run {r2}")
PY

Expected output:

PASS: upsert is idempotent
PASS: first run {'applied': 2, 'skipped': 0, 'failed': 0}, second run {'applied': 0, 'skipped': 2, 'failed': 0}

Both assertions belong in CI permanently. They are fast, deterministic, and they fail the moment somebody replaces the upsert with an append or drops the pre-write read for performance.

Failure modes

Three operations that only look safe Three rows. An append to a configuration file is not idempotent, because the second run appends again — it must be replaced with an upsert keyed on an identifier. A counter increment applied per finding double-counts on every retry and must be derived from state rather than accumulated. And a fix that reads its current state from a cached response can see stale state, decide the target still needs changing, and rewrite something that was already correct. OPERATION WHY IT IS NOT IDEMPOTENT WHAT TO DO INSTEAD Append to a config file the rule is added to the end of the file A second run appends again two rules now match the same request, and order decides Upsert by identifier replace the block keyed on a stable marker comment Increment a counter per finding remediated, once per run A retry counts twice the number is an accumulation, not a measurement Derive it from state count what is currently true, never what was done Read state through a cache to decide whether the target needs changing Stale state reads as unfixed so the job rewrites a target that was already correct Force a live read no-cache on the pre-fix read, always The third is the one that turns an otherwise idempotent job back into a non-idempotent one, because the comparison it depends on is being made against the wrong version of reality.

The rerun assertion fails on a job that looks correct

Almost always a state read that is not byte-comparable with the desired state — trailing whitespace, key ordering in a serialised object, or a timestamp the origin adds on write. Normalise both sides before comparing rather than loosening the assertion; a comparison that ignores differences will eventually ignore a real one.

Two jobs both manage the same config block

Two markers with different keys writing overlapping rules produces a file where both blocks are idempotent individually and the combination is not. Key the marker on the finding type and enforce, in review, that one config region has one owner.

The job is idempotent and the origin is not

Some origins treat a write as an event rather than a state change — appending to a rule list, or bumping a version on every PUT regardless of content. Where that is the case, the read-compare-skip pattern is doing its job and the remaining non-idempotency is the origin's; the mitigation is to keep the skip rate high so writes are rare, and to record the origin's behaviour in the job definition so nobody assumes otherwise. This is worth confirming before a job is promoted from manual playbook to unattended execution.

FAQ

Why is appending to a config file not idempotent?

Because the second run appends again. The file then contains two rules matching the same request, and which one takes effect depends on file order rather than on anything anyone decided. Worse, the state is now one the fix logic never anticipated, so a third run may produce a third rule. Writing a marker-delimited block that a later run finds and replaces makes repeated application produce a byte-identical file.

Why not just track which targets have been fixed?

Because that turns a stateless job into a stateful one, and the state can be wrong. A record saying a target was fixed does not survive the target being changed back by a deploy, a rollback or another job, and a job that trusts its own ledger will skip a target that genuinely needs fixing. Reading the current state is both simpler and correct by construction: the target is either in the desired state or it is not, and nothing else needs remembering.

What if the origin itself is not idempotent?

Then the read-compare-skip pattern is doing everything it can and the remaining risk belongs to the origin. Some systems treat every write as an event — appending to a rule list, or bumping a version on each PUT regardless of content. The mitigation is to keep the skip rate high so writes are genuinely rare, and to record the behaviour explicitly in the job definition so nobody promotes it to unattended execution on the assumption that a retry is free.