12 min read

Auto-Reverting a Failed robots.txt Deploy

robots.txt is four lines of text with the blast radius of a site-wide outage. A single stray Disallow: / removes an entire estate from crawling, produces no error, breaks no test, and looks completely normal in a code review diff. This page is part of Automating Remediation with Self-Healing Jobs and covers a guard that evaluates the file after every deploy and reverts it automatically when it removes too much.

The diff a revert guard evaluates Two columns showing the same file either side of a deploy. The previous version disallows three specific paths and allows everything else. The new version disallows the site root, which blocks the entire estate. The verdicts state that the previous version permitted the whole indexable set while the new one permits none of it, which is the condition the guard compares against a maximum-loss threshold. PREVIOUS — known good Disallow: /admin/ a specific administrative path Disallow: /cart/ a specific transactional path Disallow: /search a specific parameterised path Everything else allowed 412,000 URLs remain crawlable Loss vs previous: none NEW — deployed Disallow: / the site root No allow rules follow it nothing narrows the block Every path now matches including every product page Nothing remains crawlable 412,000 URLs blocked at once Loss vs previous: 100%

Environment isolation and dependency declaration 

set -euo pipefail
export ROB_URL="https://example.com/robots.txt"
export ROB_KNOWN_GOOD="/data/robots/known_good.txt"     # the last accepted file
export ROB_INVENTORY="/data/inventory/urls_current.parquet"
export ROB_SOFT_LOSS_PCT=2                              # warn above this
export ROB_HARD_LOSS_PCT=25                             # revert above this
export ROB_INVENTORY_MAX_AGE_H=48
export ROB_ALERT_WEBHOOK="https://alerts.internal/hook/robots"
/opt/audit/.venv/bin/pip install "protego==0.3.1" "pandas==2.2.2"

ROB_HARD_LOSS_PCT at twenty-five rather than one hundred is the important choice. A file that blocks a quarter of an estate is very rarely intentional and is far easier to ship by accident than one that blocks everything — an overly broad prefix, a missing Allow: line, a templating variable that rendered empty.

Implementation 

#!/usr/bin/env python3
# /opt/audit/heal/robots_guard.py
# Evaluate a deployed robots.txt against the known-good one and revert
# when the crawlable estate shrinks past the hard threshold.
from __future__ import annotations
import os
import sys
import time
import pandas as pd
import requests
from protego import Protego

UA = "AuditBot"
SOFT = float(os.environ.get("ROB_SOFT_LOSS_PCT", "2"))
HARD = float(os.environ.get("ROB_HARD_LOSS_PCT", "25"))
MAX_AGE_H = int(os.environ.get("ROB_INVENTORY_MAX_AGE_H", "48"))


def load_inventory(path: str) -> list[str]:
    age_h = (time.time() - os.path.getmtime(path)) / 3600
    if age_h > MAX_AGE_H:
        sys.exit(f"FAIL: inventory is {age_h:.0f}h old — refusing to evaluate")
    return pd.read_parquet(path, columns=["url"])["url"].tolist()


def allowed_count(robots_text: str, urls: list[str]) -> int:
    rp = Protego.parse(robots_text)
    return sum(1 for u in urls if rp.can_fetch(u, UA))


def evaluate(new_text: str, good_text: str, urls: list[str]) -> dict:
    before = allowed_count(good_text, urls)
    after = allowed_count(new_text, urls)
    if before == 0:
        sys.exit("FAIL: known-good file allows nothing — refusing to compare")
    loss_pct = (before - after) / before * 100
    return {"before": before, "after": after, "loss_pct": round(loss_pct, 2),
            "verdict": "revert" if loss_pct >= HARD
                       else "warn" if loss_pct >= SOFT else "accept"}

Line by line, the choices that make this safe to run unattended:

  • The comparison is against a URL inventory, not against the file's text. A textual diff cannot tell a harmless reordering from a catastrophic broadening; counting how many real URLs each version permits can.
  • A stale inventory aborts the evaluation rather than proceeding. Measuring loss against a site that has changed produces a number that is wrong in an unpredictable direction, and this guard has the authority to revert on it.
  • A known-good file that allows nothing aborts too. Without that check, a previous bad revert becomes the new baseline and every subsequent broadening reads as zero loss.
  • Protego parses the file the way a crawler would, including Allow precedence and wildcard handling, rather than approximating it with prefix matching — the precedence rules are exactly where an accidental block hides.
  • The verdict is returned, not acted on. The evaluation function has no side effects, which is what lets the same code run in a pre-deploy check, in CI, and in the guard.
What the guard does with a change A question card asks how much of the previously crawlable URL set the new robots file blocks, with three outcomes. Under the soft threshold the change is accepted and recorded. Between the soft and hard thresholds it is accepted but raises a warning for review, because a deliberate broadening looks the same as an accidental one. Above the hard threshold it is reverted automatically to the last known-good version, because no legitimate robots change removes most of a site in one deploy. How much of the crawlable estate does the new file block? under the soft threshold Accept ordinary tightening — recorded, nothing to do soft to hard threshold Accept and warn deliberate and accidental broadening look identical above the hard threshold Revert automatically no legitimate change removes most of a site at once The hard threshold is deliberately not one hundred percent. A file that blocks ninety percent of an estate is as much of an emergency as one that blocks all of it, and it is far easier to ship by accident.

The action half is deliberately separate and always alerts:

#!/usr/bin/env bash
# /opt/audit/heal/robots_revert.sh — called after every deploy.
set -euo pipefail
NEW="$(curl -fsS --max-time 10 "$ROB_URL")"
VERDICT="$(printf '%s' "$NEW" | python3 -m audit.heal.robots_guard --stdin | jq -r .verdict)"

case "$VERDICT" in
  accept) echo "robots change accepted" ;;
  warn)   curl -fsS -X POST "$ROB_ALERT_WEBHOOK"             -d "{\"severity\":\"warning\",\"msg\":\"robots.txt broadened\"}" ;;
  revert)
    # Restore, then fail the deploy — a reverted file inside a shipped
    # release leaves the deploy internally inconsistent.
    aws s3 cp "$ROB_KNOWN_GOOD" "s3://${SITE_BUCKET}/robots.txt" --cache-control "max-age=60"
    curl -fsS -X POST "$ROB_ALERT_WEBHOOK"       -d "{\"severity\":\"critical\",\"msg\":\"robots.txt auto-reverted\"}"
    exit 1 ;;
esac

Verification and smoke test 

/opt/audit/.venv/bin/python3 - <<'PY'
from robots_guard import evaluate
urls = [f"https://example.com/p/{i}" for i in range(1000)] +        ["https://example.com/admin/x", "https://example.com/cart/y"]
good = "User-agent: *\nDisallow: /admin/\nDisallow: /cart/\n"

# 1. A catastrophic broadening is caught.
r = evaluate("User-agent: *\nDisallow: /\n", good, urls)
assert r["verdict"] == "revert", r
print(f"PASS: full block caught — loss {r['loss_pct']}%")

# 2. A harmless reordering is accepted.
r2 = evaluate("User-agent: *\nDisallow: /cart/\nDisallow: /admin/\n", good, urls)
assert r2["verdict"] == "accept", r2
print(f"PASS: reordering accepted — loss {r2['loss_pct']}%")

# 3. A partial broadening lands in the warn band.
r3 = evaluate("User-agent: *\nDisallow: /admin/\nDisallow: /cart/\nDisallow: /p/1\n",
              good, urls)
assert r3["verdict"] in ("accept", "warn"), r3
print(f"PASS: partial broadening -> {r3['verdict']} at {r3['loss_pct']}%")
PY

Expected output is three PASS lines, with the first reporting a loss near one hundred percent. The second assertion matters as much as the first: a guard that reverts on harmless changes will be disabled within a week, and a disabled guard protects nothing.

Failure modes

Three ways the revert misfires Three rows. Evaluating the rules against a stale URL inventory measures loss against a site that no longer exists. Reverting without checking whether the deploy also changed other things restores robots into a codebase that has moved on. And a revert that does not notify anyone lets the same broken file ship again on the next deploy, repeatedly. SYMPTOM ROOT CAUSE FIX Loss is reported as huge on a change that only blocked one small path A stale URL inventory the loss was measured against a site that has since changed Refresh, then evaluate assert inventory freshness before the comparison runs The revert restores a file that no longer matches the rest of the deploy One change among several reverting it alone leaves the deploy internally inconsistent Revert and block restore the file and fail the deploy rather than continuing The same file ships again on the next deploy, and the one after that The revert notified nobody the pipeline healed itself and said nothing about why Page on every revert an automatic revert is an incident, not a maintenance action The third row is why an automatic revert must always alert. A guard that silently undoes the same mistake every day has converted a visible outage into an invisible one.

The guard reverts a deliberate, correct change

Somebody genuinely intended to block a large section — a staging environment promoted by mistake, or a legal takedown. Provide an explicit override that is recorded rather than a threshold that gets loosened: a commit message token or a deploy-time flag that the guard reads and writes into the audit trail, so the decision is attributable.

The known-good file drifts from reality

It is only updated when a change is accepted, so a long series of warn verdicts leaves the baseline months behind. Update the known-good file on every accept, and treat a warn as requiring an explicit accept-or-revert decision rather than as a passive notification that ages quietly.

Loss is measured against a URL set the site no longer serves

The freshness guard above catches the extreme case; the subtle case is an inventory that is current but scoped differently from the estate the robots file governs. Both must cover the same host set, or a change to one subdomain is measured against URLs from another — the same scoping discipline described in defining crawl depth and scope for enterprise sites.

FAQ

Why compare permitted URL counts instead of diffing the file text?

Because a textual diff cannot distinguish a harmless change from a catastrophic one. Reordering two Disallow lines is a large diff and a zero-impact change; adding a single character to turn a specific path into a prefix that matches everything is a tiny diff and a site-wide outage. Counting how many real URLs each version permits measures the thing that actually matters, and it does so in a unit anyone can reason about.

Why set the hard threshold well below one hundred percent?

Because a file blocking a quarter of an estate is as much of an emergency as one blocking all of it, and it is considerably easier to ship by accident. An overly broad path prefix, a missing Allow line, or a templating variable that rendered empty all produce partial blocks that look entirely plausible in review. Setting the hard threshold at a level no legitimate single change would ever reach catches those without waiting for the total case.

Should an automatic revert alert, or is healing silently fine?

It must always alert, at critical severity. A guard that silently undoes the same mistake on every deploy has converted a visible outage into an invisible recurring defect, and the underlying cause — a broken template, a bad merge, a misconfigured environment variable — never gets fixed because nobody knows it is happening. The revert buys time; the alert is what makes the time useful.