12 min read

Scoring Crawlability Regressions by Blast Radius

A crawl audit can surface a dozen crawlability regressions in a single run — a template that started emitting noindex, a redirect rule that now loops, a robots.txt line that quietly blocked a whole subdirectory. Triaging them by how alarming they sound wastes the response budget on the loudest finding instead of the widest one. This page is part of Risk Scoring Frameworks for Technical Debt and walks through a single scoring script that counts exactly how many indexable URLs each regression affects, so the ranked list tells you which fire to put out first.

Blast Radius Scoring Pipeline Diagram showing a baseline crawl graph passing through regression application, reachability recomputation, and an indexable-set diff, producing a ranked blast-radius score per regression. Baseline Graph BFS from root reachable + indexable apply Regression blocked node/edge noindex / robots / link loss re-run Recompute post-regression BFS new reachable set diff Diff Sets baseline − post affected indexable URLs rank Blast Radius Score + severity sorted, widest first

Environment Isolation and Dependency Declaration

The scoring script only needs the standard library, but pin the interpreter and isolate the working directory anyway — a graph-diffing script that silently runs against the wrong Python minor version is a bad place to discover an ordering bug in set iteration.

# /opt/audit/blast_radius — absolute working directory for this scorer
export PYTHONPATH=/opt/audit/blast_radius
export BLAST_RADIUS_GRAPH=/opt/audit/blast_radius/input/crawl_graph.json
export BLAST_RADIUS_REGRESSIONS=/opt/audit/blast_radius/input/regressions.json
export BLAST_RADIUS_ROOT="https://example.com/"
set -euo pipefail
cd /opt/audit/blast_radius
python3 --version   # pin to 3.11.x in your crawl-runner image
mkdir -p input output

Both input files are produced upstream: crawl_graph.json comes from the same crawl snapshot your risk-scoring intake already consumes, and regressions.json comes from whatever diffed the current crawl against the last one — a template rule change, a robots.txt commit, or a redirect map update.

Implementation

The core idea: compute the set of indexable URLs reachable from the site root before a regression, recompute that same set after applying only that regression's blocked nodes and edges, and subtract. The size of the difference is the blast radius. Doing this as a graph walk instead of a path-prefix match is what catches orphaned pages — URLs that stay technically indexable in isolation but lose every inbound link because the hub page pointing to them lost its outbound links.

#!/usr/bin/env python3
# /opt/audit/blast_radius/score_regressions.py
"""
Score crawlability regressions by blast radius: the number of indexable URLs
that lose crawl reachability or indexability because of one rule change.
Reads BLAST_RADIUS_* env vars, a crawl graph snapshot, and a regressions feed.
"""

import json
import os
from collections import deque

GRAPH_PATH = os.environ["BLAST_RADIUS_GRAPH"]
REGRESSIONS_PATH = os.environ["BLAST_RADIUS_REGRESSIONS"]
ROOT_URL = os.environ["BLAST_RADIUS_ROOT"]

# severity bands: (minimum affected URLs, label) — checked high to low
SEVERITY_BANDS = [
    (5000, "P1-critical"),
    (500, "P2-high"),
    (50, "P3-medium"),
    (0, "P4-low"),
]

# ── 1. load the crawl graph snapshot ─────────────────────────────────────────
with open(GRAPH_PATH) as f:
    graph = json.load(f)
# nodes: {url: {"indexable": bool}}  -- final indexability after robots/noindex/canonical
# edges: {from_url: [to_url, ...]}   -- internal links discovered by the last crawl
nodes = graph["nodes"]
edges = graph["edges"]

def reachable_set(blocked_urls, blocked_edges):
    """BFS from ROOT_URL, skipping blocked nodes/edges, returns reachable URL set."""
    seen = {ROOT_URL}
    queue = deque([ROOT_URL])
    while queue:
        current = queue.popleft()
        for target in edges.get(current, []):
            if target in blocked_urls or (current, target) in blocked_edges:
                continue
            if target not in seen:
                seen.add(target)
                queue.append(target)
    return seen

# ── 2. baseline reachable + indexable set, before any regression ────────────
baseline_reachable = reachable_set(blocked_urls=set(), blocked_edges=set())
baseline_indexable = {
    u for u in baseline_reachable if nodes.get(u, {}).get("indexable", False)
}

# ── 3. load the regressions feed — one entry per detected rule change ───────
with open(REGRESSIONS_PATH) as f:
    regressions = json.load(f)

results = []
for reg in regressions:
    blocked_urls = set(reg.get("directly_blocked_urls", []))
    blocked_edges = {tuple(e) for e in reg.get("removed_edges", [])}

    # recompute reachability with only this regression's blocks applied
    post_reachable = reachable_set(blocked_urls, blocked_edges)
    post_indexable = {
        u for u in post_reachable
        if nodes.get(u, {}).get("indexable", False) and u not in blocked_urls
    }

    # affected = indexable before, no longer indexable/reachable after
    affected = baseline_indexable - post_indexable
    score = len(affected)
    pct_of_index = round(100 * score / max(len(baseline_indexable), 1), 2)
    severity = next(label for floor, label in SEVERITY_BANDS if score >= floor)

    results.append({
        "regression_id": reg["regression_id"],
        "type": reg["type"],
        "affected_url_count": score,
        "pct_of_index": pct_of_index,
        "severity": severity,
    })

# ── 4. rank regressions by blast radius, widest damage first ────────────────
results.sort(key=lambda r: r["affected_url_count"], reverse=True)

out_path = "/opt/audit/blast_radius/output/ranked_regressions.json"
with open(out_path, "w") as f:
    json.dump(results, f, indent=2)

for r in results:
    print(f"{r['severity']:<12} {r['affected_url_count']:>6} urls "
          f"({r['pct_of_index']:>5}% of index)  {r['regression_id']}  [{r['type']}]")
print(f"\nWrote {len(results)} ranked regressions -> {out_path}")

Line-by-line, the two things doing the real work are reachable_set and the affected = baseline_indexable - post_indexable diff. reachable_set is a plain breadth-first search that treats a directly_blocked_urls entry as a node the walk cannot land on (covers noindex_added and robots_disallow regressions) and a removed_edges entry as a link the walk cannot traverse (covers a hub page losing its outbound links, the hub_links_removed case). Running the same BFS twice — once with nothing blocked, once with one regression's blocks applied — and subtracting the indexable sets is what turns "this rule changed" into "this many indexable URLs are now unreachable or unindexable." The severity bands are deliberately count-based rather than percentage-based first, because a 5,000-URL regression on a two-million-URL site is still a P1 even though it is under 1% of the index.

Verification and Smoke Test

Run this immediately after every scoring pass, and again after any change to the graph snapshot or the regressions feed — a passing run on stale input is worse than a failing run on current input.

set -euo pipefail
cd /opt/audit/blast_radius

# 1. execute the scorer
python3 score_regressions.py

# 2. confirm output was written and is non-empty
test -s output/ranked_regressions.json && echo "PASS: output written"

# 3. every regression must be scored — no silent drops
REG_IN=$(python3 -c "import json; print(len(json.load(open('input/regressions.json'))))")
REG_OUT=$(python3 -c "import json; print(len(json.load(open('output/ranked_regressions.json'))))")
[ "$REG_IN" -eq "$REG_OUT" ] && echo "PASS: scored $REG_OUT of $REG_IN regressions" \
  || echo "FAIL: input=$REG_IN scored=$REG_OUT"

# 4. output must be sorted descending by affected_url_count
python3 -c "
import json
rows = json.load(open('output/ranked_regressions.json'))
counts = [r['affected_url_count'] for r in rows]
assert counts == sorted(counts, reverse=True), 'ranking is not sorted descending'
print('PASS: ranking sorted descending')
"

# 5. no negative or impossible counts
python3 -c "
import json
rows = json.load(open('output/ranked_regressions.json'))
bad = [r for r in rows if r['affected_url_count'] < 0]
print('PASS: all counts non-negative') if not bad else print(f'FAIL: {len(bad)} negative counts')
"

Expected output:

PASS: output written
PASS: scored 7 of 7 regressions
PASS: ranking sorted descending
PASS: all counts non-negative

If the sorted-descending check fails, the ranking logic was bypassed downstream — verify nothing between the script and the consumer re-sorts by regression_id or timestamp.

Failure Modes

Crawl graph snapshot is older than the regression it is scoring

If the snapshot predates the rule change, baseline_indexable already reflects the damage and the diff against post_indexable comes back empty or near-empty — a real P1 regression scores as P4-low.

# diagnostic: compare snapshot timestamp to the regression's detected_at
python3 -c "
import json
graph = json.load(open('input/crawl_graph.json'))
regs = json.load(open('input/regressions.json'))
print('graph snapshot_at:', graph.get('snapshot_at'))
print('earliest regression detected_at:', min(r['detected_at'] for r in regs))
"
# fix: re-crawl before re-scoring, never score against a stale snapshot

Overlapping regressions double-count the same URLs

Two regressions that both touch the same subtree — say a broken redirect on a category page and a robots.txt change on the same path — each independently count every URL in that subtree. Summing their scores to estimate total site damage overstates it.

# diagnostic: check pairwise overlap between two regressions' affected sets
python3 -c "
import json
rows = json.load(open('output/ranked_regressions.json'))
# re-run affected-set computation per regression (not shown) and intersect
# before combining totals; never sum affected_url_count across regressions
# without first checking for shared URLs.
"
# fix: score regressions independently for ranking, but compute a single
# combined-remediation estimate by unioning affected sets, not summing scores

Raw count ignores URL value and gets applied as the final priority order

A 200-URL regression on your highest-converting template can matter more than a 2,000-URL regression on a low-value archive. Blast radius by count is a triage input, not the final remediation order.

# diagnostic: join ranked_regressions.json against your health-score value tiers
python3 -c "
import json
rows = json.load(open('output/ranked_regressions.json'))
for r in sorted(rows, key=lambda r: r['affected_url_count'], reverse=True)[:5]:
    print(r['regression_id'], r['affected_url_count'], r['severity'])
"
# fix: multiply affected_url_count by a value weight downstream, in the
# mapping-audit-findings-to-remediation-workflows triage step, not here

FAQ

What counts as an indexable URL for blast radius purposes?

A URL is indexable if it is reachable in the crawl graph, returns a 200 status, has no noindex directive, is not blocked by robots.txt, and is either canonical or self-canonicalizing. Any URL failing one of those checks is excluded from the baseline indexable set before scoring begins, so a regression cannot inflate its own score by counting URLs that were already unindexable.

How is blast radius different from just counting URLs that match the broken rule?

A pattern match only counts URLs directly touched by the rule, such as everything under a path prefix. Blast radius also walks the crawl graph to catch second-order damage: pages that become orphaned because their only inbound links passed through a hub page the regression broke. Pattern matching alone systematically undercounts regressions that damage internal linking rather than individual pages.

Should the score account for URL value or traffic, not just raw count?

Raw count is deliberately the first-pass score because it is cheap to compute and hard to argue with during triage. Once regressions are ranked, multiply the count by a value weight from your health score pipeline before final prioritization, as described in mapping audit findings to remediation workflows. Do not skip the raw count step — value weighting on top of a wrong count just produces a confidently wrong ranking.

How fresh does the crawl graph snapshot need to be?

The snapshot must be no older than the regression it is scoring against. If a redirect broke three days ago and your graph snapshot is a week old, the baseline reachable set predates the break and the diff will either miss the regression entirely or misattribute unrelated drift to it. Re-crawl or re-snapshot before scoring, not after — the same discipline used when fixing broken redirect chains demands a fresh trace, not a cached one.