Mapping HTTP Status Codes to Health Score Penalties #

Status codes look like the easiest thing in an audit to score: the crawler already has them, they are standardised, and everyone agrees a 500 is worse than a 200. The difficulty is that a status code describes a request rather than a page, and the same code means very different things depending on how the crawler arrived at the URL. This page is part of Health Score Reference Tables and gives a penalty table with the context rules that make it usable.

The penalty table #

Penalties are expressed on a 0-100 indexability scale, where 100 is a total loss of indexability for that URL. They are combined into the composite through the indexability weight described in designing custom health score algorithms.

The 429 row is the one most often got wrong. A rate-limit response says the crawler asked too fast, not that the page is unhealthy — scoring it as a page defect penalises the site for the audit's own behaviour, and the correct response is to fix the rate limiting and re-crawl.

Implementation #

#!/usr/bin/env python3
# /opt/audit/scoring/status_penalty.py
# Penalties are a function of (status, discovery source, hop count) — never
# of the status alone.
from __future__ import annotations
from dataclasses import dataclass

SITEMAP, INTERNAL, EXTERNAL = "sitemap", "internal_link", "external"


@dataclass(frozen=True)
class Observation:
    status: int | None      # None means the request timed out
    source: str             # SITEMAP | INTERNAL | EXTERNAL
    hops: int = 0           # redirect hops before the final status
    in_public_scope: bool = True


def penalty(o: Observation) -> tuple[float, str]:
    """Return (penalty, reason). The reason is emitted with the score so a
    number can always be traced back to the rule that produced it."""
    if o.status is None:
        return 90.0, "timeout"
    if o.status == 429:
        # Not a page defect: the crawl was too aggressive. Flag the run.
        return 0.0, "rate_limited_crawler_fault"
    if 500 <= o.status < 600:
        return 100.0, "server_error"
    if o.status in (404, 410):
        return ({SITEMAP: (100.0, "gone_but_declared"),
                 INTERNAL: (60.0, "broken_internal_link")}
                .get(o.source, (20.0, "expected_attrition")))
    if o.status == 403:
        return 70.0, "forbidden"
    if o.status == 401:
        return (30.0, "auth_required") if o.in_public_scope else (0.0, "gated_expected")
    if o.status == 400:
        return 60.0, "bad_request"
    if o.status in (301, 308):
        extra = max(0, o.hops - 1)
        return 10.0 * (1 + extra), f"permanent_redirect_{o.hops}_hops"
    if o.status in (302, 307):
        return 25.0, "temporary_redirect"
    if o.status in (204, 206):
        return 20.0, "no_indexable_body"
    return 0.0, "ok"

The parts that matter:

• source is a required field, not an optional refinement. A scorer that accepts a status without knowing where the URL came from cannot distinguish a broken promise from expected attrition, and those two deserve a five-fold difference in penalty.
• hops scales the redirect penalty. A single 301 is ordinary site maintenance; four stacked ones is the finding described in fixing broken redirect chains.
• 429 returns zero and a distinguishing reason. The reason string is what lets a downstream check count rate-limited responses and fail the crawl rather than the site.
• Every branch returns a reason. A penalty without a traceable rule is a number nobody can argue with or correct.

Verification and smoke test #

/opt/audit/.venv/bin/python3 - <<'PY'
from status_penalty import Observation, penalty, SITEMAP, INTERNAL, EXTERNAL

# 1. The same status scores differently by source.
p_sitemap, _ = penalty(Observation(404, SITEMAP))
p_link, _    = penalty(Observation(404, INTERNAL))
p_ext, _     = penalty(Observation(404, EXTERNAL))
assert p_sitemap > p_link > p_ext, (p_sitemap, p_link, p_ext)
print(f"PASS: 404 scores {p_sitemap}/{p_link}/{p_ext} by source")

# 2. Redirect penalty grows with hop count.
assert penalty(Observation(301, INTERNAL, hops=1))[0] == 10.0
assert penalty(Observation(301, INTERNAL, hops=4))[0] == 40.0
print("PASS: redirect penalty scales with hops")

# 3. A rate limit is never charged to the page.
pen, reason = penalty(Observation(429, INTERNAL))
assert pen == 0.0 and "crawler" in reason
print(f"PASS: 429 -> {pen} ({reason})")

# 4. Every penalty is bounded and carries a reason.
for s in (200, 204, 301, 302, 400, 401, 403, 404, 500, 503, None):
    p, r = penalty(Observation(s, INTERNAL, hops=1))
    assert 0.0 <= p <= 100.0 and r
print("PASS: all penalties bounded, all carry a reason")
PY

Expected output is four PASS lines. The first is the assertion worth keeping permanently, because it encodes the whole idea of the table: a status code is not scoreable on its own.

Failure modes #

Every 404 in the report is high severity

The discovery source is missing from the crawl export, so every 404 falls into the default branch. Confirm the crawler records how each URL was discovered; without it, no status-based scoring can be more than approximate, and the sensible interim is to score all 404s at the internal-link level rather than the sitemap level.

A rate-limited crawl reports a collapsed health score

429s were scored as page defects. The reason string exists to catch this: count rate_limited_crawler_fault and fail the run when it exceeds a small fraction of requests, rather than publishing a score that describes the crawler's manners.

Redirect penalties dominate the indexability subscore

Hop counts are being taken from the raw request chain including scheme upgrades, so almost every URL shows at least two hops. Count hops after normalising the scheme and trailing slash, or every http:// seed is penalised for a redirect that is the site working correctly.

FAQ #

Why does the same 404 get three different penalties?

Because a status code describes a request and a score has to describe a page. A 404 on a URL the sitemap declares is the site asserting a page exists that does not — a broken promise, and the most serious form. A 404 reached only by an internal link is a broken link, which is a real fault of lesser consequence. A 404 on a URL found only in an old external referrer is ordinary attrition and is not a defect at all. Scoring the code without the source collapses all three into one number.

Should a 429 count against the health score?

No. A rate-limit response says the crawler asked too fast, not that the page is unhealthy, so scoring it as a page defect penalises the site for the audit own behaviour. Return a zero penalty with a distinguishing reason, count those reasons across the run, and fail the crawl when they exceed a small fraction of requests — the correct remedy is throttling and a re-crawl, not a published score describing the crawler manners.

Should the first status or the final status be scored?

Both, separately. The final status describes what a visitor actually receives and is what the page score should reflect. The hops taken to reach it are a distinct finding about redirect hygiene, penalised by depth rather than by the code. Scoring only the first status penalises a correctly redirecting page for redirecting; scoring only the final one hides a four-hop chain behind a healthy 200.

---

Related #

• Health Score Reference Tables — the parent reference covering thresholds, severity tiers, weights and score bands
• Designing Custom Health Score Algorithms — the indexability weight these penalties feed into
• Prioritizing Critical vs Non-Critical Site Errors — ranking the resulting errors by request share rather than raw count