12 min read

Failing a Pull Request on a Health Score Regression

A scheduled crawl tells you the site regressed last week. A pull-request gate tells you which change did it, before it ships — which is a much more useful thing to know and a much harder thing to measure, because a sampled crawl of a preview environment carries real variance. This page is part of Integrating Custom Crawlers with CI/CD Pipelines and covers building that gate so it fires on regressions and not on noise.

The PR crawl is not the baseline Two columns. The pull-request crawl samples fifty to a hundred representative template URLs, must finish inside the review loop, is compared against the same sample from the base branch, and never contributes to the trend series. The scheduled crawl covers the full URL set, runs off-peak under a concurrency guard, is compared against its own history, and is the only series percentile baselines are derived from. PULL-REQUEST CRAWL 50-100 sampled URLs one or two per template, fixed Minutes, not hours or reviewers stop reading it Compared against the base branch the same sample, both sides Never feeds the trend a sample would bias the series Answers: did this change hurt? SCHEDULED CRAWL The full URL set every page in audit scope Off-peak, concurrency-guarded nothing is waiting on it Compared against its own history the rolling baseline Is the trend series baselines are derived here Answers: how is the site doing?

Environment isolation and dependency declaration 

set -euo pipefail
export PR_SAMPLE_FILE="audit/pr_sample_urls.txt"   # pinned, in version control
export PR_BASE_ARTIFACT="s3://audit/base/${GITHUB_BASE_REF}/latest.parquet"
export PR_NOISE_FLOOR=1.5        # measured, in composite score points
export PR_BLOCK_DELTA=5.0        # fail the check past this
export PR_OVERRIDE_LABEL="audit-override"
export PR_MAX_MINUTES=8

PR_NOISE_FLOOR has to be measured rather than assumed. Run the same commit through the gate five times against the same preview environment, take the spread of the composite score, and set the floor above it. A threshold below the noise floor produces a gate that fires on changes touching nothing but copy, and a gate that does that is muted within a fortnight.

What the gate does with a delta A question card asks how the measured score change compares with the noise floor of the sampled crawl, with three outcomes. A delta inside the noise floor passes silently, because the gate cannot distinguish it from run-to-run variance. A delta outside the noise floor but inside the block threshold posts a comment and passes, so the author sees it without being stopped. A delta past the block threshold fails the check and requires an explicit override that is recorded. How large is the delta compared with the noise floor? inside the noise floor Pass silently the gate cannot tell this from run-to-run variance outside noise, under block Comment and pass the author sees it without being stopped past the block threshold Fail the check overridable, but the override is recorded Measuring the noise floor first is what makes the gate credible. A gate that fires on variance is disabled within a fortnight, and a disabled gate blocks nothing.

Implementation 

# .github/workflows/audit-gate.yml
name: Health score gate
on:
  pull_request:
    paths: ['src/**', 'templates/**', 'public/**', 'next.config.js']

concurrency:
  group: audit-gate-${{ github.head_ref }}
  cancel-in-progress: true          # only the newest push matters

jobs:
  gate:
    runs-on: ubuntu-24.04
    timeout-minutes: 12
    steps:
      - uses: actions/checkout@v4

      - name: Wait for the preview deployment
        run: ./audit/wait_for_preview.sh "${{ github.event.pull_request.head.sha }}"

      - name: Crawl the pinned sample
        run: |
          python3 -m audit.crawl             --urls "$PR_SAMPLE_FILE"             --base-url "$PREVIEW_URL"             --output pr.parquet             --timeout-minutes "$PR_MAX_MINUTES"

      - name: Compare against the base branch
        id: compare
        run: python3 -m audit.gate.compare --head pr.parquet                --base "$PR_BASE_ARTIFACT" --json > delta.json

      - name: Comment the delta
        uses: actions/github-script@v7
        with:
          script: |
            const d = require('./delta.json');
            const body = `**Health score:** ${d.head.toFixed(1)} `
              + `(${d.delta >= 0 ? '+' : ''}${d.delta.toFixed(1)} vs base)\n\n`
              + d.rows.map(r => `- \`${r.template}\`: ${r.delta.toFixed(1)}`).join('\n');
            await github.rest.issues.createComment({
              issue_number: context.issue.number, owner: context.repo.owner,
              repo: context.repo.repo, body });

      - name: Fail on a real regression
        run: |
          DELTA=$(jq -r '.delta' delta.json)
          OVERRIDE='${{ contains(github.event.pull_request.labels.*.name, 'audit-override') }}'
          python3 - "$DELTA" "$OVERRIDE" <<'PY'
          import os, sys
          delta, override = float(sys.argv[1]), sys.argv[2] == "true"
          floor = float(os.environ["PR_NOISE_FLOOR"])
          block = float(os.environ["PR_BLOCK_DELTA"])
          if delta > -floor:
              print(f"PASS: delta {delta:+.1f} is inside the noise floor")
          elif delta > -block:
              print(f"WARN: delta {delta:+.1f} exceeds noise but is under the block threshold")
          elif override:
              print(f"OVERRIDE: delta {delta:+.1f} accepted via label")
          else:
              sys.exit(f"FAIL: delta {delta:+.1f} exceeds the block threshold of -{block}")
          PY

The parts that decide whether anyone trusts this:

  • concurrency with cancel-in-progress so a series of pushes does not queue five crawls against one preview environment, which would both waste the window and make the results inconsistent.
  • The sample file is pinned in version control. Changing it is a deliberate rebaseline that shows up as a diff, not something that drifts between runs.
  • The comment is posted before the pass/fail step. An author whose check fails should already have the per-template breakdown in front of them, rather than having to open a job log to find out which template moved.
  • The override is a label, so it is recorded on the pull request and visible to a reviewer. An override with no trace is indistinguishable from a gate nobody ran.

Verification and smoke test 

set -euo pipefail
# 1. Measure the noise floor before trusting any threshold.
for i in 1 2 3 4 5; do
  python3 -m audit.crawl --urls "$PR_SAMPLE_FILE" --base-url "$PREVIEW_URL"     --output "noise_$i.parquet" >/dev/null
done
python3 -c "
import pyarrow.parquet as pq, statistics as st
scores = [pq.read_table(f'noise_{i}.parquet').column('composite').to_pylist()[0]
          for i in range(1, 6)]
spread = max(scores) - min(scores)
print(f'spread over 5 identical runs: {spread:.2f} points')
assert spread < float('${PR_NOISE_FLOOR}'), 'noise floor is set too low'
print('PASS: noise floor is above the observed spread')"

# 2. A deliberately regressed branch must fail the gate.
python3 -m audit.gate.compare --head fixtures/regressed.parquet   --base fixtures/base.parquet --json | jq -e '.delta < -5' >/dev/null   && echo "PASS: a known regression is detected"

Expected output is two PASS lines, with an observed spread comfortably below the configured floor. Re-run the first check whenever the sample or the preview environment changes — noise is a property of the harness, and the harness moves.

Failure modes

Three ways the gate loses trust Three rows. A gate with no measured noise floor fires on ordinary run-to-run variance and is muted. A sample that changes between runs compares two different populations and reports a regression that is entirely the sample. And a gate with no override path forces authors to merge around it, which removes the signal entirely rather than just this once. SYMPTOM ROOT CAUSE FIX The gate fires on changes that touch no rendering code at all No measured noise floor the threshold is smaller than run-to-run variance Measure it first run the same commit five times and set the floor above the spread A regression appears with no plausible cause in the diff The sample moved two different URL sets were compared against each other Pin the sample in version control, and change it as a deliberate rebaseline Authors merge around it routinely, and nobody reads the output No override path so the only way past a false positive is to bypass the gate Add a recorded override a label or a token, written into the audit trail Each row ends the same way: the gate stops being read. A health gate only works while the team believes its output, which makes credibility a design constraint rather than a nice-to-have.

The gate is slower than the review

A crawl that takes twenty minutes is a crawl nobody waits for. Cut the sample rather than the depth: one or two URLs per template is enough to catch a template-level regression, and template-level regressions are what a pull request can actually cause.

The base artifact is stale

The gate compares against whatever was last published for the base branch, which after a quiet fortnight may predate several merges. Refresh the base artifact on every merge to the base branch, and fail the gate when the artifact is older than a configured maximum rather than comparing against it silently.

Every pull request shows a small regression

The preview environment is systematically slower than the base environment — usually cold caches, or a preview build without production optimisations. Compare like with like by crawling the base branch's own preview rather than a production artifact, and treat the resulting delta as the measurement. This is the same environment-parity problem covered in normalizing performance data across device types, applied to environments rather than devices.

FAQ

Why measure a noise floor instead of picking a threshold?

Because a sampled crawl of a preview environment has real run-to-run variance, and a threshold below it produces a gate that fires on pull requests touching nothing but copy. That is fatal, because the gate is only useful while people read its output — a check that fires on noise is muted or bypassed within a fortnight, and a muted gate blocks nothing at all. Running the same commit five times and setting the floor above the observed spread makes the first failure a real one.

Should the pull-request crawl feed the trend series?

No. It is a sample of one or two URLs per template, selected for speed rather than representativeness of the whole estate, and it runs against a preview environment rather than production. Feeding it into the baseline would bias the trend toward whichever templates happen to be in the sample and toward preview-environment performance. The scheduled full crawl is the trend series; the pull-request crawl answers a different question about a specific change.

Should there be a way to override the gate?

Yes, and it should be recorded. Without an override, the only way past a false positive is to bypass the check entirely — which removes the signal permanently rather than for one pull request, and teaches everyone that the gate is an obstacle rather than information. An explicit label that appears on the pull request and lands in the audit trail keeps the escape hatch visible and countable, so a rising override rate becomes its own finding.