13 min read

Capturing a First-Crawl Baseline Snapshot

Every later audit on a new domain answers one question: what changed since the beginning? That question is unanswerable unless the beginning was captured deterministically. Teams that skip this step end up comparing quarter two against quarter one and quietly reconstructing what "one" must have looked like from memory and old spreadsheets — a process that manufactures false regressions and false clean bills of health in equal measure. This page is part of Establishing Baseline Health Metrics for New Domains and covers the narrow, mechanical part of that workflow: running the very first crawl, checksumming its output, and writing a manifest that cannot drift.

The goal is not just "a crawl happened." The goal is an artifact you could hand to someone in eighteen months, along with its manifest, and have them reconstruct with certainty which bytes were audited, which tool version produced them, and when. Without that, every future comparison — including the trend analysis covered in tracking metric trends across release cycles — is built on an assumption instead of a fact.

First-crawl baseline capture pipeline Four sequential stages: a pinned crawl writes compressed output to a scratch directory, the artifact is hashed with SHA-256, an immutable manifest.json is written recording the hash and tool versions, and only then is the verified pair promoted atomically into the permanent baseline directory. Pinned crawl scratch/ output fixed image digest crawl.jsonl.gz SHA-256 checksum hash artifact write .sha256 file before any copy manifest.json hash, versions, url count, run_ts chmod 444 Baseline dir atomic promote read-only diff target promotion only proceeds if checksum + manifest write both succeed

Why the naive approach fails

The instinct is to run the crawler and save whatever it produces straight into a folder called baseline/. Three things go wrong with that shortcut. First, if the crawl is interrupted — a timeout, a rate-limit stall, a container OOM kill — you now have a partial file sitting in the one location every future audit assumes is complete. Second, without a recorded tool version, a crawler upgrade six months later changes what "the same crawl" even means, and a real regression can hide inside what looks like a tooling-caused diff. Third, without a checksum captured at the moment of creation, you have no way to prove the file in cloud storage today is bit-for-bit the file the crawler wrote — a corrupted upload or a well-meaning teammate's manual edit is indistinguishable from the original.

The fix is mechanical, not clever: crawl into a scratch location, hash it, write an immutable record of what you hashed, and only then promote the pair into the permanent baseline path — mirroring the discipline used when storing and versioning crawl artifacts in cloud storage.

Implementation

The block below does the entire job for a single domain: it runs a pinned crawl into a scratch directory, computes a SHA-256 digest of the compressed output, writes a manifest.json recording everything a future audit needs to trust the artifact, locks the manifest read-only, and only then promotes the verified pair into the permanent baseline path.

#!/usr/bin/env bash
# /opt/audit/bin/capture-baseline.sh
# Usage: capture-baseline.sh example.com
set -euo pipefail

DOMAIN="${1:?usage: capture-baseline.sh <domain>}"
AUDIT_HOME="/opt/audit"
CRAWLER_IMAGE="ghcr.io/your-org/technical-audit:3.1.2@sha256:8f2c1e0a9b7d4c6e5f3a2b1c0d9e8f7a6b5c4d3e2f1a0b9c8d7e6f5a4b3c2d1e"
RUN_TS="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
SCRATCH_DIR="${AUDIT_HOME}/scratch/${DOMAIN}-${RUN_TS//[:]/}"
BASELINE_DIR="${AUDIT_HOME}/baselines/${DOMAIN}"

mkdir -p "$SCRATCH_DIR"

# 1. Run the crawl against a scratch path — never write directly into
#    the baseline directory, so a failed run can never corrupt an
#    existing snapshot. Rendering is pinned off for determinism on the
#    first pass; JS-rendered crawls are captured as a separate,
#    explicitly labelled artifact.
docker run --rm \
  -e CRAWL_DOMAIN="$DOMAIN" \
  -e CRAWL_RENDER_JS="false" \
  -e CRAWL_MAX_DEPTH="0" \
  -v "${SCRATCH_DIR}:/out" \
  "$CRAWLER_IMAGE" \
  --output /out/crawl.jsonl.gz

# 2. Checksum the artifact immediately, before it is copied, uploaded,
#    or touched by anything else. This digest is the ground truth this
#    entire baseline exists to protect.
sha256sum "${SCRATCH_DIR}/crawl.jsonl.gz" \
  | awk '{print $1}' > "${SCRATCH_DIR}/crawl.jsonl.gz.sha256"
DIGEST="$(cat "${SCRATCH_DIR}/crawl.jsonl.gz.sha256")"

# 3. Pull run metadata needed to make the manifest self-describing.
URL_COUNT="$(zcat "${SCRATCH_DIR}/crawl.jsonl.gz" | wc -l)"
CRAWLER_VERSION="$(docker run --rm "$CRAWLER_IMAGE" --version)"

# 4. Write the manifest. This file — not the crawl output alone — is
#    what every later audit diffs its own manifest against.
cat > "${SCRATCH_DIR}/manifest.json" <<EOF
{
  "domain": "${DOMAIN}",
  "run_timestamp": "${RUN_TS}",
  "artifact_file": "crawl.jsonl.gz",
  "artifact_sha256": "${DIGEST}",
  "url_count": ${URL_COUNT},
  "crawler_image": "${CRAWLER_IMAGE}",
  "crawler_version": "${CRAWLER_VERSION}",
  "render_js": false,
  "baseline_kind": "first_crawl"
}
EOF

# 5. Lock the manifest read-only immediately. If a baseline ever needs
#    correcting, the process is to archive this file and generate a new
#    one — never to reopen and edit it in place.
chmod 444 "${SCRATCH_DIR}/manifest.json"

# 6. Promote atomically. `mv` on the same filesystem is a single
#    rename syscall, so the baseline directory never exists in a
#    half-written state — it is either the previous baseline or the
#    new one, never a partial mix of both.
if [[ -d "$BASELINE_DIR" ]]; then
  mv "$BASELINE_DIR" "${BASELINE_DIR}.superseded-$(date -u +%Y%m%d%H%M%S)"
fi
mkdir -p "$(dirname "$BASELINE_DIR")"
mv "$SCRATCH_DIR" "$BASELINE_DIR"

echo "Baseline captured: ${BASELINE_DIR}/manifest.json (sha256=${DIGEST:0:12}...)"

Line-by-line, the discipline that matters: the crawler image is pinned to a full digest (@sha256:...), not just a version tag, because tags are mutable and digests are not. The crawl writes into a timestamped scratch directory so two runs — including a re-run after a failure — never collide. The checksum happens before anything else touches the file. The manifest embeds enough metadata (crawler_version, render_js, url_count) that a person reading it in a year does not have to guess what produced it. chmod 444 is a deliberate, cheap tripwire — it will not stop a determined root user, but it stops the much more common failure mode of an editor accidentally saving over the file. The final promotion step archives rather than deletes any prior baseline directory at that path, so a mistaken re-run of this script against an existing domain never silently destroys history.

Verification and smoke test

Run these checks immediately after the script above completes, before treating the new baseline as trustworthy:

set -euo pipefail
BASELINE_DIR="/opt/audit/baselines/example.com"

# 1. Manifest exists and is read-only
test -f "${BASELINE_DIR}/manifest.json" && echo "PASS: manifest present"
[ ! -w "${BASELINE_DIR}/manifest.json" ] && echo "PASS: manifest is read-only" \
  || echo "FAIL: manifest is still writable"

# 2. Recorded checksum matches the artifact bytes on disk right now
RECORDED="$(python3 -c "import json;print(json.load(open('${BASELINE_DIR}/manifest.json'))['artifact_sha256'])")"
ACTUAL="$(sha256sum "${BASELINE_DIR}/crawl.jsonl.gz" | awk '{print $1}')"
[ "$RECORDED" = "$ACTUAL" ] && echo "PASS: checksum matches" \
  || echo "FAIL: recorded=$RECORDED actual=$ACTUAL"

# 3. url_count in the manifest matches a fresh line count of the artifact
MANIFEST_COUNT="$(python3 -c "import json;print(json.load(open('${BASELINE_DIR}/manifest.json'))['url_count'])")"
FILE_COUNT="$(zcat "${BASELINE_DIR}/crawl.jsonl.gz" | wc -l)"
[ "$MANIFEST_COUNT" -eq "$FILE_COUNT" ] && echo "PASS: url_count matches ($FILE_COUNT)" \
  || echo "FAIL: manifest=$MANIFEST_COUNT file=$FILE_COUNT"

Expected output on a clean capture:

PASS: manifest present
PASS: manifest is read-only
PASS: checksum matches
PASS: url_count matches (4,812)

Any FAIL line means the baseline is not safe to diff against yet — treat it the same as a failed crawl and re-run the capture script rather than patching the existing directory by hand.

Failure modes

The crawl is non-deterministic between runs of the same domain state. If render_js is left enabled for the first crawl, JavaScript-heavy pages can return slightly different DOM snapshots run to run due to timing-dependent hydration, producing a different checksum even though nothing on the site changed. Fix: capture the first baseline with CRAWL_RENDER_JS=false for a stable, static-HTML reference point, and if a rendered snapshot is also needed, capture it as a second, separately labelled manifest.json with "render_js": true rather than overwriting the static one.

An unpinned crawler tag silently changes behavior. Referencing ghcr.io/your-org/technical-audit:latest (or even a mutable minor tag) means a routine registry push can change what the "first crawl" tool actually does, weeks or months after the fact, with no record of when it happened.

# diagnostic: does the manifest reference a floating tag?
python3 -c "import json;print(json.load(open('/opt/audit/baselines/example.com/manifest.json'))['crawler_image'])"
# fix: re-pin to a digest and re-capture
docker inspect --format='{{index .RepoDigests 0}}' ghcr.io/your-org/technical-audit:3.1.2

A baseline gets treated as mutable and edited in place. Someone opens manifest.json in an editor to "fix a typo" or bumps url_count by hand after adding a page manually to the crawl output, breaking the checksum-to-file relationship that makes the artifact trustworthy.

# diagnostic: was the manifest modified after chmod 444 was applied?
stat -c '%Y %n' /opt/audit/baselines/example.com/manifest.json
sha256sum /opt/audit/baselines/example.com/crawl.jsonl.gz
# compare against artifact_sha256 recorded in the manifest — any mismatch
# means the baseline must be re-captured, not repaired
# fix: archive the tampered baseline and re-run capture-baseline.sh
mv /opt/audit/baselines/example.com /opt/audit/baselines/example.com.tampered-$(date -u +%Y%m%d)

FAQ

Why checksum the artifact instead of just trusting the crawler's exit code?

An exit code of 0 only tells you the crawler process did not crash. It says nothing about whether the output file was truncated by a disk-full condition, partially written before a network blip, or silently corrupted during a copy to cloud storage. A SHA-256 digest computed immediately after the crawl, and re-verified after every copy or transfer, is the only signal that the bytes an auditor reads later are the exact bytes the crawler produced.

What makes a baseline manifest immutable in practice?

Two things: filesystem permissions and process discipline. Set the manifest file to chmod 444 immediately after writing it so accidental edits fail loudly. On top of that, never add an update path to the promotion script — if a baseline needs to change, archive the old manifest.json under a dated suffix and generate an entirely new one, rather than opening the existing file and editing a field.

How long should I keep the scratch crawl directory after promotion?

Keep it for at least one full audit cycle (commonly a quarter) as a recovery point, then let a lifecycle rule expire it. The promoted, checksummed copy in the baseline directory is the artifact of record; the scratch copy only exists so you can re-run the checksum step without repeating the crawl if the promotion script itself needs debugging.

Can I capture a baseline from a crawl that already ran for another purpose?

Only if that crawl used the same pinned toolchain and you still have the raw, uncompressed output before any downstream processing touched it. In practice this is fragile — most teams run a dedicated first-crawl job specifically for the baseline, because a crawl run for another purpose (say, redirect auditing) may have used a different crawl depth or JavaScript-rendering setting, which would make the resulting baseline non-comparable to later runs.