Handling Authentication & Session State in Crawls #

An audit that stops at the login form is auditing a marketing site, not the product. Account dashboards, checkout flows, saved-cart pages, admin surfaces and authenticated search results are usually where the highest-value templates live, and they are invisible to a crawler that has no session. This page is part of Automated Crawling & Pipeline Tooling and covers the layer that carries a credential through a crawl: capturing session state once, attaching it to every worker, refreshing it before it expires, and proving on every run that the pages coming back are the authenticated ones rather than a very consistent set of login screens.

The failure mode this guide exists to prevent is not a crash. An unauthenticated crawl of a private surface completes successfully, writes a full artifact, and reports a healthy score — because a login page is fast, accessible, returns 200, and has a canonical tag. Every check passes. What you have measured is your login form, several thousand times.

Prerequisites & Environment Setup #

Export these before any step below. Note that the credential itself is read from a secrets manager at process start and is never written to the environment file that is committed:

export AUTH_MODE="storage_state"          # storage_state | bearer | header
export AUTH_LOGIN_URL="https://app.example.com/login"
export AUTH_SENTINEL_SELECTOR="[data-testid=account-menu]"
export AUTH_STATE_PATH="/run/audit/state/session.json"
export AUTH_REFRESH_MARGIN_S=300          # refresh this long before expiry
export AUTH_MAX_SESSION_AGE_S=3600        # hard ceiling, recapture past this
export AUTH_PERSONA="standard_user"       # one storage state per persona

AUTH_STATE_PATH deliberately points at /run, a tmpfs on most Linux distributions, so a captured session never survives a reboot and never lands on a persistent disk that gets snapshotted into a backup. AUTH_SENTINEL_SELECTOR is the single most important variable here: it is the element that only exists when the session is real, and every stage below asserts on it.

Step 1 — Initialization: capture the session once #

Log in once per crawl run, not once per URL. A login form submission is expensive, is often rate-limited far more aggressively than ordinary reads, and on many applications triggers a security notification email — a crawler that logs in on every request will get the audit account locked within minutes.

#!/usr/bin/env python3
# /opt/audit/auth/capture_state.py
# Log in once and write a replayable storage state. Never logs the credential.
from __future__ import annotations
import os
import sys
from playwright.sync_api import sync_playwright

LOGIN_URL = os.environ["AUTH_LOGIN_URL"]
STATE_PATH = os.environ["AUTH_STATE_PATH"]
SENTINEL = os.environ["AUTH_SENTINEL_SELECTOR"]


def read_credential() -> tuple[str, str]:
    """Read from the platform secret store at call time. Never from argv,
    never from a committed .env, never cached on disk."""
    user = os.environ.get("AUDIT_USER")
    pw = os.environ.get("AUDIT_PASS")
    if not user or not pw:
        sys.exit("FAIL: credential not injected into the process environment")
    return user, pw


def main() -> int:
    user, pw = read_credential()
    os.makedirs(os.path.dirname(STATE_PATH), exist_ok=True)

    with sync_playwright() as p:
        browser = p.chromium.launch(args=["--disable-gpu"])
        ctx = browser.new_context()
        page = ctx.new_page()
        page.goto(LOGIN_URL, wait_until="domcontentloaded")
        page.fill("input[name=username]", user)
        page.fill("input[name=password]", pw)
        page.click("button[type=submit]")

        # The sentinel is the acceptance test. A redirect back to /login,
        # an MFA challenge, or a rate-limit page all fail here rather than
        # silently producing an unauthenticated state file.
        try:
            page.wait_for_selector(SENTINEL, timeout=15000)
        except Exception:
            browser.close()
            sys.exit("FAIL: sentinel never appeared — login did not complete")

        ctx.storage_state(path=STATE_PATH)
        browser.close()

    os.chmod(STATE_PATH, 0o600)
    print(f"OK: storage state written to {STATE_PATH}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Three details carry the design. The sentinel wait is the acceptance test, not a convenience — without it, a redirect back to the login page, an unexpected MFA challenge, or a rate-limit interstitial all produce a perfectly valid JSON file containing an anonymous session. The chmod 0o600 runs after the write rather than before, because Playwright creates the file itself. And the credential is read from the process environment at call time so it appears in exactly one frame of one function, rather than being threaded through the crawl as an argument that ends up in a log line or a crash traceback.

Step 2 — Core configuration: the session parameters #

AUTH_CONCURRENCY deserves the most attention. Many applications invalidate a session when the same cookie is used from too many concurrent connections, or silently rotate a session identifier on each request and expect the client to carry the newest one forward. Both behaviours turn a parallel crawl into a self-inflicted logout. Start at 2 and raise it only after confirming a full run at the higher value still passes the sentinel assertion on its last URL, and pair it with the per-host pacing described in managing crawl budget and rate limiting so an authenticated crawl is not simultaneously the most aggressive traffic the origin sees.

Step 3 — Execution & scheduling: refresh ahead of expiry #

A crawl long enough to matter will outlive its credential. The refresh has to be proactive: check the remaining lifetime before dispatching each batch, and renew when it falls inside the margin. Refreshing reactively — on the first 401 — guarantees that at least one request fails, and on a paginated or JSON endpoint that failure is frequently recorded as an empty but successful page rather than an error.

# /opt/audit/auth/session.py — proactive refresh, shared by every worker.
from __future__ import annotations
import os
import threading
import time

MARGIN = int(os.environ.get("AUTH_REFRESH_MARGIN_S", "300"))
MAX_AGE = int(os.environ.get("AUTH_MAX_SESSION_AGE_S", "3600"))


class Session:
    def __init__(self, capture, refresh):
        self._capture = capture          # full re-login, expensive
        self._refresh = refresh          # token exchange, cheap
        self._lock = threading.Lock()
        self._expires_at = 0.0
        self._captured_at = 0.0

    def ensure_valid(self) -> None:
        now = time.time()
        if now < self._expires_at - MARGIN and now - self._captured_at < MAX_AGE:
            return                       # comfortably valid, do nothing
        with self._lock:
            # Re-check inside the lock: another worker may have just renewed.
            now = time.time()
            if now < self._expires_at - MARGIN and now - self._captured_at < MAX_AGE:
                return
            if now - self._captured_at >= MAX_AGE:
                self._expires_at = self._capture()   # past the ceiling
                self._captured_at = now
            else:
                self._expires_at = self._refresh()   # ordinary renewal

The double-check inside the lock is what keeps a fleet of workers from all deciding to re-login at the same moment — the classic thundering-herd shape, and one that authentication endpoints are especially unforgiving about. MAX_AGE exists separately from expiry because refresh tokens themselves eventually stop being accepted; past the ceiling the only correct move is a full recapture.

Schedule authenticated crawls in their own window rather than alongside the anonymous crawl. Two runs sharing a target and a session will fight over it, and when they are wired through CI/CD pipelines the concurrency guard should key on the persona as well as the host, so a standard-user crawl and an admin crawl do not overlap.

Step 4 — Artifact capture & storage: what must never be written #

Authenticated crawls collect material that an anonymous crawl never sees, and the artifact policy has to reflect that. Three rules:

• Never persist the credential or the storage state. The state file lives on tmpfs for the duration of the run and is deleted in a finally block. It is not an artifact.
• Redact Set-Cookie and Authorization from any captured HAR. A HAR from an authenticated session is a replayable credential, and HAR files routinely get attached to tickets.
• Tag every artifact with the persona. A record from an admin crawl and a record from a standard-user crawl describe genuinely different pages at the same URL, and pooling them produces a score for a page nobody sees.

#!/usr/bin/env bash
# /opt/audit/auth/scrub_har.sh — strip credentials before a HAR is stored.
set -euo pipefail
IN="$1"; OUT="$2"
jq '(.log.entries[].request.headers[] | select(.name | ascii_downcase == "authorization") | .value) = "REDACTED"
    | (.log.entries[].request.headers[] | select(.name | ascii_downcase == "cookie") | .value) = "REDACTED"
    | (.log.entries[].response.headers[] | select(.name | ascii_downcase == "set-cookie") | .value) = "REDACTED"'    "$IN" > "$OUT"
echo "scrubbed -> $OUT"

Retention follows the same shape described in storing and versioning crawl artifacts in cloud storage, with one addition: authenticated artifacts get a shorter hot window than anonymous ones, because the value of keeping them decays faster than the risk of holding them.

Verification checklist #

1. The state file exists and is not anonymous. jq '.cookies | length' "$AUTH_STATE_PATH" returns a non-zero count, and jq -r '.cookies[].name' "$AUTH_STATE_PATH" includes the application's session cookie name.
2. The sentinel is present on a sampled authenticated page. Re-render one URL from the middle of the crawl and assert AUTH_SENTINEL_SELECTOR is in the DOM. This is the check that catches an expiry partway through a run.
3. The last URL crawled is still authenticated. Assert the sentinel on the final record, not just the first — a session that died at 40% produces an artifact whose opening pages look perfect.
4. No login-page fingerprint in the corpus. grep -c 'name="password"' artifacts/*.html should return 0. Any non-zero result means unauthenticated pages were recorded as content.
5. Personalised URLs differ across personas. Crawl one known account-scoped URL under two personas and confirm the DOM hashes differ. Identical hashes mean the persona switch is not taking effect.
6. No credential in any artifact. grep -riE 'authorization:|set-cookie:' artifacts/ | wc -l returns 0 after the scrub step.
7. The state file is gone. test ! -f "$AUTH_STATE_PATH" passes after the run's cleanup block.

Troubleshooting #

Every authenticated URL returns the login page #

Root cause: the storage state was captured successfully but never attached — browser.new_context() was called without storage_state=, so every worker started anonymous.

Fix: pass the path explicitly and assert immediately, before the crawl loop starts.

ctx = browser.new_context(storage_state=os.environ["AUTH_STATE_PATH"])
probe = ctx.new_page()
probe.goto(os.environ["AUTH_LOGIN_URL"].replace("/login", "/account"))
assert probe.query_selector(os.environ["AUTH_SENTINEL_SELECTOR"]), "context is anonymous"
probe.close()

The crawl authenticates, then degrades partway through #

Root cause: the session expired mid-run and nothing refreshed it. Because the application redirects rather than returning 401, the thin login pages are recorded as ordinary 200 responses.

Fix: wire Session.ensure_valid() into the fetch loop, and add a per-record sentinel check so an unauthenticated response is classified as a failure rather than as thin content.

# diagnostic: how far into the run did the session die?
jq -r 'select(.has_sentinel == false) | .index' artifacts/records.jsonl | head -n1

MFA blocks the capture step #

Root cause: the audit account is enrolled in an interactive second factor, so the login flow cannot complete headlessly.

Fix: do not attempt to automate the interactive factor. Provision a dedicated audit account with a non-interactive mechanism — a service credential, a long-lived API token scoped to read-only, or an allowlisted source IP — and treat that account as the crawl identity. If your organisation cannot exempt an account, capture the state manually into tmpfs and run the crawl inside that session's lifetime.

Concurrent workers log each other out #

Root cause: the application rotates the session identifier on each response and expects the client to carry the newest one; several workers sharing one cookie jar each hold a stale identifier after the first rotation.

Fix: lower AUTH_CONCURRENCY to 1 for that target and, if throughput matters, capture one storage state per worker from separate audit accounts rather than sharing one.

The token refresh succeeds but the crawl still 401s #

Root cause: the refreshed token is being stored on the Session object but the already-constructed HTTP client is still sending the header value it captured at construction time.

Fix: read the header from the session on every request rather than binding it once.

# wrong: header bound once at client construction
client = httpx.Client(headers={"Authorization": f"Bearer {session.token}"})
# right: resolved per request, after ensure_valid()
session.ensure_valid()
resp = client.get(url, headers={"Authorization": f"Bearer {session.token}"})

Personalised pages score worse than they should #

Root cause: account-scoped pages legitimately carry more uncached, personalised content than public templates, so a single site-wide threshold flags them constantly.

Fix: treat authenticated templates as their own section when calibrating error thresholds for different site sections, rather than comparing them against public marketing pages that are served from an edge cache.

FAQ #

Why does an unauthenticated crawl of a private surface still pass every check?

Because a login page is a genuinely healthy page. It returns 200, it is fast, it is usually accessible, and it carries a canonical tag. Every structural check a crawler makes will pass on it. The only thing that distinguishes a real authenticated crawl from several thousand captures of one login form is an assertion on content that exists solely inside a session — which is why the sentinel selector is a required parameter rather than an optional one.

Should the crawler log in once per run or once per URL?

Once per run. Login endpoints are typically rate-limited far more aggressively than ordinary reads, and many applications send a security notification on each new sign-in, so a per-URL login will lock the audit account within minutes. Capture one storage state at the start of the run, replay it across workers, and refresh the underlying credential on a margin rather than re-authenticating from scratch.

How do I handle an audit account that is enrolled in multi-factor authentication?

Do not try to automate the interactive factor. Provision a dedicated audit identity with a non-interactive mechanism instead — a service credential, a read-only API token, or an allowlisted source IP range for the crawl runners. If the organisation cannot exempt an account, the workable fallback is to capture the storage state manually into tmpfs and run the crawl inside that session lifetime, accepting that the run cannot be fully unattended.

Can one crawl cover several user roles at once?

No, and it should not try. Content that varies per role means the same URL is a different page for each persona, so records from two roles cannot share a score. Run one crawl per persona with its own storage state, tag every artifact with the persona, and keep the results in separate partitions. Pooling them produces an average of two pages that no single user ever sees.

---

Related #

• Automated Crawling & Pipeline Tooling — the parent section covering the full crawl pipeline from provisioning to remediation routing
• Configuring Headless Browsers for JS-Heavy Sites — browser context setup and hydration waits, which the storage state is attached to
• Managing Crawl Budget & Rate Limiting — pacing controls that matter more on an authenticated surface than an anonymous one
• Storing & Versioning Crawl Artifacts in Cloud Storage — retention and redaction policy for artifacts captured inside a session