10 min read

Setting Up a Quarterly Technical Audit Schedule

Running audits on demand produces data snapshots that cannot be compared across time. Without a fixed execution cadence, LCP regressions introduced between deployments go undetected, 404 clusters accumulate unnoticed across CMS migrations, and no one has a repeatable baseline to diff against. This page shows you how to wire a deterministic quarterly schedule on top of the baseline health metrics workflow so every run produces an artifact that can be diffed against the previous quarter with zero ambiguity.

Environment isolation and dependency declaration

Establish a fixed execution user, working directory, and dependency set before writing a single cron expression. All paths below use absolutes so the schedule fires correctly regardless of the shell's PATH.

#!/usr/bin/env bash
# /etc/audit/env.sh — sourced by every audit script
set -euo pipefail

export AUDIT_HOME="/opt/audit"
export AUDIT_RUN_TS="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
export AUDIT_QUARTER="$(date -u +%Y-Q)$((( $(date -u +%-m) - 1) / 3 + 1))"
export CRAWL_CONFIG="${AUDIT_HOME}/config/scope.json"
export ARTIFACT_BUCKET="gs://your-org-audit-artifacts"
export LOG_FILE="/var/log/audit/quarterly-${AUDIT_QUARTER}.log"
export DOCKER_IMAGE="ghcr.io/your-org/technical-audit:3.1.2"   # pinned, never :latest

# Runtime dependencies — checked at startup, not install time
for bin in docker flock jq sha256sum gsutil; do
  command -v "$bin" >/dev/null 2>&1 || { echo "FATAL: $bin not found"; exit 1; }
done

mkdir -p "${AUDIT_HOME}/runs/${AUDIT_QUARTER}" /var/log/audit

Pin the Docker image tag to a specific digest in production; 3.1.2 is the version tag — add @sha256:<digest> for supply-chain hardening. The AUDIT_QUARTER variable (2026-Q2) becomes the primary key for storing and versioning crawl artifacts in cloud storage.

Implementation: cron expression, flock guard, and scope-lock script

The three files below form the complete scheduling unit. Deploy them together; changing any one in isolation breaks the others.

1 — Cron expression 

# /etc/cron.d/quarterly-audit
# Fires at 02:00 UTC on the first Sunday of January, April, July, and October.
# The day-of-month range 1-7 combined with the day-of-week check enforces "first Sunday".
CRON_TZ=UTC
MAILTO=""
0 2 1-7 1,4,7,10 0 audit-runner /opt/audit/bin/run-quarterly-audit.sh >> /var/log/audit/cron.log 2>&1

Run as a dedicated low-privilege user (audit-runner) rather than root. Create it with useradd -r -s /sbin/nologin audit-runner and give it write access only to /opt/audit/runs/ and /var/log/audit/.

2 — Runner script with flock guard 

#!/usr/bin/env bash
# /opt/audit/bin/run-quarterly-audit.sh
set -euo pipefail

LOCKFILE="/var/run/audit/quarterly.lock"
mkdir -p "$(dirname "$LOCKFILE")"

# flock -n exits immediately if another run holds the lock.
# Replace -n with a timeout (flock -w 300) if you want to queue rather than skip.
exec 200>"$LOCKFILE"
flock -n 200 || { echo "[$(date -u +%FT%TZ)] SKIP: previous run still active"; exit 0; }

source /etc/audit/env.sh

echo "[${AUDIT_RUN_TS}] START quarter=${AUDIT_QUARTER}" | tee -a "$LOG_FILE"

/opt/audit/bin/validate-scope.sh   || { echo "SCOPE CHECK FAILED"; exit 2; }
/opt/audit/bin/execute-crawl.sh    || { echo "CRAWL FAILED";       exit 3; }
/opt/audit/bin/archive-artifacts.sh || { echo "ARCHIVE FAILED";   exit 4; }
/opt/audit/bin/route-alerts.sh      || { echo "ALERT ROUTING FAILED"; exit 5; }

echo "[$(date -u +%FT%TZ)] DONE quarter=${AUDIT_QUARTER}" | tee -a "$LOG_FILE"

The flock -n flag skips the run rather than queuing it. This prevents a stalled audit (e.g. waiting on a slow API response) from causing a second instance to pile up. For environments where you need overlap protection with queuing, replace with flock -w 300 (wait up to 5 minutes) and log the wait duration.

3 — Scope-lock validator

Scope drift is the most common source of false regression signals. The script below reads a manifest generated from the previous quarter's crawl and asserts the current URL count falls within ±2%:

#!/usr/bin/env bash
# /opt/audit/bin/validate-scope.sh
set -euo pipefail
source /etc/audit/env.sh

PREV_QUARTER_DIR="${AUDIT_HOME}/runs/$(date -d 'now -3 months' +%Y-Q$((( $(date -d 'now -3 months' +%-m) - 1) / 3 + 1)))"
PREV_MANIFEST="${PREV_QUARTER_DIR}/url-manifest.txt"
CURRENT_MANIFEST="${AUDIT_HOME}/runs/${AUDIT_QUARTER}/url-manifest.txt"

# Generate current manifest from live sitemap
curl -fsSL "https://$(jq -r '.domain' "$CRAWL_CONFIG")/sitemap.xml" \
  | grep -oP '(?<=<loc>)[^<]+' \
  | sort > "$CURRENT_MANIFEST"

PREV_COUNT=$(wc -l < "$PREV_MANIFEST" 2>/dev/null || echo 0)
CURR_COUNT=$(wc -l < "$CURRENT_MANIFEST")

if [[ "$PREV_COUNT" -gt 0 ]]; then
  DEVIATION=$(echo "scale=4; ($CURR_COUNT - $PREV_COUNT) / $PREV_COUNT" | bc | tr -d -)
  THRESHOLD="0.0200"
  if [[ "$(echo "$DEVIATION > $THRESHOLD" | bc)" == "1" ]]; then
    echo "SCOPE DRIFT: prev=${PREV_COUNT} curr=${CURR_COUNT} deviation=${DEVIATION}"
    exit 2
  fi
fi

echo "Scope OK: ${CURR_COUNT} URLs (prev=${PREV_COUNT})"

When the deviation check fires, diff "$PREV_MANIFEST" against "$CURRENT_MANIFEST" to identify which URLs disappeared or appeared. A sudden drop often signals a robots.txt change or CMS misconfiguration — the same patterns covered under defining crawl depth and scope for enterprise sites.

Architecture: quarterly audit pipeline

Quarterly audit pipeline Five sequential stages: cron trigger fires runner script, flock guard prevents overlapping runs, scope validator checks URL deviation, crawl executes in Docker container, then artifacts are archived and alerts are routed to incident management. Cron first Sun/quarter flock guard skip if running Scope validator ±2% URL tolerance Docker crawl pinned image Archive + alert routing skip (overlapping run)

Verification and smoke test

After deploying the three files above, trigger a manual run and confirm each gate fires cleanly before relying on the automated schedule:

# 1. Dry-run the scope validator (no crawl, no artifacts)
sudo -u audit-runner /opt/audit/bin/validate-scope.sh
# Expected: "Scope OK: <N> URLs (prev=<N>)"

# 2. Confirm flock prevents a second concurrent execution
sudo -u audit-runner /opt/audit/bin/run-quarterly-audit.sh &
sudo -u audit-runner /opt/audit/bin/run-quarterly-audit.sh
# Expected second run: "SKIP: previous run still active"

# 3. Verify cron is installed and readable
crontab -u audit-runner -l | grep quarterly-audit
# Expected: the line from /etc/cron.d/quarterly-audit

# 4. Confirm artifact directory exists after a full run
ls -lh /opt/audit/runs/$(date -u +%Y-Q$((( $(date -u +%-m) - 1) / 3 + 1)))/
# Expected: url-manifest.txt and at least one .json.gz report

# 5. Assert log contains START and DONE for the current run
grep "quarter=$(date -u +%Y-Q$((( $(date -u +%-m) - 1) / 3 + 1)))" /var/log/audit/cron.log | tail -4
# Expected: one START line and one DONE line with matching timestamps

Failure signal: if the log shows a SCOPE CHECK FAILED exit code 2, the URL deviation exceeded 2%. Run diff /opt/audit/runs/<prev>/url-manifest.txt /opt/audit/runs/<curr>/url-manifest.txt to locate the structural change before re-running.

Failure modes

Cron fires but no log appears. The audit-runner user lacks write permission to /var/log/audit/. Fix: chown audit-runner:audit-runner /var/log/audit && chmod 750 /var/log/audit. Verify with sudo -u audit-runner touch /var/log/audit/test && rm /var/log/audit/test.

Scope validator exits 2 after a planned site restructure. The previous quarter's manifest is stale relative to an intentional URL architecture change. Update the baseline: copy the current manifest to the previous quarter's directory and update EXPECTED_URL_COUNT in scope.json, then commit both changes with a comment referencing the migration ticket. This is the same recalibration process described in aligning audit goals with business KPIs when scope changes are driven by strategic priorities.

Docker container exits non-zero mid-crawl. Inspect the container logs: docker logs $(docker ps -lq --filter "ancestor=${DOCKER_IMAGE}"). The most common causes are OOM kills (increase memory in Compose to 6 GB), DNS resolution failures on the audit host (check /etc/resolv.conf inside the container), and stale session tokens for authenticated crawls. The integrating custom crawlers with CI/CD pipelines guide covers container environment debugging in detail.

FAQ

What if flock is not available on my audit host?

Install util-linux — flock ships with it on Debian/Ubuntu (apt-get install -y util-linux). On Alpine: apk add util-linux. As a POSIX-safe fallback, use an atomic mkdir lock:

LOCKDIR="/var/run/audit/quarterly.lock"
mkdir "$LOCKDIR" 2>/dev/null || { echo "SKIP: lock held"; exit 0; }
trap 'rmdir "$LOCKDIR"' EXIT INT TERM

mkdir is atomic on local POSIX filesystems. It is not safe across NFS; use flock on shared volumes.

How do I shift the execution window to a different timezone?

Set CRON_TZ in the crontab header before the schedule line:

CRON_TZ=America/New_York
0 2 1-7 1,4,7,10 0 audit-runner /opt/audit/bin/run-quarterly-audit.sh

For systemd timers, append the timezone to the OnCalendar value: OnCalendar=Sun *-1,4,7,10-01..07 02:00:00 America/New_York. Regardless of trigger timezone, always record AUDIT_RUN_TS in UTC inside the script so quarter-over-quarter diff comparisons remain consistent.

Can I run this in GitHub Actions instead of a system cron?

Yes. Use the schedule trigger with the same cron expression. GitHub Actions schedule has ±15 min execution drift, which is acceptable for a quarterly cadence. Record the actual start time from $(date -u +%Y-%m-%dT%H:%M:%SZ) in AUDIT_RUN_TS and embed it in the artifact filename — never derive it from the schedule value. For preventing overlapping runs in GitHub Actions, use the concurrency key:

concurrency:
  group: quarterly-audit
  cancel-in-progress: false   # queue rather than cancel
What do I do when the URL count deviation check fails?

First check whether a sitemap update, CMS migration, or robots.txt change accounts for the difference. Run diff <prev-manifest> <curr-manifest> | head -60 to identify which URLs appeared or disappeared. If the change is intentional, update the baseline manifest and commit the config change with a migration note. If unexpected, investigate the crawl boundary — often a new Disallow directive or a sitemap path change is the cause. The scope-locking pattern here ties directly into the risk scoring frameworks for technical debt when large structural changes need to be triaged by priority.