GitHub Actions vs GitLab CI for Crawler Scheduling #

Most teams do not choose a CI platform for their audit crawler — they inherit whichever one already hosts the application repository — but the scheduling primitives underneath GitHub Actions and GitLab CI differ enough that a config copied verbatim between them silently breaks. This guide is part of Integrating Custom Crawlers with CI/CD Pipelines and gives a side-by-side decision for teams choosing, or migrating between, the two schedulers.

Both platforms can run the exact same crawl on the exact same interval. What differs is how cron is expressed, how overlapping runs are prevented, how secrets are scoped to a schedule, and how self-hosted runners are declared. Get any one of those wrong and the failure mode is not a build error — it is a crawl that either never fires, fires twice, or fires with the wrong credentials.

Comparison Table #

The last row is the one teams miss most often during a migration: a crawl that has run reliably on GitLab CI for a year will simply stop firing on GitHub Actions if the repository goes 60 days without a commit, with no error surfaced anywhere except a gap in the crawl history.

Two Parity Configs #

Both configs below run the identical crawl — a containerized crawler invoking the weekly cron pattern — every Monday at 03:00 UTC, with a concurrency guard and scoped secrets.

# .github/workflows/scheduled-crawl.yml
name: scheduled-audit-crawl

on:
  schedule:
    # 03:00 UTC every Monday — cron is ALWAYS evaluated in UTC
    - cron: "0 3 * * 1"
  workflow_dispatch: {}   # allows a manual re-run on the same guard

concurrency:
  group: scheduled-audit-crawl-${{ github.ref }}
  cancel-in-progress: false   # queue, do not kill a run mid-crawl

jobs:
  crawl:
    runs-on: ubuntu-22.04
    environment: crawl-production   # gates secrets behind this environment
    timeout-minutes: 45
    steps:
      - uses: actions/checkout@v4
      - name: Run crawl
        env:
          CRAWL_TARGET: ${{ vars.CRAWL_TARGET }}
          CRAWL_API_TOKEN: ${{ secrets.CRAWL_API_TOKEN }}
        run: |
          set -euo pipefail
          docker run --rm \
            -e CRAWL_TARGET -e CRAWL_API_TOKEN \
            ghcr.io/site-health/crawler:1.9.2 \
            --output /out/crawl-$(date -u +%Y%m%dT%H%M%SZ).json
      - uses: actions/upload-artifact@v4
        with:
          name: crawl-artifact
          path: out/
          retention-days: 30

# .gitlab-ci.yml
stages:
  - crawl

scheduled-audit-crawl:
  stage: crawl
  image: docker:25.0
  services:
    - docker:25.0-dind
  resource_group: production-crawl-target   # serializes overlapping runs
  timeout: 45m
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule"'
    - if: '$CI_PIPELINE_SOURCE == "web"'      # allow a manual re-run
  variables:
    CRAWL_TARGET: "$CRAWL_TARGET"             # set as protected + masked
  script:
    - set -euo pipefail
    - |
      docker run --rm \
        -e CRAWL_TARGET -e CRAWL_API_TOKEN \
        registry.gitlab.com/site-health/crawler:1.9.2 \
        --output out/crawl-$(date -u +%Y%m%dT%H%M%SZ).json
  artifacts:
    paths:
      - out/
    expire_in: 30 days

The CRAWL_API_TOKEN reference is deliberately absent from the GitLab job's variables: block above — it is injected automatically at runtime from a project-level, protected, masked CI/CD variable, the same way secrets.CRAWL_API_TOKEN is injected in the GitHub job without appearing in the workflow file. Neither platform's schedule-only trigger needs the pipeline schedule owner's personal token; both use a machine identity scoped to the crawl job.

The GitLab pipeline schedule itself (frequency, target branch, variable overrides) is created once in Settings → CI/CD → Pipeline schedules and is not expressed in YAML — only the rules gate that reacts to it lives in the repository.

Verification #

Run this after the first scheduled window passes on either platform to confirm exactly one run fired and produced an artifact.

set -euo pipefail

# GitHub Actions — list runs for the schedule event, expect exactly one
# per window, status "completed", conclusion "success"
gh run list \
  --workflow scheduled-crawl.yml \
  --event schedule \
  --limit 5 \
  --json databaseId,status,conclusion,createdAt

# GitLab CI — list pipelines triggered by the schedule source
curl -sS --header "PRIVATE-TOKEN: ${GITLAB_TOKEN}" \
  "https://gitlab.example.com/api/v4/projects/${CI_PROJECT_ID}/pipelines?source=schedule&per_page=5" \
  | jq '.[] | {id, status, created_at}'

Expected output on both platforms is a single row per scheduled window with a success status. Two rows for the same window, or a row with status queued that never transitions, both point to a concurrency-guard misconfiguration rather than a crawler bug — check the guard before touching the crawler code.

Failure Modes #

Cron interpreted in local time instead of UTC

A schedule of 0 22 * * 5 written assuming "10 PM Friday, US Eastern" actually fires at 22:00 UTC — 6 PM Eastern during daylight saving, 5 PM during standard time. Both platforms document this, but the assumption slips through code review because the YAML gives no visual cue.

# diagnostic: confirm the platform's interpretation
# GitHub — check the workflow run's createdAt against the cron in UTC
gh run list --workflow scheduled-crawl.yml --json createdAt --limit 1
# GitLab — pipeline schedules UI always displays "UTC" next to the cron field

Fix: write the UTC hour directly in the cron expression and add a one-line comment stating the source timezone and the date the conversion was made, since a daylight-saving shift silently moves the correct UTC hour by one, twice a year.

Overlapping runs against the same target host

A crawl that normally finishes in 20 minutes takes 90 minutes because the target site is degraded, and the next scheduled trigger fires while the first is still running. Without a guard, both processes hit the host simultaneously, doubling request volume right when the target is least able to absorb it, and corrupting any crawl-budget counters that assume a single active run — see managing crawl budget and rate limiting for why a shared counter cannot tolerate two writers.

# GitHub — confirm the concurrency group name is identical across triggers
grep -A2 "^concurrency:" .github/workflows/scheduled-crawl.yml
# GitLab — confirm the resource_group is set on the job, not just implied
grep -A1 "resource_group" .gitlab-ci.yml

Fix: add (or correct) concurrency.group in GitHub Actions, or resource_group in GitLab CI, keyed by target host rather than by branch, so two schedules pointed at the same host queue instead of racing even if they live in different repositories.

Secrets scoped too broadly

A crawl credential added as a repository-level GitHub secret, or an unprotected GitLab CI/CD variable, is readable by any workflow or pipeline in the repository — including one triggered from a fork's pull request or an unprotected feature branch. This turns a scheduling convenience into a credential leak.

# GitHub — list which environments actually gate this secret
gh api repos/:owner/:repo/environments/crawl-production/secrets
# GitLab — confirm the variable is both protected AND masked
curl -sS --header "PRIVATE-TOKEN: ${GITLAB_TOKEN}" \
  "https://gitlab.example.com/api/v4/projects/${CI_PROJECT_ID}/variables/CRAWL_API_TOKEN"

Fix: move the credential into a GitHub environment restricted to the schedule's branch, or mark the GitLab variable protected and masked, then rotate the credential since any prior broad exposure should be treated as already compromised.

FAQ #

Does GitHub Actions cron respect my repository's timezone setting?

No. Both GitHub Actions and GitLab CI evaluate schedule cron expressions in UTC regardless of any repository, profile, or organization timezone setting. If you want a crawl to start at 2 AM local time, convert that offset to UTC yourself and add a comment documenting the source timezone and the date the conversion was made, since daylight saving shifts the correct UTC hour twice a year.

What happens if two scheduled crawls overlap on the same target host?

Without a concurrency guard, a slow crawl run and its next scheduled trigger both hit the target concurrently, doubling request volume and corrupting shared crawl-budget counters. GitHub Actions solves this with a concurrency group plus cancel-in-progress; GitLab CI solves it with a resource_group that queues the newer run until the older one exits.

How do GitHub Actions environments compare to GitLab CI protected variables for secret scoping?

Both achieve the same outcome through different primitives. A GitHub Actions environment gates a secret behind required reviewers and a branch filter; a GitLab CI/CD variable marked protected and masked is only injected into pipelines running on protected branches or tags. Neither should be attached to variables available on merge-request or fork pipelines, since a crawl credential exposed there can be exfiltrated by an untrusted contributor.

Can I trigger the same crawl manually outside its schedule on both platforms?

Yes. GitHub Actions exposes workflow_dispatch as a second trigger alongside schedule, and GitLab CI exposes the same job through a web or API pipeline as long as the rules clause also matches CI_PIPELINE_SOURCE values of web and api, not only schedule. Keep the manual trigger behind the same concurrency guard so an ad-hoc run cannot collide with the next scheduled one.

---

Related #

• Integrating Custom Crawlers with CI/CD Pipelines — parent guide covering runner setup, environment injection, and artifact capture
• Setting Up a Cron Job for Weekly Site Crawls — the underlying schedule pattern both configs on this page implement
• Managing Crawl Budget & Rate Limiting — why overlapping scheduled runs corrupt shared budget counters