Auditing International & Multi-Locale Sites #

An estate serving several languages or markets breaks an assumption that sits underneath every audit tool: that a URL identifies a page. Depending on how the locale distinction is expressed, the same URL may serve German to one visitor and Japanese to another, or the "same" page may exist at four addresses that must declare each other. This page is part of Technical Audit Fundamentals & Scope Mapping and covers scoping, crawling and scoring a multi-locale estate without producing a confident number for the wrong pages.

Prerequisites & environment setup #

export INTL_LOCALE_MODE="subfolder"        # ccTLD | subdomain | subfolder | header
export INTL_LOCALES="en-gb,de-de,fr-fr,ja-jp"
export INTL_DEFAULT_LOCALE="en-gb"
export INTL_ACCEPT_LANGUAGE=""             # set per crawl when mode is header
export INTL_XDEFAULT_REQUIRED=1            # require an x-default alternate
export INTL_BASELINE_PER_LOCALE=1          # never pool locales into one baseline

INTL_BASELINE_PER_LOCALE defaulting to on is the setting that prevents the quietest failure in this whole area. A site-wide baseline computed across four markets will absorb one failing market inside three healthy ones, and the composite will barely move while an entire region degrades.

Step 1 — Classify where the locale actually lives #

Before scoping anything, establish which of the four shapes the estate uses, because every later decision follows from it.

#!/usr/bin/env python3
# /opt/audit/intl/classify.py — determine the locale carrier for a URL set.
from __future__ import annotations
import re
from collections import Counter
from urllib.parse import urlsplit
import tldextract

LOCALE_RE = re.compile(r"^[a-z]{2}(-[a-z]{2})?$", re.I)


def carrier(urls: list[str]) -> str:
    hosts, firsts, suffixes = Counter(), Counter(), Counter()
    for u in urls:
        parts = urlsplit(u)
        ext = tldextract.extract(u)
        suffixes[ext.suffix] += 1
        hosts[ext.subdomain] += 1
        seg = parts.path.strip("/").split("/")
        if seg and LOCALE_RE.match(seg[0]):
            firsts[seg[0].lower()] += 1

    if len({s for s in suffixes if s}) > 1:
        return "cctld"
    if sum(1 for h in hosts if LOCALE_RE.match(h or "")) > 1:
        return "subdomain"
    if len(firsts) > 1:
        return "subfolder"
    return "header"          # nothing in the URL distinguishes locale

A return value of header is the important one, and it is a conclusion rather than a default: nothing in the URL distinguishes locale, so the content must be varying on a request header or a cookie. That estate cannot be audited by one crawl, and no amount of care in the rest of the pipeline changes it.

Step 2 — Core configuration: the scoping parameters #

INTL_ENTRY_REDIRECT_MAX exists because geo-redirection is the most common source of inflated hop counts on an international estate. A crawler requesting /de/pricing from an address that geolocates elsewhere may be bounced to /en-gb/pricing before anything is measured, and every URL in the run then carries an extra hop that has nothing to do with the site's redirect hygiene.

Step 3 — Execution: one crawl shape per carrier #

For a subfolder estate, one crawl covers everything and the locale becomes a record field:

def locale_of(url: str, default: str) -> str:
    seg = urlsplit(url).path.strip("/").split("/")
    return seg[0].lower() if seg and LOCALE_RE.match(seg[0]) else default

For a header-carried estate, the locale has to be pinned per crawl and asserted per record, because nothing in the URL will tell you afterwards which version was captured:

ctx = browser.new_context(
    locale="de-DE",
    extra_http_headers={"Accept-Language": "de-DE,de;q=0.9"},
)
# Assert what was actually served, rather than what was requested.
served = page.eval_on_selector("html", "el => el.lang") or ""
if not served.lower().startswith("de"):
    raise RuntimeError(f"requested de-DE, got lang={served!r}")

That assertion is the whole discipline for header-carried estates. Without it, a crawl configured for four locales that silently receives the default for all four looks exactly like a healthy, consistent estate.

For a ccTLD estate, each domain gets its own crawl, its own budget and its own rate-limit configuration, because a shared allowance is spent almost entirely by the largest market.

Step 4 — Artifact capture: locale is a partition, not a column #

Partition the artifact by locale as well as by date. A locale column works for filtering and fails for everything else: retention, backfill and rebaselining all operate per partition, and a market that needs re-collecting after a migration should not require rewriting the whole day.

/data/scored/date=2026-07-31/locale=de-de/part-0.parquet
/data/scored/date=2026-07-31/locale=fr-fr/part-0.parquet

Reconciling hreflang alternate sets #

An alternate set is a reciprocity claim: each locale variant of a page declares every other variant, including itself, and one of them declares x-default for visitors whose preference matches nothing. The claim is only meaningful if it holds in both directions, and the failure mode is asymmetry — page A lists page B, page B does not list page A, and a consumer reading either one gets a different picture of the estate.

Reconciliation is a set comparison rather than a per-page check, which is why it belongs at the audit layer rather than in a page-level linter:

#!/usr/bin/env python3
# /opt/audit/intl/reciprocity.py — an alternate set is a claim both sides make.
from __future__ import annotations
from collections import defaultdict


def reciprocity_findings(alternates: dict[str, dict[str, str]]) -> list[dict]:
    """alternates: {url: {hreflang: target_url}} after redirect resolution."""
    findings: list[dict] = []
    declared = defaultdict(set)
    for src, mapping in alternates.items():
        for target in mapping.values():
            declared[src].add(target)

    for src, targets in declared.items():
        for target in targets:
            if target == src:
                continue
            if target not in declared:
                findings.append({"url": src, "target": target,
                                 "code": "ALTERNATE_TARGET_NOT_CRAWLED"})
            elif src not in declared[target]:
                findings.append({"url": src, "target": target,
                                 "code": "ALTERNATE_NOT_RECIPROCAL"})
        if not any(k.lower() == "x-default" for k in alternates[src]):
            findings.append({"url": src, "target": None,
                             "code": "ALTERNATE_MISSING_XDEFAULT"})
    return findings

Three details make the difference between a useful report and a noisy one. Alternates are compared after redirect resolution, because a variant that 301s elsewhere declares one address and resolves to another, and comparing the pre-redirect URLs produces asymmetry findings for pages that are actually consistent. ALTERNATE_TARGET_NOT_CRAWLED is kept distinct from ALTERNATE_NOT_RECIPROCAL because they have different causes — the first usually means a scope gap in the crawl, the second means a genuine declaration mismatch on the site, and routing them to the same owner wastes everybody's time. And the x-default check runs per source rather than per set, so a set where only one variant declares it is reported once rather than once per member.

The severity of these findings is worth calibrating deliberately rather than inheriting a default. A missing x-default on a marketing page is a minor finding; the same on the entry point that every market's traffic lands on is not. Bind the impact rating to the template as described in assigning severity labels to audit findings, so an alternate finding on a checkout flow outranks fifty of them on an archive.

Verification checklist #

1. Every locale in scope produced records. A locale present in INTL_LOCALES and absent from the output means the crawl never reached it, which on a header-carried estate is the default failure.
2. The served locale matches the requested one. Assert html[lang] against the crawl's configured locale on every record, not on a sample.
3. Scores differ across locales. Identical composite scores across markets is the signature of one locale being crawled repeatedly.
4. Entry hop counts are inside the ceiling. A uniform extra hop across the whole run is a geo-redirect, not a site problem.
5. Alternate sets are reciprocal. Every locale variant declares every other, and one declares x-default.
6. Baselines are per locale. Confirm the scoring config resolves a distinct baseline per market rather than one site-wide distribution.

Troubleshooting #

Every locale returns the default language

The crawler's Accept-Language is unset, so the origin applies its own default, or a geo-IP rule overrides the header entirely. Header-based negotiation is advisory and many CDNs ignore it in favour of the request's source address. Where a locale-specific URL exists, request that instead of negotiating; where it does not, the crawl has to run from an address the origin will geolocate correctly.

Hreflang alternates point at redirects

A locale variant that 301s to another URL breaks reciprocity, because the target declares a different address than the one that pointed at it. Resolve alternates to their final URL before comparing sets, and treat the redirect itself as a separate finding routed through the redirect-chain playbook.

One market's score collapses and the site score does not move

Locales are pooled into one baseline. Set INTL_BASELINE_PER_LOCALE and recompute; a market is only comparable against its own history, because language, device mix and network conditions all differ structurally between regions.

The German site is crawled thoroughly and the Japanese one barely at all

A single crawl budget shared across ccTLDs is consumed by whichever market has the most URLs. Separate the crawls and allocate per site, as described in defining crawl depth and scope for enterprise sites.

Alternate sets are correct in the HTML and absent from the sitemap

Both are valid places to declare alternates and they must agree. A set declared only in HTML is invisible to a sitemap-driven consumer, and one declared only in the sitemap is invisible to anything reading the page. Reconcile both sources and treat a disagreement as a finding rather than picking whichever is more convenient.

FAQ #

Why can a header-carried locale not be covered by one crawl?

Because the same URL serves different content depending on a request header or cookie, which breaks the assumption a crawler is built on — that a URL identifies a page. One crawl can only hold one negotiation preference at a time, so it captures one locale and records it under URLs that also represent three others. The only reliable approach is one crawl per locale with the preference pinned, and an assertion on every record that the served locale matches the requested one.

Should locales share a baseline or have their own?

Their own, in almost every case. Markets differ structurally — device mix, network conditions, page weight from translated content and locally hosted third parties all vary — so a single pooled baseline lets a healthy majority absorb a failing minority. The site-wide composite barely moves while an entire region degrades, and the alert that should have fired never does. Pooling is only reasonable when the markets are genuinely comparable and each is too thin to baseline alone.

How should geo-redirects on entry be handled?

Measure them, then avoid them. Requesting a locale-specific URL from an address that geolocates elsewhere often produces a redirect before anything is measured, which adds one hop to every URL in the run and looks like a site-wide redirect-hygiene problem. Record the entry redirect as its own finding, then seed the crawl with locale-specific paths and, where possible, run it from an address the origin geolocates into the intended market.

Do alternate declarations in HTML and in the sitemap need to agree?

Yes, and a disagreement is itself a finding. Both are valid places to declare locale alternates, and consumers differ in which they read: a set declared only in HTML is invisible to a sitemap-driven consumer, and one declared only in the sitemap is invisible to anything parsing the page. Reconcile both sources during the audit and report the divergence rather than quietly preferring whichever is easier to parse.

---

Related #

• Technical Audit Fundamentals & Scope Mapping — the parent section covering audit charter, scope definition and risk scoring
• Defining Crawl Depth & Scope for Enterprise Sites — the boundary definition each locale crawl inherits
• Managing Crawl Budget & Rate Limiting — per-site allocation, which country-code domains need separately
• Normalizing Performance Data Across Device Types — the same stratification argument applied to devices rather than markets