11 min read

Scrapy vs Playwright for JS-Heavy Audits

Every crawl-tooling team eventually asks the same question: does this audit need a real browser, or is a fast HTTP fetch enough? Get the answer wrong in one direction and you burn hours re-rendering static blog pages through a full Chromium instance; get it wrong in the other and you silently miss the hydrated product grid that only exists after JavaScript runs. This guide is part of Configuring Headless Browsers for JS-Heavy Sites and gives you a comparison matrix, a routing decision you can drop into a crawl pipeline today, and the failure modes that show up once the routing logic meets a real site.

Scrapy vs Playwright Routing Decision Diagram showing an incoming crawl URL evaluated against a needs_js flag. A "no" branch routes to a Scrapy static-fetch path; a "yes" branch routes to a Playwright browser-render path. Both paths converge into a single unified crawl output. Incoming URL from frontier needs_js flag? no Scrapy path static HTML, fast high throughput yes Playwright path renders JS, waits higher cost, slower Unified crawl output one schema

Comparison Matrix

Both tools crawl URLs and produce structured output, but they solve different problems. Scrapy is an HTTP-first crawling framework; Playwright is a browser-automation library repurposed as a crawler. The matrix below is what should actually drive the decision, not tool familiarity.

Dimension Scrapy Playwright
Rendering fidelity Raw HTML and XHR responses only — no JavaScript execution Full Chromium/Firefox/WebKit render — executes JS, waits for hydration
Throughput per worker Roughly 500-2,000 requests/min on static templates Roughly 20-80 pages/min per browser context
Memory per worker ~50-150 MB ~300-600 MB per open browser context
Setup complexity Low — pip install scrapy, a spider class, done Higher — browser binaries, wait_until strategy, viewport and network idle tuning
JS-rendered content Missed entirely: client-side routing, lazy-loaded blocks, hydrated widgets Captured after a configurable wait state
Screenshot / visual diff Not supported natively Native page.screenshot() per URL
Best URL class Static templates, server-rendered pages, sitemaps, feeds, docs Single-page app routes, hydration-heavy pages, infinite scroll, client-rendered facets
Cost per 10,000 URLs Low — a single small worker pool High — needs a worker pool plus a browser context pool

Neither tool "wins" outright. A site with a server-rendered marketing section and a React-based product catalog needs both, which is why the routing decision below matters more than the tool comparison itself.

Routing Decision: Implementation

The pattern is a classifier that tags each URL with a needs_js flag before it ever reaches a crawler, then dispatches it to the matching queue. Classification order matters: an explicit allow-list beats a path pattern, and a path pattern beats a runtime heuristic, because runtime heuristics are the slowest and least reliable signal.

set -euo pipefail
# /opt/audit/js_routing — pinned dependency versions
python3 -m venv /opt/audit/js_routing/.venv
/opt/audit/js_routing/.venv/bin/pip install --quiet \
  scrapy==2.11.2 \
  playwright==1.45.0 \
  redis==5.0.7
/opt/audit/js_routing/.venv/bin/playwright install --with-deps chromium
#!/usr/bin/env python3
# /opt/audit/js_routing/route_by_needs_js.py
"""
Classify each crawl-frontier URL as needs_js True/False, then dispatch
to the Scrapy fast path or the Playwright render path.
"""
import json
import os
import re
from dataclasses import dataclass

# Patterns that reliably indicate a URL requires a full browser render.
JS_HEAVY_PATH_PATTERNS = [
    re.compile(r"^/app/"),        # SPA shell mounted under /app/
    re.compile(r"^/search\?"),    # client-side filtered search results
    re.compile(r"/product/.*#"),  # hash-routed product variants
]

# Explicit allow-list from a prior manual audit — always trusted over heuristics.
KNOWN_STATIC_TEMPLATES = {"/blog/", "/docs/", "/legal/"}


@dataclass
class RoutedURL:
    url: str
    needs_js: bool
    reason: str


def classify(url: str, path: str, response_snippet: str) -> RoutedURL:
    # 1. explicit allow-list wins first — cheapest and most reliable signal
    if any(path.startswith(p) for p in KNOWN_STATIC_TEMPLATES):
        return RoutedURL(url, False, "known_static_template")

    # 2. pattern match against known JS-heavy route shapes
    if any(p.search(path) for p in JS_HEAVY_PATH_PATTERNS):
        return RoutedURL(url, True, "js_heavy_pattern")

    # 3. fallback heuristic — a near-empty <body> on first pass means content
    #    is injected client-side after hydration
    visible_text_len = len(re.sub(r"<[^>]+>", "", response_snippet).strip())
    if visible_text_len < 200:
        return RoutedURL(url, True, "empty_body_heuristic")

    return RoutedURL(url, False, "default_static")


def dispatch(routed: RoutedURL) -> None:
    queue = "playwright_render_queue" if routed.needs_js else "scrapy_fast_queue"
    payload = json.dumps({"url": routed.url, "reason": routed.reason})
    os.system(f"redis-cli -n 2 LPUSH {queue} '{payload}'")
    print(f"[{queue}] {routed.url} ({routed.reason})")


if __name__ == "__main__":
    import sys
    for line in sys.stdin:
        url, path, snippet = line.rstrip("\n").split("\t", 2)
        dispatch(classify(url, path, snippet))

Line by line: step 1 checks KNOWN_STATIC_TEMPLATES, a set you build up manually from prior audits and never re-derive automatically, because it is your ground truth. Step 2 matches JS_HEAVY_PATH_PATTERNS — route shapes you already know are client-rendered, such as an /app/ shell or hash-routed variants that never appear in a raw HTML response. Step 3 is the fallback: if neither the allow-list nor the pattern list matches, a first-pass HTTP fetch (done cheaply, without a browser) is stripped of tags and measured; a body under 200 visible characters almost always means the real content loads after JavaScript executes. dispatch() pushes the URL and the reason it was classified that way onto one of two Redis lists that a Scrapy spider and a Playwright worker pool each drain independently, feeding the same distributed crawl orchestration layer that already shards work across your crawl workers. Recording reason alongside the flag is what makes the verification step below possible — you can audit why a URL landed where it did, not just where it landed.

Verification

Run this after every classifier deploy and after any change to JS_HEAVY_PATH_PATTERNS or the allow-list:

set -euo pipefail
cd /opt/audit/js_routing

# 1. confirm both queues received work
SCRAPY_COUNT=$(redis-cli -n 2 LLEN scrapy_fast_queue)
PW_COUNT=$(redis-cli -n 2 LLEN playwright_render_queue)
echo "scrapy_fast_queue=$SCRAPY_COUNT playwright_render_queue=$PW_COUNT"
[ "$SCRAPY_COUNT" -gt 0 ] && [ "$PW_COUNT" -gt 0 ] && echo "PASS: both paths populated"

# 2. spot re-render check — sample 25 URLs flagged needs_js=False
# and re-render them through Playwright to confirm nothing was missed
.venv/bin/python spot_check_static_flags.py \
  --sample-size 25 \
  --queue scrapy_fast_queue \
  --max-text-delta-pct 5

Expected output on a healthy routing pass:

scrapy_fast_queue=8412 playwright_render_queue=1103
PASS: both paths populated
Sampled 25 URLs flagged needs_js=False
Mean visible-text delta vs Playwright render: 1.2%
PASS: all samples within 5% tolerance

A delta consistently above the tolerance on any single path prefix means that prefix belongs on the JS-heavy pattern list, not the default static path — move it and re-run the classifier before trusting the crawl output for scoring.

Failure Modes

A hybrid template ships server-rendered above the fold and client-rendered below it

The classifier's empty_body_heuristic only catches near-empty bodies. A page that renders 300 characters of hero copy server-side but loads its entire product comparison table client-side passes the visible-text-length check and gets misrouted to Scrapy, silently dropping the table from every audit.

# diagnostic: compare Scrapy and Playwright link counts for a suspect path
.venv/bin/python spot_check_static_flags.py --path-prefix /compare/ --mode links
# fix: add the prefix to JS_HEAVY_PATH_PATTERNS and re-run the classifier

The Playwright queue backs up faster than the Scrapy queue drains

Because a Playwright worker processes roughly two orders of magnitude fewer pages per minute than a Scrapy worker, a naive shared worker pool starves the render path. The crawl finishes "on time" by Scrapy's clock while thousands of JS-heavy URLs sit unprocessed in playwright_render_queue.

# diagnostic: check queue depth trend over the run
watch -n 30 'redis-cli -n 2 LLEN playwright_render_queue'
# fix: scale the Playwright worker pool independently and cap Scrapy
# concurrency so it cannot outrun the render path's SLA

The needs_js flag goes stale after a site migration

A section that was server-rendered at classification time later migrates to a client-side framework. The allow-list entry is never revisited, so every audit after the migration quietly routes those URLs to Scrapy and reports a health score built on pre-migration content.

# fix: schedule a quarterly full re-classification pass that ignores
# the existing allow-list and re-derives it from a fresh sample
.venv/bin/python route_by_needs_js.py --ignore-allow-list --sample-only

FAQ

Can Scrapy render JavaScript on its own?

No. Scrapy's downloader fetches raw HTTP responses and never executes JavaScript. Some setups pair it with Splash or a scrapy-playwright middleware, but at that point you are running a browser anyway and should treat those URLs as the Playwright path rather than pretending plain Scrapy alone handled them.

Why not just run everything through Playwright to be safe?

Cost and time. A Playwright browser context uses roughly four to six times the memory of a Scrapy worker and processes tens of pages per minute instead of hundreds to thousands. On a 200,000-URL enterprise crawl that difference is the gap between a nightly run finishing on schedule and one that spills into business hours, which is exactly the throughput problem that handling dynamic content in automated crawls works through in more detail.

How do I set the needs_js flag correctly for a URL I have never crawled before?

Default new, unclassified path prefixes to needs_js=True for the first crawl only. Compare the Scrapy and Playwright output for that sample; if the visible text and internal link count match within a small tolerance, downgrade the prefix to needs_js=False in the allow-list. This costs one extra Playwright pass per new template but prevents silently missing hydrated content the first time a section is crawled.