11 min read

Parquet vs JSONL for Crawl Artifact Storage

Crawl artifacts get written once and read many times, usually by something that wants three fields out of forty. That asymmetry is what the format choice is really about — not storage cost, which compression largely settles, but whether a reader has to decompress every field of every row to answer a question about two of them. This page is part of Storing & Versioning Crawl Artifacts in Cloud Storage.

The same million rows, four ways A horizontal bar chart of stored size for one million crawl records. Uncompressed JSON Lines is about 2100 megabytes. Gzipped JSON Lines is about 310. Parquet with Snappy compression is about 180. Parquet with ZSTD is about 130. The chart notes that only the Parquet rows also allow reading three columns without decompressing the other forty. STORED SIZE FOR 1,000,000 CRAWL RECORDS JSONL, uncompressed ~2,100 MB JSONL + gzip ~310 MB Parquet + Snappy ~180 MB Parquet + ZSTD ~130 MB Size is the less important axis. Gzipped JSONL is close enough on storage; what it cannot do is let a query read three columns out of forty-three without decompressing every byte of the file.

Comparison Table

Dimension Parquet JSON Lines
Layout Columnar, in row groups Row-oriented, one JSON object per line
Reading a subset of columns Reads only those columns Reads and parses every field
Schema Typed and embedded; enforced at write None; each line stands alone
Schema drift Fails loudly at write or read Absorbed silently until something downstream breaks
Predicate pushdown Row-group statistics let readers skip blocks None; every line must be parsed
Compression Per column, so like values compress together Whole-file, over mixed types
Appending Not incremental; a file is written whole Natural — one line at a time
Shell tooling Needs a library or a CLI head, grep, jq, wc -l all work
Typical size, 1M rows 130-180 MB 310 MB gzipped, 2.1 GB raw
Best role The analytical copy that gets queried The transport copy the crawler writes
Where each format is actually better Two columns. Parquet is columnar so a query reads only the columns it needs, carries a typed schema, stores per-column statistics that let a reader skip row groups, and is awkward to inspect without a library. JSON Lines is row-oriented so any read touches every field, is schema-free and therefore tolerant of drift, is greppable and streamable with ordinary shell tools, and appends cleanly one record at a time. PARQUET Columnar read 3 columns of 43, not all of them Typed schema, enforced a type change fails at write time Row-group statistics readers skip blocks that cannot match Needs a library to inspect no head, no grep, no tail JSON LINES Row-oriented every read touches every field Schema-free tolerant of drift — for better and worse Greppable and streamable head, jq, tail -f all work Appends one record at a time a crawler can write as it goes This is why most pipelines end up using both: JSON Lines as the crawler writes, Parquet as the analytical layer reads, with one conversion step between them.

The pattern most pipelines converge on

Rather than choosing, use each where it is strong: the crawler streams JSON Lines as it goes, and a conversion step produces the Parquet the scoring layer reads.

#!/usr/bin/env python3
# /opt/audit/artifacts/convert.py
# JSONL -> Parquet with an explicitly pinned schema, so type drift fails the
# conversion rather than landing silently in a partition.
from __future__ import annotations
import os
import sys
import pyarrow as pa
import pyarrow.json as paj
import pyarrow.parquet as pq

# The schema is declared, not inferred. Inference reads the first block and
# will happily type a column int64 today and double tomorrow, producing
# partitions that cannot be read together.
SCHEMA = pa.schema([
    ("url", pa.string()),
    ("final_url", pa.string()),
    ("status", pa.int32()),
    ("depth", pa.int16()),
    ("lcp_ms", pa.float64()),
    ("cls", pa.float64()),
    ("indexable", pa.bool_()),
    ("canonical_url", pa.string()),
    ("crawled_at", pa.timestamp("us", tz="UTC")),
])


def convert(src: str, dst: str, compression: str = "zstd") -> int:
    table = paj.read_json(
        src,
        parse_options=paj.ParseOptions(explicit_schema=SCHEMA,
                                       unexpected_field_behavior="error"),
    )
    if table.schema != SCHEMA:
        sys.exit(f"FAIL: schema mismatch in {src}\n{table.schema}")
    pq.write_table(table, dst, compression=compression,
                   use_dictionary=["status", "canonical_url"],
                   row_group_size=100_000)
    return table.num_rows


if __name__ == "__main__":
    n = convert(sys.argv[1], sys.argv[2])
    print(f"converted {n:,} rows -> {sys.argv[2]}")

The decisions that matter:

  • explicit_schema rather than inference. Inference reads only the first block, so a column that is integer for the first hundred thousand rows and floating point afterwards produces a file whose type depends on batch ordering. Pinning it turns that into a conversion failure with a clear message.
  • unexpected_field_behavior="error". A new field appearing in the crawler output is a schema change and should be a deliberate one. Silently ignoring it means the field never reaches the analytical layer and nobody notices for a quarter.
  • use_dictionary on low-cardinality columns. Status codes and canonical URLs repeat heavily; dictionary encoding is where most of the size advantage over gzipped JSONL actually comes from.
  • row_group_size=100_000. Row groups are the unit a reader can skip. Too large and predicate pushdown stops helping; too small and the per-group metadata overhead grows.

Verification 

set -euo pipefail
SRC=/data/crawl/2026-07-31/pages.jsonl
DST=/data/crawl/2026-07-31/pages.parquet
python3 -m audit.artifacts.convert "$SRC" "$DST"

# 1. Row counts match exactly.
JSONL_ROWS=$(wc -l < "$SRC")
PARQ_ROWS=$(python3 -c "import pyarrow.parquet as pq,sys;print(pq.read_metadata(sys.argv[1]).num_rows)" "$DST")
[ "$JSONL_ROWS" -eq "$PARQ_ROWS" ] || { echo "FAIL: $JSONL_ROWS vs $PARQ_ROWS"; exit 1; }
echo "PASS: $PARQ_ROWS rows preserved"

# 2. The schema matches the pinned one, field for field.
python3 -c "
import pyarrow.parquet as pq, sys
from audit.artifacts.convert import SCHEMA
got = pq.read_schema(sys.argv[1])
assert got == SCHEMA, f'schema drift:\n{got}'
print('PASS: schema matches the pinned definition')" "$DST"

# 3. A columnar read touches only the columns asked for.
python3 -c "
import pyarrow.parquet as pq, sys
t = pq.read_table(sys.argv[1], columns=['status','lcp_ms'])
assert t.num_columns == 2, t.schema
print(f'PASS: read 2 of {len(pq.read_schema(sys.argv[1]))} columns')" "$DST"

Expected output is three PASS lines. The schema assertion is the one to keep in CI permanently — it is the only cheap defence against a type change that will not surface until a query spans two partitions written months apart.

Failure Modes

Three ways the format choice bites Three rows. Writing Parquet directly from a crawler means holding a batch in memory and losing everything in it when the process dies. Keeping JSONL as the analytical format means every query decompresses every field of every row. And converting without pinning the schema lets a type change land silently, so a later read fails on partitions written months apart. DECISION WHAT IT COSTS FIX Parquet straight from the crawler, with no intermediate file A crash loses the batch rows are buffered until a row group is complete Write JSONL first convert after the crawl exits cleanly JSONL as the analytical format, queried directly by the scoring job Every query reads everything forty-three fields decompressed to compute one average Convert once and query the columnar copy, not the transport one Converting without pinning the schema explicitly Type drift lands silently one partition has int64, the next has float64, and a join fails Pin the schema and fail the conversion when the inferred type differs The third is the one that surfaces months later, when a query spanning two quarters fails on a type mismatch that was introduced by a single batch nobody noticed.

A read spanning several months fails on a type mismatch

One partition has a column as int64 and another as double, because an early conversion used inference. Rewrite the affected partitions against the pinned schema rather than casting at read time; a cast at read time has to be repeated by every consumer forever.

The Parquet file is larger than the gzipped JSONL

Almost always a row group size far smaller than the row count, so per-group metadata dominates, or dictionary encoding disabled on the columns that would benefit most. Check with pq.read_metadata(path).row_group(0) and compare the number of row groups against the row count.

The crawler crashes and the batch is gone

Parquet is written whole, so rows buffered toward an incomplete row group are lost. This is exactly why the crawler should write JSON Lines and the conversion should run after the crawl exits cleanly — the transport format needs to be append-safe, and the analytical format does not. Retention for both is covered in the parent guide, and only the columnar copy needs to survive long enough for drift-detection queries to run against it.

FAQ

Why not write Parquet directly from the crawler?

Because Parquet is written whole. Rows accumulate toward a complete row group, and if the crawler process dies before the group is flushed, every buffered row is gone. On a long crawl that can be tens of thousands of records. JSON Lines appends one record at a time, so a crash costs at most the record in flight, which is why the transport format and the analytical format are usually different files with one conversion between them.

Is the size difference the main reason to use Parquet?

No. Gzipped JSON Lines is close enough on storage that the difference rarely justifies a conversion step on its own. The reason is columnar reads: a scoring job that needs status and LCP out of forty-three fields reads two columns from Parquet and decompresses every byte of a gzipped JSONL file to get the same answer. On repeated queries over months of partitions, that difference compounds far past the storage saving.

Why pin the schema instead of letting the writer infer it?

Because inference reads only the first block of input, so the type of a column depends on which rows happen to arrive first. A field that is integer for the first hundred thousand records and floating point afterwards produces a file typed by batch ordering, and the failure appears much later when a query spans two partitions typed differently. Pinning turns an invisible data defect into a conversion error with a clear message on the day it is introduced.