12 min read

Handling Small Sample Sizes in Percentile Buckets

A percentile computed over a thousand observations is a statement about a distribution. The same percentile computed over eight observations is roughly one of those eight observations, wearing a label that implies far more. Both arrive in the same column of the same table, formatted identically, and every downstream consumer treats them the same way. This page is part of Percentile Normalization Across Metric Distributions and covers what to do about thin buckets.

How much a p75 can be wrong A horizontal bar chart of the approximate width of a 95 percent confidence interval around a p75 estimate at five sample sizes, expressed in percentile points. At five samples the interval spans roughly 55 points, at twenty roughly 28, at fifty roughly 18, at two hundred roughly 9, and at a thousand roughly 4. The estimate at five samples is barely a statement about the distribution at all. APPROXIMATE 95% INTERVAL WIDTH AROUND A p75 n = 5 ± ~27 pts n = 20 ± ~14 pts n = 50 ± ~9 pts n = 200 ± ~4 pts n = 1,000 ± ~2 pts At five samples the p75 could be almost anywhere in the distribution. Publishing that number next to one computed from a thousand samples, in the same column, is the problem this page is about.

Environment isolation and dependency declaration 

set -euo pipefail
export PCT_MIN_SAMPLES=200         # publish normally at or above this
export PCT_SOFT_MIN=100            # publish with an interval between the two
export PCT_ROLLUP_ORDER="segment,template,site"   # what a thin bucket rolls into
export PCT_ALERT_MIN_SAMPLES=200   # never alert below this, whatever is published
/opt/audit/.venv/bin/pip install "numpy==1.26.4" "pandas==2.2.2"

PCT_ALERT_MIN_SAMPLES being separate from the publication threshold is deliberate. A low-confidence estimate is still worth showing to a human who can read the interval next to it; it is never worth paging on, because the interval is wider than any threshold you would set.

Implementation 

#!/usr/bin/env python3
# /opt/audit/scoring/thin_buckets.py
# A percentile is an estimate. Carry its uncertainty and its sample count.
from __future__ import annotations
import math
import os
import numpy as np
import pandas as pd

MIN_N = int(os.environ.get("PCT_MIN_SAMPLES", "200"))
SOFT_N = int(os.environ.get("PCT_SOFT_MIN", "100"))
ROLLUP = os.environ.get("PCT_ROLLUP_ORDER", "segment,template,site").split(",")


def percentile_ci(values: np.ndarray, q: float = 0.75,
                  conf: float = 0.95) -> tuple[float, float, float]:
    """Point estimate plus a distribution-free interval from the binomial
    order statistics — no normality assumption, which latency data violates."""
    x = np.sort(values)
    n = len(x)
    if n == 0:
        return (math.nan, math.nan, math.nan)
    point = float(np.quantile(x, q))
    if n < 2:
        return (point, math.nan, math.nan)
    z = 1.959963985 if conf == 0.95 else 1.6448536
    spread = z * math.sqrt(n * q * (1 - q))
    lo_i = max(0, int(math.floor(n * q - spread)))
    hi_i = min(n - 1, int(math.ceil(n * q + spread)))
    return point, float(x[lo_i]), float(x[hi_i])


def summarise(df: pd.DataFrame, metric: str, key: str) -> pd.DataFrame:
    rows = []
    for name, g in df.groupby(key):
        vals = g[metric].dropna().to_numpy()
        point, lo, hi = percentile_ci(vals)
        n = len(vals)
        rows.append({
            key: name, "n": n, "p75": round(point, 1),
            "p75_lo": round(lo, 1), "p75_hi": round(hi, 1),
            "confidence": ("high" if n >= MIN_N
                           else "low" if n >= SOFT_N else "insufficient"),
            "alertable": n >= int(os.environ.get("PCT_ALERT_MIN_SAMPLES", "200")),
        })
    return pd.DataFrame(rows)

The parts that carry it:

  • percentile_ci is distribution-free. It uses binomial order statistics rather than assuming normality, because latency and layout-shift distributions are heavily right-skewed and a normal-theory interval is wrong in the direction that matters.
  • n is a column of the output, not a footnote. Every consumer that formats a p75 can also format its sample count, and the ones that do not at least cannot claim they were not given it.
  • confidence has three levels rather than two. The middle level exists because a bucket that is thin but not hopeless is genuinely useful to a human reading an interval and genuinely useless as an alert input.
  • alertable is computed separately from confidence. Publication and alerting have different requirements, and conflating them means either alerting on noise or hiding usable information.
What a thin bucket earns A question card asks how many samples the bucket holds against the minimum, with three outcomes. At or above the minimum the percentile is published normally. Between half the minimum and the minimum it is published with an explicit confidence interval and a low-confidence flag. Below half the minimum the bucket is rolled up into its parent, and if no parent has enough samples either, no percentile is published for it at all. How many samples does this bucket hold? at or above the minimum Publish the estimate is stable enough to compare and to alert on half the minimum to minimum Publish with an interval flagged low-confidence, never bare below half the minimum Roll up, or suppress a percentile over a handful of samples is one of them Suppressing has to be visible. A bucket that silently vanishes from a report is read as healthy, which is the opposite of what a missing estimate means.

Verification and smoke test 

/opt/audit/.venv/bin/python3 - <<'PY'
import numpy as np
from thin_buckets import percentile_ci

rng = np.random.default_rng(42)
dist = rng.lognormal(mean=7.4, sigma=0.55, size=100_000)   # right-skewed

# 1. The interval narrows as n grows.
widths = []
for n in (5, 20, 50, 200, 1000):
    _, lo, hi = percentile_ci(rng.choice(dist, n))
    widths.append(hi - lo)
assert all(a > b for a, b in zip(widths, widths[1:])), widths
print("PASS: interval narrows monotonically with n:",
      [round(w) for w in widths])

# 2. The interval covers the true p75 most of the time at a usable n.
truth = float(np.quantile(dist, 0.75))
hits = sum(1 for _ in range(500)
           if (lambda r: r[1] <= truth <= r[2])(percentile_ci(rng.choice(dist, 200))))
assert hits >= 450, hits
print(f"PASS: interval covered the true p75 in {hits}/500 samples at n=200")
PY

Expected output shows the interval narrowing sharply across the five sample sizes and a coverage rate near 95 percent. The coverage check is the one that validates the interval is actually an interval rather than a decoration — a formula that produces plausible-looking bounds with the wrong coverage is worse than no bounds at all.

Failure modes

Three ways a thin bucket does harm Three rows. A thin bucket published without its sample count reads exactly like a well-populated one, so a decision is made on an estimate that could be anywhere. A thin bucket alerting on its own rolling baseline pages constantly, because the baseline moves with every new sample. And a thin bucket that silently disappears from a dashboard is read as having no problems rather than no data. SYMPTOM ROOT CAUSE FIX A segment is prioritised on a number computed from four samples The sample count was dropped the estimate looks identical to one from a thousand samples Carry n everywhere sample count is part of the estimate, not metadata One thin segment pages every night with no real change behind it Its baseline moves each new sample shifts a percentile over a dozen values Do not alert thin buckets roll them up, or exclude them from alerting entirely A section disappears from the dashboard and nobody notices Suppression is invisible an absent row renders as nothing to worry about Render the suppression show the bucket with a reason, never omit the row All three come from treating a percentile as a number rather than as an estimate with an uncertainty attached, which is a distinction that only matters when the sample is small — and is invisible exactly then.

A rolled-up bucket is indistinguishable from a native one

The roll-up worked and nothing recorded that it happened, so a segment reported at template granularity looks like a segment reported at its own. Emit the resolved granularity alongside the requested one, exactly as the parent guide requires for fallback segments.

Confidence intervals are ignored downstream

Predictable, and the fix is not more documentation. Make the alertable flag the thing consumers read, so a thin bucket cannot enter an alert path regardless of whether anyone looked at the interval. Reserve the interval for human-facing surfaces, where it informs rather than gates.

A bucket oscillates between confidence levels

Its sample count sits on a threshold, so ordinary variation flips it nightly. Add hysteresis — two consecutive runs above the threshold to be promoted, two below to be demoted — the same treatment a coverage floor needs for the same reason.

FAQ

Why is a percentile over a small sample so unreliable?

Because a percentile is an order statistic — it picks a position in the sorted sample rather than summarising all of it. Over eight observations the p75 is essentially the sixth value, so it moves by a large amount whenever any single observation changes. The practical consequence is that the confidence interval around a p75 at n equals five spans most of the distribution, so the number carries almost no information while looking exactly like one that carries a great deal.

Should a thin bucket be suppressed or rolled up?

Rolled up where a sensible parent exists, and suppressed only when none does. Rolling a thin segment into its template keeps some signal and is honest as long as the resolved granularity is recorded in the output — otherwise a rolled-up bucket is indistinguishable from a native one. Where no parent has enough samples either, publish nothing, but publish the absence visibly: a missing row on a dashboard reads as no problem rather than no data.

Why separate the alerting threshold from the publication threshold?

Because they answer different questions. A low-confidence estimate shown to a human alongside its interval is genuinely useful — the reader can see the uncertainty and weigh it. The same estimate fed to a threshold comparison is not, because the interval is wider than any threshold worth setting, so the alert fires on sampling variation rather than on the site. Keeping the two thresholds separate lets you show more than you alert on, which is almost always the right balance.