17 min read

Wiring Health Scores into Dashboards & Alerts

A composite health score that exists only in a Parquet file has never changed a decision. Getting it in front of people means two very different consumers: a dashboard, which a human reads when they choose to, and an alert, which chooses to interrupt a human. Both must be reading the same number, computed once, or the first ten minutes of every incident are spent establishing which of the two was telling the truth. This page is part of Metric Scoring & Data Normalization and covers the serving layer that sits between the scoring artifact and both consumers.

One number, two very different consumers Four stacked hops. The scoring artifact is the source of truth and is immutable once written. The serving layer reshapes it for reading without recomputing anything. The dashboard reads that layer and is allowed to aggregate and format but never to re-derive a score. The alert evaluator reads the same layer and is allowed to compare against thresholds but likewise never to recompute. The note beside each hop names the one thing it must not do. Scoring artifact partitioned parquet, written once the only place a score is computed Serving layer a view or a metrics endpoint reshapes for reading, recomputes nothing Dashboard aggregates and formats must not re-derive a score of its own Alert evaluator compares against thresholds reads the same layer the dashboard does The moment a dashboard computes its own version of the composite, the number on the wall and the number in the alert diverge, and every incident review starts with an argument about which one was right.

Prerequisites & environment setup

Tool / library Pinned version Purpose
Python 3.11.9 Serving-layer refresh job
pyarrow 16.0.0 Reading the partitioned score artifact
prometheus-client 0.20.0 Exposing scores as metrics for the alert evaluator
PyYAML 6.0.1 Reading the shared threshold config 
export SCORE_ARTIFACT_DIR="/data/scored"
export SERVING_DB_URL="postgresql://audit@warehouse/serving"
export THRESHOLD_CONFIG="/opt/alert-routing/thresholds.json"   # the ONE copy
export SERVING_STALE_AFTER_S=7200      # alert on staleness past this
export SERVING_REFRESH_LOCK="/var/run/serving-refresh.lock"

THRESHOLD_CONFIG pointing at the alert-routing config rather than a dashboard-local copy is the single most consequential line here. The dashboard should render the thresholds it reads from that file, not restate them — a duplicated number is a number that will eventually differ.

Step 1 — Initialization: build the serving layer

The serving layer is a narrow, denormalised table refreshed once per scoring run. It exists so that neither consumer ever queries the raw artifact and neither ever recomputes anything.

#!/usr/bin/env python3
# /opt/audit/serving/refresh.py — rebuild the read layer from one artifact.
from __future__ import annotations
import os
import datetime as dt
import pyarrow.parquet as pq
import pandas as pd
import sqlalchemy as sa

ART = os.environ["SCORE_ARTIFACT_DIR"]
DB = sa.create_engine(os.environ["SERVING_DB_URL"], future=True)

COLUMNS = ["url", "section", "template", "device", "composite_score",
           "coverage", "imputed_share", "scoring_version", "scored_at"]


def refresh(run_date: str) -> int:
    table = pq.read_table(f"{ART}/date={run_date}", columns=COLUMNS)
    df = table.to_pandas()
    # Freshness is a column, not an inference from the partition name.
    df["served_at"] = dt.datetime.now(dt.timezone.utc)
    with DB.begin() as conn:
        conn.exec_driver_sql("TRUNCATE serving.scores_current")
        df.to_sql("scores_current", conn, schema="serving",
                  if_exists="append", index=False, method="multi")
    return len(df)

Three things are deliberate. COLUMNS is an explicit allowlist, so a new column appearing in the scoring artifact does not silently widen the serving contract. served_at is written as data rather than inferred from a partition name, because a partition named for today can be several hours old and nothing in the name says so. And the refresh runs inside a single transaction with a truncate, so a reader never sees a half-replaced table.

Step 2 — Core configuration

Parameter Type Default Purpose
SERVING_STALE_AFTER_S int 7200 Age past which the layer itself is alerted on
THRESHOLD_CONFIG path — The single source of severity floors, shared with routing
SERVING_SCOPE_LABEL string all Which population this layer covers; rendered on every tile
SERVING_REFRESH_LOCK path — flock guard so two runs never rebuild concurrently
SERVING_RETAIN_RUNS int 90 History kept in the trend table behind the current one

SERVING_SCOPE_LABEL looks cosmetic and is not. A dashboard filtered to one section while alerts evaluate the entire estate produces the most confusing failure in this whole area: a green wall and a firing pager, describing the same site, both correct.

Query the warehouse, or serve a layer Two columns. Querying the warehouse directly means each tile issues its own scan, cost scales with viewers rather than with data, two tiles can disagree because they ran seconds apart, and the dashboard breaks whenever the underlying schema changes. A purpose-built serving layer is refreshed once per scoring run, costs the same regardless of how many people look at it, gives every tile the same snapshot, and insulates the dashboard from schema changes behind a stable contract. DASHBOARD QUERIES THE WAREHOUSE Each tile issues its own scan cost scales with viewers, not data Tiles can disagree two scans, seconds apart, one refresh Schema changes break tiles no contract between the two Slow tiles get muted and then nobody notices they are stale DASHBOARD READS A SERVING LAYER Refreshed once per scoring run cost is fixed and predictable Every tile sees one snapshot the same rows, by construction A stable contract the warehouse can change underneath Freshness is a first-class field staleness is visible, not inferred The freshness field is the part teams add last and need most: a dashboard with no visible timestamp is indistinguishable from a dashboard whose pipeline stopped three days ago.

Step 3 — Execution & scheduling: refresh on the run, not on a timer

Trigger the refresh from the scoring run's completion, not from a schedule of its own. A timer-based refresh drifts relative to scoring and eventually rebuilds the layer from yesterday's artifact for an hour every day.

#!/usr/bin/env bash
# /opt/audit/serving/refresh.sh — called by the scoring job on success.
set -euo pipefail
RUN_DATE="${1:?run date required}"
exec 9>"${SERVING_REFRESH_LOCK}"
flock --nonblock 9 || { echo "refresh already running; skipping"; exit 0; }
/opt/audit/.venv/bin/python3 /opt/audit/serving/refresh.py "$RUN_DATE"

The staleness alert is the counterpart, and it has to be evaluated against the serving layer rather than against the pipeline's own logs — a pipeline that fails silently still writes a log, but it does not update served_at.

-- Alerting query: has the serving layer gone stale?
SELECT EXTRACT(EPOCH FROM (now() - MAX(served_at))) AS age_seconds
FROM serving.scores_current;

Step 4 — Artifact capture & storage

Keep a trend table behind the current one, appended rather than truncated, holding the same columns plus the run date. It is what a dashboard's trend tile reads and what a release comparison joins against, and it is far cheaper than re-reading partitioned Parquet for every chart. Retention of ninety runs matches the scoring artifact retention described in storing and versioning crawl artifacts in cloud storage; past that, the artifact remains the archive of record.

Common mistakes

  • Treating the dashboard as the product and the alert as an afterthought. The dashboard is read when somebody chooses to look; the alert is read when it interrupts somebody. The alert path is the one that has to be correct at three in the morning, so it should be built first and the dashboard should be a second reader of the same layer — not the other way round, with alerting bolted onto a panel definition afterwards.
  • Putting business logic in the visualisation tool. Panel-level expressions that combine metrics, apply weights, or reclassify severity are a second scoring implementation living in a tool with no tests, no review process and no version history that anyone reads. Every such expression is a place where the wall and the pager will eventually disagree. If a number needs computing, it belongs in the scoring pipeline where it can be asserted on.
  • Rendering an average where the pipeline computed a percentile. A tile that shows the mean composite across a section is not a summary of the section's score, it is a different statistic with different behaviour — it moves for reasons the percentile does not, and it will disagree with an alert evaluated on the percentile. Render exactly the statistic the threshold is evaluated against.
  • Leaving the scope of a tile implicit. A panel filtered to one device class, environment or section is answering a narrower question than its title suggests. The filter belongs in the title, not only in the query, so that a green tile cannot be mistaken for a green site.
  • Alerting on the score and not on the pipeline. A scoring run that fails produces no new rows, no new alerts, and a dashboard that keeps rendering the last good snapshot indefinitely. Staleness has to be its own alert with its own threshold, evaluated against served_at, or the most common failure mode of the whole system is also its quietest.
  • Letting the trend chart span a scoring-version boundary without saying so. A weight change produces a step in the series that looks exactly like a site regression. scoring_version travels with every served row precisely so the chart can draw that boundary as an annotation, and so a comparison that crosses it can be refused rather than silently produced.
  • Refreshing the serving layer on a timer. A timer drifts relative to the scoring run it depends on, and eventually spends part of every day serving yesterday's artifact while reporting today's date. Trigger the refresh from the scoring job's success path, keep the flock guard, and let the staleness alert catch the case where the scoring job did not run at all.

Verification checklist

  1. The serving layer is fresher than the staleness threshold. SELECT now() - MAX(served_at) FROM serving.scores_current is under SERVING_STALE_AFTER_S.
  2. Row count matches the artifact. The count in scores_current equals the row count of the source partition. A shortfall means the truncate-and-append raced with a reader or a partial write.
  3. No consumer recomputes a score. Grep the dashboard definitions for arithmetic on component metrics; any expression combining two metrics is a re-derivation and must be replaced with a read of composite_score.
  4. Thresholds resolve to one file. The dashboard config and the routing config both reference THRESHOLD_CONFIG; neither contains a literal floor.
  5. Scope labels agree. SERVING_SCOPE_LABEL matches the filter used by the alert evaluator, and appears on every dashboard tile.
  6. Staleness itself alerts. Stop the refresh job deliberately and confirm a staleness alert fires within one evaluation interval — a dashboard that cannot tell you it is stale is worse than no dashboard.

Troubleshooting

Four ways the wall and the pager disagree Four rows. A dashboard recomputing its own composite drifts from the alert evaluator the moment a weight changes. A tile with no freshness indicator renders a stale snapshot exactly like a live one. Alert thresholds duplicated into the dashboard config drift from the routing config. And a dashboard filtered to a subset while alerts evaluate the whole estate makes a green wall coexist with a firing page. SYMPTOM ROOT CAUSE FIX The tile and the alert disagree on the same metric and the same day The dashboard recomputes it derives its own composite from the component metrics Read, never derive both consumers read one pre-computed serving layer Everything looks healthy for days after the pipeline actually stopped No freshness on the tile a stale snapshot renders exactly like a live one Show the timestamp and alert on staleness itself, not only on the value A threshold change lands in one place and not the other Thresholds are duplicated the dashboard config carries its own copy of the numbers One source, referenced the dashboard reads the routing config, it does not restate it A green wall while the pager is going off about the same site Different populations the dashboard is filtered, the alert evaluates everything State the scope put the filter in the tile title and match it in the alert Every row here is a disagreement about what "the score" refers to. None of them is a bug in the scoring pipeline, and all of them are usually blamed on it.

The dashboard is fast and the numbers are hours old

The refresh is running on a timer that has drifted relative to scoring, so it rebuilds from whichever artifact happens to be latest. Move the trigger into the scoring job's success path and let the staleness alert catch any gap.

Two tiles on the same dashboard disagree

They are querying the warehouse independently and ran either side of a refresh. This is exactly what the serving layer exists to prevent; a tile that still points at the warehouse should be migrated rather than cached.

An alert fires for a section that the dashboard shows as healthy

Compare the populations before comparing the numbers. A dashboard filtered to device = 'desktop' while the alert evaluates all devices is not a disagreement about the score, it is two different questions. Put the filter in the tile title and mirror it in the alert scope, as described in configuring alert thresholds and routing.

A threshold change does not take effect on the dashboard

The dashboard is rendering a cached copy of the threshold file. Reference it at read time and let the dashboard's own refresh interval pick up the change, rather than baking values into a panel definition at deploy time.

The trend chart has a step change with no deploy behind it

The scoring version changed. scoring_version is carried through the serving layer precisely so a chart can annotate this rather than present it as a site regression; render it as a marker and treat comparisons across the boundary as invalid, exactly as tracking metric trends across release cycles describes.

The refresh occasionally leaves the table empty

TRUNCATE followed by an append is atomic only inside one transaction. If the append fails after the truncate commits, the table is empty and every tile renders zero. Confirm both statements share a transaction, and add a post-refresh row-count assertion that rolls back rather than reporting success.

FAQ

Why should a dashboard never compute its own composite score?

Because the alert evaluator computes one too, and the moment a weight or a threshold changes the two versions drift apart. Once they differ, every incident review opens with an argument about which number was correct rather than about what to do. Compute the score exactly once, in the scoring pipeline, and let both consumers read the result — a dashboard is allowed to aggregate and format, never to re-derive.

Why alert on staleness rather than only on the score?

Because a stale dashboard renders exactly like a live one. If the scoring pipeline stops, the last good snapshot keeps displaying healthy numbers indefinitely, and the absence of alerts is indistinguishable from a healthy site. Recording served_at as a column and evaluating its age turns "nothing has fired" into a statement you can trust, which is the only condition under which a green dashboard means anything.

Should the dashboard query the warehouse directly?

Not for anything on the default view. Direct queries mean cost scales with the number of viewers rather than the volume of data, two tiles that run seconds apart can disagree because a refresh landed between them, and every warehouse schema change breaks panels. A serving layer refreshed once per scoring run gives every tile the same snapshot at a fixed cost and puts a stable contract between the dashboard and the underlying tables.

What causes a green dashboard and a firing pager at the same time?

Almost always different populations rather than different numbers. A dashboard filtered to one device class, section or environment while the alert evaluates the whole estate produces two correct answers to two different questions. Render the scope on every tile and mirror it in the alert configuration, so the two can be compared at a glance instead of reconciled during an incident.