Skip to main content
Monitor online (per judge) scores a deterministic sample of fresh logged traffic continuously: capped per tick, billed as normal usage, and never double-judging a request. Trace-unit judges monitor completed runs: a trace quiet for 10 minutes is judged once, as a whole trajectory, keyed to the trace so a straggler step never causes a re-judge. The rolling corrected pass rate feeds:
  • quality_low alerts: fired when the rolling corrected pass rate (its point estimate, over the last 24 hours) drops under your floor with at least your minimum number of judgments. The minimum is the noise guard: a bad hour on 8 samples never reaches the check. An alert is a prompt to look, not a verdict, so it fires on the best estimate; the interval rides along in the alert detail. (Firing on the CI lower bound would page a workspace more the less precisely its judge is calibrated, since honest corrected intervals widen with calibration uncertainty.)
  • The quality-gated canary: set gate_* fields on a model alias canary split and the online scorer judges both arms of the live split separately. The gate:
    • holds until both arms have gate_min_samples scored requests,
    • rolls back when the canary’s corrected pass rate (point estimate) sits below gate_rollback_threshold. Rollback is the safe action (the champion keeps serving; only the experiment stops), so it triggers on the best estimate saying “below the floor”, not on proof beyond doubt. Rollback is checked before promotion, so a measurably bad canary never promotes.
    • promotes only when the canary’s observed-rate Wilson lower bound reaches the incumbent’s observed point estimate: demonstrably not-worse, not point-estimate lucky. The comparison runs on observed rates, not corrected ones, deliberately: both arms are scored by the same judge, and the correction is one affine map applied to both sides, so the ordering is identical on either scale while any error in the judge’s measured rates is common-mode and cancels. Corrected numbers are for display and for absolute thresholds like the rollback line.
    • With an unvalidated judge (no measured TPR/TNR) the gate runs on observed rates throughout and the verdict says so.
Every gate lives on the Live switches tab (arms, sample counts, and the latest verdict) and the same gate config is editable in the alias dialog on the Models page, not only via the API. Trace-quality gating: point gate_criterion_id at a trace-unit judge and the gate scores completed runs of the alias’s traffic: a model swap evaluated on whole agent runs or conversations. Arm attribution is honest or absent: a run whose steps touched both models counts toward neither. Monitor config: everything in the Monitor dialog (online_enabled, online_percent, the population (tag and segment), and the weekly spend ceiling online_cap_usd, 0 = uncapped) is equally settable via PATCH /v1/criteria/{id}, same validation, no forked rules. When a judge’s online judging bill reaches its cap, that judge pauses until the week rolls over, and everything else keeps running. Every billed micro-cent counts, judge retries included. Changing the cap never voids calibration (money config is not instrument config); changing the population does. gate_mode: "recommend" (default) surfaces verdicts on GET /v1/aliases and via your alert email/webhook; you click the button. "auto" lets the gate repoint the alias itself, always audited with the numbers, and requires a Calibrated judge (API trust: "trustworthy": both error rates proven at the 90% bar on their CI lower bounds, see the trust badge) that is not currently drift-flagged. A judge with a weaker or stale signal may recommend, but it cannot repoint traffic itself.

​
Drift detection

Corrected pass rates rely on the judge’s TPR/TNR staying true on live traffic. When your traffic shifts (a release, a new failure mode, a change in what users ask), a judge that was calibrated last month may quietly drift, and a stale correction would report confident numbers that are wrong. That is what errorbar watches for. A flagged judge shows drift_status: "flagged" with a drift_signal naming which of four checks raised it: The review queue prompts you to re-calibrate on a fresh slice of grades, recomputing a current, localized TPR/TNR. Drift is derived on read: a calibration newer than the flag supersedes it immediately, on every surface (dashboard, GET /v1/criteria, the certificate, and every gate). The instrument that catches drift in your models is held to the same standard itself: a validated number that goes stale gets caught and re-measured, not trusted indefinitely.

​
Segment quality

Live quality by segment (Dashboard → Evals → Results) answers “which segments or tags are failing?” from the last 7 days of online judged verdicts, grouped by task tag and by auto-detected traffic segment. Rows sort worst first by each rate’s CI upper bound, so a segment proven bad even under its most optimistic reading outranks a thin sample that merely might be bad. The rules that keep the view honest:
  • One row is one population under one judge. Two judges’ verdicts are never pooled into a single rate: each judge has its own measured error rates, and a blend of two instruments’ corrected numbers is a number no calibration backs. A request can appear under both its tag and its segment, so volumes don’t sum across rows.
  • Population binding holds at display time. A tag-scoped judge speaks only for its tag row and a segment-scoped judge only for its segment row. Verdicts that reached a scoped judge from outside its calibrated population are dropped and counted in the footer, never blended in. Verdicts whose segment membership can’t be proven (content aged out, log store unreachable) are dropped as unattributed rather than counted into a row their calibration may not cover.
  • Corrected where warranted, null with a reason where not. Rows show error-corrected rates only when the judge’s calibration supports them. For an uncalibrated judge, a judge whose measured accuracy is too close to chance, or a drift-flagged calibration, the row shows the observed rate and states why the correction is suspended, because a corrected number from an untrusted instrument is exactly the lie this view exists to stop.
  • Trends are claimed, not implied. The trend compares observed rates across the current and prior windows of the same judge (its error is common-mode and cancels), and a direction is claimed only when the two intervals are disjoint. A window too thin to compare reads “unknown”, not “flat”.
  • Thin groups are counted, not shown. Groups under 5 verdicts would let one flaky answer own the rate; the footer reports how many were hidden, along with off-population drops, unattributed verdicts, and any window truncation.
Each row links to the Logs view filtered to that tag or segment, so a bad rate is one click from the traffic behind it.

​
Design rules

  • Binary verdicts everywhere. Pass/fail aligns with humans; 1–5 scores don’t.
  • Judges rank and monitor in aggregate; they never gate a single request. Inline guardrails are deterministic; evaluators are sampled and statistical.
  • Promotion needs proof; stopping doesn’t. Promotion gates on an interval bound. Rollback and quality_low fire on the best estimate, because the safe action should not wait for statistical certainty.
  • Partial responses are review signals, never data. Aborted streams are visible in the review queue and Logs (opt-in view) but structurally excluded from datasets, exports, comparison sampling, and online scoring.
  • The human stays the judge. Suggestions, scans, and auto-improve amplify your grades; nothing auto-grades, and only a calibrated judge in explicit auto mode may touch routing.