POST /v1/evals with eval_kind: "criterion") grades each model’s fresh answers pass/fail and reports two
numbers per model: the judge’s observed pass rate, and the Rogan–Gladen
corrected rate, what a human would most likely have scored given the
judge’s measured error rates, each with a 95%
interval. When a judge has no usable signal (TPR + TNR ≈ 1), errorbar refuses
to correct rather than amplify noise.
Assertions are the deterministic half. Ten kinds, up to 10 per run,
free, exact, validated at creation (an uncompilable regex or unparseable
schema is a 400, never a queued-broken run), and run against every
generated response’s final answer with reasoning stripped, the same text the
judge sees:
completed fails any output cut off by the token limit
(finish_reason: length) or a reasoning trace that never produced a final
answer. This is the failure every text-reading judge misses (a truncated
answer is usually fluent right up to the cliff), so results also report
a mechanical cut-off count per model whether or not you add the
assertion.
tool_called and no_tool_call match the canonical action notation,
[tool call] name(args), which every generated answer and stored reference
renders tool calls in, so they match on a format the platform controls, not
on model prose. They only make sense on runs whose samples carry tool
definitions; on toolless samples tool_called fails and no_tool_call
passes, both honestly. See
tool-using traffic.
The CI gate
GET /v1/evals/{id}/gate turns any finished run into a deploy decision:
The certified switch test
When the decision is “replace the model behind this traffic”, run the comparison in its strictest shape: set the baseline to the literal__stored__. Your logged answers become the incumbent arm; the judge grades
what you actually shipped against fresh candidate answers on the same prompts,
and three extra rules apply:
- A calibrated judge is required. No trusted instrument, no run.
- Population matching is strict. Where an ordinary run warns when the judge’s calibration population and the sampled traffic don’t match, a certified run refuses: a switch decision must be measured on the population it will affect, not a blend.
- The gate is a noninferiority test, not a raw comparison. The margin is scaled by
the judge’s own measured error (its Youden index), so a noisier judge must
clear a wider bar. If the sample is too small to conclude at that bar, the
gate answers
412(underpowered, refused) instead of pretending.
Evidence bundles
A verdict that cannot leave the platform is a verdict a reviewer has to take on faith.GET /v1/evals/{id}/evidence returns one JSON document for a
finished run that links the claim, the instrument, the data, and the
integrity of the log they sit in, so a result can be forwarded to a reviewer,
a customer, or an auditor with everything that backs it attached:
DONE runs only. Anything else answers 412, the
same fail-closed semantic as the gate: an unfinished run is evidence of
nothing, and a pipeline polls until it gets a verdict.
A trimmed excerpt (one sample shown, certificate abbreviated):
Transcripts
Pass?with_content=true to add each sampled prompt, the stored reference,
and every model’s generated answer to the lineage. It is honored only when
the workspace has request logging enabled:
content export follows the same opt-in as content storage, and
content.reason states which case applied. Lineage (ids, verdicts,
orderings) is always included and stands on its own.
What the attestation proves
Audit rows are hash-chained: each row’s hash covers its content plus the previous row’s hash, with strictly monotonic sequence numbers. Bundling runs the verification live, rehashing every chained row and checking linkage and sequence up to the current head. Editing a row breaks its own hash, deleting one leaves a sequence gap, and reordering breaks the linkage, soattestation.ok is a statement that the log the run sits in is intact, not
a formality.
chain_head is the anchor for comparing exports. Export a bundle today and
another one later: the later chain head must extend the earlier one, because
an append-only log only grows. A later export whose chain does not extend
this one indicates tampering, not routine growth. A verification that could
not run reports ok: false with the reason; it is never silently omitted.
The bundle is deterministic given the stored data. Between two exports of
the same run, only
generated_at and the dated live-standing sections (the
certificate and the chain verification) change, and they should: they are
attestations about now, not part of the frozen claim. The same document is
downloadable from the eval report dialog in the dashboard, in the same
format.