Skip to main content
Every RL run on judged work optimises against a grader, and the grader is the part nobody measures. errorbar’s judges are calibrated against your graders’ pass/fail decisions and carry their sensitivity and specificity with intervals. A reward session hands that judge to your trainer as a reward endpoint, with the certificate attached, and errorbar never runs the training — which is what makes the certificate independent.

​
What a session gives you

  • A validated reward. Only calibrated, non-drift-flagged judges bound to the population you are training on. An unvalidated judge is refused with the reason, never downgraded.
  • The certificate. Sensitivity, specificity and κ with intervals, the trust verdict, and what voids it — a prompt or model edit, a population change, a drift flag, revised grades, or the provider retiring the judge model (validity.judge_model_available) — returned at creation, before the first rollout is scored.
  • A ledger-true budget. The reward budget is held in your wallet at creation and every scoring call is metered; when the ledger crosses the ceiling the session refuses (402 budget_exhausted). Your trainer never self-reports spend.
  • Masking, not zeros. A judge that cannot parse a verdict returns grade: null for that item. Mask it out of the loss; treating it as 0 trains the policy to confuse the parser.
  • The anchor. Optionally, a second independent judge that holds the run when the reward diverges from it. See below.

​
Create a session

The response is the session with certificates[] and, if you asked for one, anchor. Sessions expire (ttlHours, default 72) and can be stopped with DELETE /v1/reward/sessions/{id}, which releases the hold.

​
Score a rollout group

step namespaces the billing ids, so re-sending the same step and request ids after a crash settles into the same ledger rows. Agentic sessions send steps[] (assistant and tool turns) instead of response; the server renders the trajectory through the same instrument your trace judge was calibrated on.

​
Adapters

The errorbar_reward Python package (in the errorbar repo under rl/errorbar_reward) exposes the same contract in three shapes:
  • ErrorbarRewardClient — create, read, stop, score, anchor, resume.
  • trl_reward_func(client, session_id) — a TRL GRPOTrainer reward_funcs callable; None grades pass through so TRL masks them.
  • ErrorbarJudge — a Prime Intellect verifiers v1 Judge; a masked item raises, matching verifiers’ rule that a malformed verdict must error the rollout, not score the model.

​
The mid-run anchor

In-loop statistics catch a policy collapsing onto a template. They do not catch the reward rising while true quality does not, because the policy found what the reward judge rewards rather than what the task wants. That is only visible from a measurement the policy never optimised against. The anchor is a second calibrated judge that must be a different instrument from every reward judge — a different model or prompt, not a clone with a new id — aligned to the reward bar, not drift-flagged, on the same unit. Every N optimizer steps your trainer generates one completion per prompt in a frozen prompt set and posts them:
The server scores the outputs twice — with the session’s reward, and with the anchor judge corrected for its measured error — and reads two signatures over the history, each measured against its own noise rather than a fixed margin:
  • Gap. The reward’s read minus the anchor’s read on the same outputs, compared with the run’s first check. Because both reads are on the same items, what is common to them cancels and only their disagreement is left, which is what a fooled grader changes. The gap must widen by more than the 95% half-width of the paired difference between the two checks (each check carries the paired standard error of grade − verdict), and never by less than 5 points.
  • Drop. The anchor’s read below its best so far by more than the 95% half-width of the difference between the two corrected rates, never by less than 2 points.
A signature must show on two consecutive checks. Together they give a three-state readout, and every state is a hold: A hold means /v1/reward/score answers 409 held, and the response names state, reason, and pinned_step, the step of the best anchor point — the checkpoint to ship. Points with fewer than 20 gradable prompts hold nothing; the response says underpowered instead. A hold is cleared by a person with POST /v1/reward/sessions/{id}/resume; it is never automatic. Our own trainer pauses on a hold, pins the checkpoint and polls for the resume, rather than ending the run. Hold, never kill. A false hold costs an interruption you can reverse; a missed hack costs a shipped regression. Every verdict carries the anchor’s interval and n, so the false-hold rate is a number you can read rather than a promise. On the cheap-grader anchor study (two corrupted GRPO runs and a healthy control, 300 problems every 16 steps) this rule holds the corrupted runs at steps 64 and 96 in grader_fooled, before the drop test alone would (96 and 112), and never holds the control. What the gap can see is bounded by the anchor’s own false passes — the reward’s false passes that the anchor shares are invisible to it — which is why the certificate’s shared-error check belongs next to the anchor.

​
The seam list

Every check also answers with items: each frozen prompt with the reward’s grade, the anchor’s verdict and a kind. counts totals them. The list is not stored server-side; the trainer keeps it. Our trainer appends each check to anchor_seam.jsonl with the prompt and the completion, and the Applied Compute grader exposes seam_items(). On the study’s corrupted runs about half of every GRPO group that carried gradient was a seam group, so this list is where a failure-mining step should start.

​
The anchor report

GET /v1/reward/sessions/{id}/anchor returns the whole picture for a session, uncached: the anchor’s identity and live certificate, the rule in force, every stored check, every hold with its state, reason and pinned_step (the rule replayed over the history), the current hold, and a summary (checks, holds, the widest gap and where). The same document sits inside the signed environment bundle. From Python: python -m errorbar_reward.anchor_report <session_id> [--json] prints the three-line table.

​
Independence, stated precisely

Several tools publish grader agreement numbers, and judge alignment is a documented practice. What a reward session adds is a grader that is independent of the trainer, with intervals rather than point estimates, a refusal floor when underpowered, a certificate that expires and voids on change, and a re-check against an independent judge while the run is in flight.

​
What you take away

A session is not only a reward function. It keeps every item it scored, and hands two things back. The graded trajectories. GET /v1/reward/sessions/{id}/export?format=rl|sft|pairwise returns JSONL: rl is one line per scored item (prompt, completion, reward, verdict, each judge’s grade, flags), the file you replay or re-weight under another algorithm; sft is the passing winners, strict verdict and reward at or above min_grade (default 0.9), in the fine-tuning prompts format; pairwise is chosen/rejected pairs from the same prompt within a step with a reward gap of at least 0.2, ready for DPO. The export scans at most 10,000 rows and says so in X-Errorbar-Export-Capped; narrow with step.
The environment. GET /v1/reward/sessions/{id}/environment returns a signed bundle: the reward spec, each judge’s calibration certificate, the deterministic assertions, the env tools you declared (names only), the tasks you stored on the session, the anchor and whether it is holding, and adapter snippets for TRL and verifiers. It is signed with the same key as eval evidence bundles, so POST /v1/verify checks it. ?format=tasks returns just tasks.jsonl, the shape our RL harness and OpenEnv-style loaders read. Pass tasks when you create the session and the whole environment round-trips.