{
"error": {
"message": "Insufficient balance: deploying this endpoint requires at least 1h of runway. Top up and try again.",
"type": "insufficient_quota",
"code": "insufficient_balance"
}
}
Judge every labeled trace with this criterion and measure agreement: TPR and TNR with 95% Wilson intervals, plus Cohen’s κ — never raw agreement (class imbalance inflates it). Needs at least 30 labels (aim for 100). Billed as judging; the confusion matrix is stored row by row so every disagreement stays auditable.
This number is what turns “the judge said 80%” into “the judge says 80%, agrees with YOU 91% of the time, so the corrected rate is 85% [79–90%]”.
{
"error": {
"message": "Insufficient balance: deploying this endpoint requires at least 1h of runway. Top up and try again.",
"type": "insufficient_quota",
"code": "insufficient_balance"
}
}
Label sets of 50 or fewer are judged synchronously and return the AlignmentReport. Larger sets return AlignmentQueued ({queued: true, total_labels}) and a background worker builds the report over the following minutes; poll GET /v1/criteria/{id} for the metrics and GET /v1/criteria/{id}/alignment for the full report.
Option 1
Option 2
Returned by POST /v1/criteria/{id}/align for label sets of 50 or fewer (larger sets return AlignmentQueued and the report is built over the following minutes).
True when an UNSCOPED criterion was measured across more than one tag — the metrics describe a blend of tasks, not any one of them. Scope the criterion and re-align.
n, tpr, tnr, kappa and the 95% Wilson intervals (tprCi, tnrCi; camelCase on this payload). tpr/tnr are null when the label set has no human fails/passes to measure against.
Set when null TPR/TNR is a one-class measured set (a labeling gap), naming the missing class, so it is never mistaken for "never aligned" or a broken judge.
{"docsConfig":{"theme":"mint","$schema":"https://mintlify.com/docs.json","name":"errorbar","colors":{"primary":"#3f7d39","light":"#3bd495","dark":"#2f5e2a"},"logo":{"light":"https://mintcdn.com/omnia-9828cd09/dR-r1msnM4tGAtp4/logo/light.svg?fit=max\u0026auto=format\u0026n=dR-r1msnM4tGAtp4\u0026q=85\u0026s=0a8e72505cf8b9297d6df70ae03e61f1","dark":"https://mintcdn.com/omnia-9828cd09/dR-r1msnM4tGAtp4/logo/dark.svg?fit=max\u0026auto=format\u0026n=dR-r1msnM4tGAtp4\u0026q=85\u0026s=f68c0d824584773dfd066ad0a450f930"},"favicon":"https://mintcdn.com/omnia-9828cd09/dR-r1msnM4tGAtp4/favicon.svg?fit=max\u0026auto=format\u0026n=dR-r1msnM4tGAtp4\u0026q=85\u0026s=cb75d200c189cf8c0f5c83af1e8bb3b3","navbar":{"links":[{"href":"https://www.errorbar.ai/sign-up","label":"Sign up"}]},"styling":{"latex":true},"navigation":{"pages":[]}},"deployment":{"deploymentMetadataValue":{"subdomain":"omnia-9828cd09","hasAudioNarration":false,"actualSubdomain":"omnia-9828cd09","customDomains":["docs.errorbar.ai"],"gitSource":{"type":"github","owner":"matiyas-h","repo":"omnia-docs","deployBranch":"main","contentDirectory":"","isPrivate":true,"mountPath":"","label":""},"chroma":{"collectionId":"b1fec96f-b9e8-4f8d-8036-90b4f2784b52","liveDeploymentHistoryId":"6aa66951a2a95f72566e8b62","schemaVersion":"v3","latestBuiltDeploymentHistoryId":"6aa66951a2a95f72566e8b62"},"entitlements":{"AI_CHAT":{"status":"DISABLED"}},"buildId":"6aa66951a2a95f72566e8b62","clientVersion":"0.0.3965","searchFilterCounts":[{"language":"default","tag":"API Reference","count":99},{"language":"default","tag":"Documentation","count":53},{"language":"default","tag":"Judges (criteria) API","count":16},{"language":"default","tag":"Evals API","count":13},{"language":"default","tag":"Platform","count":12},{"language":"default","tag":"Run at scale","count":11},{"language":"default","tag":"Certified reward API","count":10},{"language":"default","tag":"Improve","count":9},{"language":"default","tag":"Reinforcement learning API","count":9},{"language":"default","tag":"Fine-tuning API","count":8}]},"assistantConfig":{"enableAskAiButton":false,"enableStarterQuestions":false,"enableContextualStarterQuestions":false,"enableAskAiSearchBar":true},"hasChatPermissions":false,"hasWebMcpPermissions":false,"toggles":[{"name":"kill-isr","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-mintlify-slack","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-v7-file-tree","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-integrations-settings","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-pr-grouping","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-prefix","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"analytics-v7","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"user-flows-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"no-publish-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-onboarding","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"llms-txt-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"pause-universal-search-reconcile","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving-live","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"stripTemperatureMiddleware","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"per-site-search-delta","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"support-platform-pylon-zendesk","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"instant-rollback","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications-server","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-private-page-search","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-fable","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fixed-ai-action-pricing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automation-resolve-merge-conflict","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"scoped-publishing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"mintql","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"page-level-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-version-history-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-nav-generations","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"billing-page-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workspace-projects","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"assistant-v5","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"byok-sep-25","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fork-pr-previews","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"revalidation-jwt","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"vite-origin","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-agent-model-picker","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"billing-updated-9-15","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"hobby-abuse-scan","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"dashboard-inbox","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false}],"isAtlas":false,"googleFontsInlined":false},"customCss":[],"page":{"pagePath":"api-reference/criteria/align-a-criterion-against-your-labels","slug":"/api-reference/criteria/align-a-criterion-against-your-labels","pageMetadata":{"title":"Align a criterion against your labels","description":"Judge every labeled trace with this criterion and measure agreement: TPR and TNR with 95% Wilson intervals, plus Cohen's κ — never raw agreement (class imbalance inflates it). Needs at least 30 labels (aim for 100). Billed as judging; the confusion matrix is stored row by row so every disagreement stays auditable.\n\nThis number is what turns \"the judge said 80%\" into \"the judge says 80%, agrees with YOU 91% of the time, so the corrected rate is 85% [79–90%]\".","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/criteria/{id}/align","href":"/api-reference/criteria/align-a-criterion-against-your-labels","autogeneratedByOpenApi":true},"description":{"compiledSource":"\"use strict\";\nconst {Fragment: _Fragment, jsx: _jsx, jsxs: _jsxs} = arguments[0];\nconst {useMDXComponents: _provideComponents} = arguments[0];\nfunction _createMdxContent(props) {\n const _components = {\n p: \"p\",\n ..._provideComponents(),\n ...props.components\n };\n return _jsxs(_Fragment, {\n children: [_jsx(_components.p, {\n children: \"Judge every labeled trace with this criterion and measure agreement: TPR and TNR with 95% Wilson intervals, plus Cohen’s κ — never raw agreement (class imbalance inflates it). Needs at least 30 labels (aim for 100). Billed as judging; the confusion matrix is stored row by row so every disagreement stays auditable.\"\n }), \"\\n\", _jsx(_components.p, {\n children: \"This number is what turns “the judge said 80%” into “the judge says 80%, agrees with YOU 91% of the time, so the corrected rate is 85% [79–90%]”.\"\n })]\n });\n}\nfunction MDXContent(props = {}) {\n const {wrapper: MDXLayout} = {\n ..._provideComponents(),\n ...props.components\n };\n return MDXLayout ? _jsx(MDXLayout, {\n ...props,\n children: _jsx(_createMdxContent, {\n ...props\n })\n }) : _createMdxContent(props);\n}\nreturn {\n default: MDXContent\n};\n","frontmatter":{},"scope":{}},"pageType":"default","mdx":{"mdxSource":{"compiledSource":"\"use strict\";const{Fragment:_Fragment,jsx:_jsx}=arguments[0];const{useMDXComponents:_provideComponents}=arguments[0];function _createMdxContent(props){return _jsx(_Fragment,{})}function MDXContent(props={}){const{wrapper:MDXLayout}={..._provideComponents(),...props.components};return MDXLayout?_jsx(MDXLayout,{...props,children:_jsx(_createMdxContent,{...props})}):_createMdxContent(props)}return{default:MDXContent}","frontmatter":{},"scope":{"config":{},"pageMetadata":{"title":"Align a criterion against your labels","description":"Judge every labeled trace with this criterion and measure agreement: TPR and TNR with 95% Wilson intervals, plus Cohen's κ — never raw agreement (class imbalance inflates it). Needs at least 30 labels (aim for 100). Billed as judging; the confusion matrix is stored row by row so every disagreement stays auditable.\n\nThis number is what turns \"the judge said 80%\" into \"the judge says 80%, agrees with YOU 91% of the time, so the corrected rate is 85% [79–90%]\".","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/criteria/{id}/align","href":"/api-reference/criteria/align-a-criterion-against-your-labels","autogeneratedByOpenApi":true}}},"mdxSourceWithNoJs":null},"mdxExtracts":{"tableOfContents":[],"codeExamples":{}},"jsLinks":[],"openApiReferenceData":{"operation":{"tags":["Criteria"],"summary":"Align a criterion against your labels","operationId":"alignCriterion","description":"Judge every labeled trace with this criterion and measure agreement: TPR and TNR with 95% Wilson intervals, plus Cohen's κ — never raw agreement (class imbalance inflates it). Needs at least 30 labels (aim for 100). Billed as judging; the confusion matrix is stored row by row so every disagreement stays auditable.\n\nThis number is what turns \"the judge said 80%\" into \"the judge says 80%, agrees with YOU 91% of the time, so the corrected rate is 85% [79–90%]\".","parameters":["a0887e5c-717e-40df-895b-f8a523017379"],"responses":{"200":"95829e26-7d90-4637-97a5-36356e34366e","400":"6a257b9a-e1f0-4a51-99d1-0d0adb8b6713","401":"27c8f8a4-0a80-4777-8f86-fce581b7d392","402":"48398537-c9a8-4b8a-a41e-a5c584db237b","403":"66b96f5d-6dff-4920-b9d9-0d49a2b94ae2","404":"b28e0568-f587-47cc-b687-f4fc161f5da1"},"method":"post","type":"path","servers":["7225afda-fbb9-43d7-bb56-160196b3872f","a0fa1f4b-c484-4da1-9eb0-0d3cf3ec20d1"],"security":["d7249d09-10f7-465d-b688-3db83ef725e0"],"path":"/v1/criteria/{id}/align","baseUrlOptions":["https://gateway.errorbar.ai","https://www.errorbar.ai/api"],"title":"Align a criterion against your labels","codeSamples":[],"requiredOnlyExamples":false,"requestExampleType":"generated","responseExampleType":"generated"},"dependencies":{"parameters":{"a0887e5c-717e-40df-895b-f8a523017379":{"name":"id","in":"path","required":true,"schema":{"type":"string","isRequired":true,"uniqueKey":"","typeLabel":"string"}}},"responses":{"95829e26-7d90-4637-97a5-36356e34366e":{"description":"Label sets of 50 or fewer are judged synchronously and return the AlignmentReport. Larger sets return AlignmentQueued ({queued: true, total_labels}) and a background worker builds the report over the following minutes; poll GET /v1/criteria/{id} for the metrics and GET /v1/criteria/{id}/alignment for the full report.","content":{"application/json":{"schema":{"oneOf":[{"type":"object","description":"Returned by POST /v1/criteria/{id}/align for label sets of 50 or fewer (larger sets return AlignmentQueued and the report is built over the following minutes).","properties":{"scope_tag":{"type":"string","description":"The population measured — the criterion's tag ('' = all labeled traffic). Alignment numbers only describe the population they were measured on.","uniqueKey":"oneOf.0.scope_tag","typeLabel":"string"},"scope_family":{"type":"string","description":"The auto-detected traffic segment measured (\"\" = no segment scope). The second population handle, same binding rule as scope_tag.","uniqueKey":"oneOf.0.scope_family","typeLabel":"string"},"tag_breakdown":{"type":"array","description":"What the measured labels were made of, by tag ('' = untagged). One entry for a scoped criterion; the pollution report for an unscoped one.","items":{"type":"object","properties":{"tag":{"type":"string","uniqueKey":"oneOf.0.tag_breakdown.items.[INDEX].tag","typeLabel":"string"},"n":{"type":"integer","uniqueKey":"oneOf.0.tag_breakdown.items.[INDEX].n","typeLabel":"integer"}},"uniqueKey":"oneOf.0.tag_breakdown.items.[INDEX]","typeLabel":"object"},"uniqueKey":"oneOf.0.tag_breakdown","typeLabel":"object[]"},"mixed_population":{"type":"boolean","description":"True when an UNSCOPED criterion was measured across more than one tag — the metrics describe a blend of tasks, not any one of them. Scope the criterion and re-align.","uniqueKey":"oneOf.0.mixed_population","typeLabel":"boolean"},"excluded_other_cause":{"type":"integer","description":"Fails attributed to other criteria: in scope but excluded from this criterion's calibration set by cause attribution.","uniqueKey":"oneOf.0.excluded_other_cause","typeLabel":"integer"},"unattributed_fails":{"type":"integer","description":"In-population fails with no cause attribution. They still count as fails for every criterion; attributing them tightens narrow judges' numbers.","uniqueKey":"oneOf.0.unattributed_fails","typeLabel":"integer"},"metrics":{"type":"object","description":"n, tpr, tnr, kappa and the 95% Wilson intervals (tprCi, tnrCi; camelCase on this payload). tpr/tnr are null when the label set has no human fails/passes to measure against.","uniqueKey":"oneOf.0.metrics","typeLabel":"object"},"tier":{"type":"string","enum":["aligned","weak","misaligned","unmeasured"],"uniqueKey":"oneOf.0.tier","typeLabel":"enum\u003cstring>"},"thin_alignment_set":{"type":"boolean","description":"True under 100 judgeable labels: intervals stay wide; label more.","uniqueKey":"oneOf.0.thin_alignment_set","typeLabel":"boolean"},"skipped":{"type":"integer","description":"Labeled traces whose content aged out, or whose judge verdict was unparseable twice.","uniqueKey":"oneOf.0.skipped","typeLabel":"integer"},"holdout":{"oneOf":[{"type":"object","description":"Present at 80+ judged labels: {tune_n, report_n}. Metrics above come from the report half only; tune-half rows exist for prompt iteration.","uniqueKey":"oneOf.0.holdout.oneOf.0","typeLabel":"object"},{"type":"null","description":"Present at 80+ judged labels: {tune_n, report_n}. Metrics above come from the report half only; tune-half rows exist for prompt iteration.","uniqueKey":"oneOf.0.holdout.oneOf.1","typeLabel":"null"}],"uniqueKey":"oneOf.0.holdout","typeLabel":"object | null"},"one_class_note":{"oneOf":[{"type":"string","description":"Set when null TPR/TNR is a one-class measured set (a labeling gap), naming the missing class, so it is never mistaken for \"never aligned\" or a broken judge.","uniqueKey":"oneOf.0.one_class_note.oneOf.0","typeLabel":"string"},{"type":"null","description":"Set when null TPR/TNR is a one-class measured set (a labeling gap), naming the missing class, so it is never mistaken for \"never aligned\" or a broken judge.","uniqueKey":"oneOf.0.one_class_note.oneOf.1","typeLabel":"null"}],"uniqueKey":"oneOf.0.one_class_note","typeLabel":"string | null"},"disagreements":{"type":"array","items":{"type":"object","properties":{"request_id":{"type":"string","uniqueKey":"oneOf.0.disagreements.items.[INDEX].request_id","typeLabel":"string"},"judge_verdict":{"type":"string","uniqueKey":"oneOf.0.disagreements.items.[INDEX].judge_verdict","typeLabel":"string"},"human_verdict":{"type":"string","uniqueKey":"oneOf.0.disagreements.items.[INDEX].human_verdict","typeLabel":"string"}},"uniqueKey":"oneOf.0.disagreements.items.[INDEX]","typeLabel":"object"},"uniqueKey":"oneOf.0.disagreements","typeLabel":"object[]"}},"uniqueKey":"oneOf.0","typeLabel":"object"},{"type":"object","description":"Returned instead of a report when the label set is large: the run advances in background slices; read the finished report via GET /v1/criteria/{id}/alignment.","properties":{"queued":{"type":"boolean","uniqueKey":"oneOf.1.queued","typeLabel":"boolean"},"total_labels":{"type":"integer","uniqueKey":"oneOf.1.total_labels","typeLabel":"integer"}},"uniqueKey":"oneOf.1","typeLabel":"object"}],"uniqueKey":"","typeLabel":"object"},"examples":{"Example":{"title":"Example","value":{"scope_tag":"\u003cstring>","scope_family":"\u003cstring>","tag_breakdown":[{"tag":"\u003cstring>","n":123}],"mixed_population":true,"excluded_other_cause":123,"unattributed_fails":123,"metrics":{},"tier":"aligned","thin_alignment_set":true,"skipped":123,"holdout":{},"one_class_note":"\u003cstring>","disagreements":[{"request_id":"\u003cstring>","judge_verdict":"\u003cstring>","human_verdict":"\u003cstring>"}]}}},"exampleType":"generated"}},"exampleType":"generated"},"6a257b9a-e1f0-4a51-99d1-0d0adb8b6713":{"description":"Fewer than 30 gradeable labels in this criterion's population (the error names the population and your current count), or another invalid state.","content":{"application/json":{"schema":{"type":"object","properties":{"error":{"type":"object","properties":{"message":{"type":"string","isRequired":true,"uniqueKey":"error.message","typeLabel":"string"},"type":{"type":"string","description":"invalid_request_error, insufficient_quota, rate_limit_error, or api_error.","isRequired":true,"uniqueKey":"error.type","typeLabel":"string"},"code":{"type":"string","description":"Machine-stable cause, e.g. invalid_api_key, not_found, insufficient_permissions, precondition_failed.","isRequired":true,"uniqueKey":"error.code","typeLabel":"string"}},"required":["message","type","code"],"uniqueKey":"error","typeLabel":"object"}},"description":"Every refusal — gateway and management API alike — uses this one envelope.","uniqueKey":"","typeLabel":"object"},"examples":{"Example":{"title":"Example","value":{"error":{"message":"\u003cstring>","type":"\u003cstring>","code":"\u003cstring>"}}}},"exampleType":"generated"}},"exampleType":"generated"},"27c8f8a4-0a80-4777-8f86-fce581b7d392":{"description":"Missing, malformed, or revoked API key.","content":{"application/json":{"schema":{"type":"object","properties":{"error":{"type":"object","properties":{"message":{"type":"string","isRequired":true,"uniqueKey":"error.message","typeLabel":"string"},"type":{"type":"string","description":"invalid_request_error, insufficient_quota, rate_limit_error, or api_error.","isRequired":true,"uniqueKey":"error.type","typeLabel":"string"},"code":{"type":"string","description":"Machine-stable cause, e.g. invalid_api_key, not_found, insufficient_permissions, precondition_failed.","isRequired":true,"uniqueKey":"error.code","typeLabel":"string"}},"required":["message","type","code"],"uniqueKey":"error","typeLabel":"object"}},"description":"Every refusal — gateway and management API alike — uses this one envelope.","uniqueKey":"","typeLabel":"object"},"example":{"error":{"message":"Invalid API key","type":"invalid_request_error","code":"invalid_api_key"}},"examples":{"Example":{"title":"Example","value":{"error":{"message":"Invalid API key","type":"invalid_request_error","code":"invalid_api_key"}}}},"exampleType":"examples"}},"exampleType":"examples"},"48398537-c9a8-4b8a-a41e-a5c584db237b":{"description":"Your wallet can't cover the required prepaid runway.","content":{"application/json":{"schema":{"type":"object","properties":{"error":{"type":"object","properties":{"message":{"type":"string","isRequired":true,"uniqueKey":"error.message","typeLabel":"string"},"type":{"type":"string","description":"invalid_request_error, insufficient_quota, rate_limit_error, or api_error.","isRequired":true,"uniqueKey":"error.type","typeLabel":"string"},"code":{"type":"string","description":"Machine-stable cause, e.g. invalid_api_key, not_found, insufficient_permissions, precondition_failed.","isRequired":true,"uniqueKey":"error.code","typeLabel":"string"}},"required":["message","type","code"],"uniqueKey":"error","typeLabel":"object"}},"description":"Every refusal — gateway and management API alike — uses this one envelope.","uniqueKey":"","typeLabel":"object"},"example":{"error":{"message":"Insufficient balance: deploying this endpoint requires at least 1h of runway. Top up and try again.","type":"insufficient_quota","code":"insufficient_balance"}},"examples":{"Example":{"title":"Example","value":{"error":{"message":"Insufficient balance: deploying this endpoint requires at least 1h of runway. Top up and try again.","type":"insufficient_quota","code":"insufficient_balance"}}}},"exampleType":"examples"}},"exampleType":"examples"},"66b96f5d-6dff-4920-b9d9-0d49a2b94ae2":{"description":"The key lacks the required owner/admin permission.","content":{"application/json":{"schema":{"type":"object","properties":{"error":{"type":"object","properties":{"message":{"type":"string","isRequired":true,"uniqueKey":"error.message","typeLabel":"string"},"type":{"type":"string","description":"invalid_request_error, insufficient_quota, rate_limit_error, or api_error.","isRequired":true,"uniqueKey":"error.type","typeLabel":"string"},"code":{"type":"string","description":"Machine-stable cause, e.g. invalid_api_key, not_found, insufficient_permissions, precondition_failed.","isRequired":true,"uniqueKey":"error.code","typeLabel":"string"}},"required":["message","type","code"],"uniqueKey":"error","typeLabel":"object"}},"description":"Every refusal — gateway and management API alike — uses this one envelope.","uniqueKey":"","typeLabel":"object"},"examples":{"Example":{"title":"Example","value":{"error":{"message":"\u003cstring>","type":"\u003cstring>","code":"\u003cstring>"}}}},"exampleType":"generated"}},"exampleType":"generated"},"b28e0568-f587-47cc-b687-f4fc161f5da1":{"description":"The resource does not exist in your workspace.","content":{"application/json":{"schema":{"type":"object","properties":{"error":{"type":"object","properties":{"message":{"type":"string","isRequired":true,"uniqueKey":"error.message","typeLabel":"string"},"type":{"type":"string","description":"invalid_request_error, insufficient_quota, rate_limit_error, or api_error.","isRequired":true,"uniqueKey":"error.type","typeLabel":"string"},"code":{"type":"string","description":"Machine-stable cause, e.g. invalid_api_key, not_found, insufficient_permissions, precondition_failed.","isRequired":true,"uniqueKey":"error.code","typeLabel":"string"}},"required":["message","type","code"],"uniqueKey":"error","typeLabel":"object"}},"description":"Every refusal — gateway and management API alike — uses this one envelope.","uniqueKey":"","typeLabel":"object"},"examples":{"Example":{"title":"Example","value":{"error":{"message":"\u003cstring>","type":"\u003cstring>","code":"\u003cstring>"}}}},"exampleType":"generated"}},"exampleType":"generated"}},"security":{"d7249d09-10f7-465d-b688-3db83ef725e0":{"bearerAuth":{"type":"http","scheme":"bearer","description":"Your workspace API key, e.g. `sk_sovereign_...`, sent as `Authorization: Bearer \u003ckey>`.","scopes":[]}}},"servers":{"7225afda-fbb9-43d7-bb56-160196b3872f":{"url":"https://gateway.errorbar.ai","description":"Production"},"a0fa1f4b-c484-4da1-9eb0-0d3cf3ec20d1":{"url":"https://www.errorbar.ai/api","description":"Control plane (also served at this base URL)"}},"processedSecurityOptions":[{"schemes":[{"scheme":{"type":"http","scheme":"bearer","description":"Your workspace API key, e.g. `sk_sovereign_...`, sent as `Authorization: Bearer \u003ckey>`.","scopes":[]},"schema":{"title":"Bearer Auth","type":"string","isRequired":true,"description":"Your workspace API key, e.g. `sk_sovereign_...`, sent as `Authorization: Bearer \u003ckey>`.","format":"bearer","typeLabel":"string\u003cbearer>","deprecated":false},"schemeName":"bearerAuth","name":"Authorization"}],"title":"bearerAuth"}]},"originalFileLocation":"openapi.json"},"subdomain":"omnia-9828cd09","siteName":"errorbar","title":"Align a criterion against your labels - errorbar","navMaps":{"firstHrefInVersion":{},"firstHrefInLanguage":{}},"lastModified":"2026-09-13T09:14:06.786Z","originalFileLocation":"api-reference/criteria/align-a-criterion-against-your-labels.mdx"},"headHtml":"\u003cscript data-mint-head>(function(){var d=document.documentElement,p=null;if(!p){try{p=localStorage.getItem(\"isDarkMode\")}catch(e){}if(p==='true')p='dark';else if(p==='false')p='light';if(p!=='light'\u0026\u0026p!=='dark'\u0026\u0026p!=='system')p=\"system\"}var r=p==='system'?(matchMedia(\"(prefers-color-scheme: dark)\").matches?'dark':'light'):p;d.classList.remove(r==='dark'?'light':'dark');d.classList.add(r);d.setAttribute(\"data-theme-preference\",p)})();\u003c/script>\u003cmeta data-mint-head name=\"description\" content=\"Judge every labeled trace with this criterion and measure agreement: TPR and TNR with 95% Wilson intervals, plus Cohen\u0026#x27;s κ — never raw agreement (class imbalance inflates it). Needs at least 30 labels (aim for 100). Billed as judging; the confusion matrix is stored row by row so every disagreement stays auditable. This number is what turns \u0026quot;the judge said 80%\u0026quot; into \u0026quot;the judge says 80%, agrees with YOU 91% of the time, so the corrected rate is 85% [79–90%]\u0026quot;.\">\u003cmeta data-mint-head name=\"application-name\" content=\"errorbar\">\u003cmeta data-mint-head name=\"generator\" content=\"Mintlify\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/omnia-9828cd09/ubKwIdVYRFtlXDaF/_generated/favicon/android-chrome-192x192.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"180x180\" href=\"/mintlify-assets/_mintlify/favicons/omnia-9828cd09/ubKwIdVYRFtlXDaF/_generated/favicon/apple-touch-icon.png\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/omnia-9828cd09/ubKwIdVYRFtlXDaF/_generated/favicon/favicon-16x16.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/omnia-9828cd09/ubKwIdVYRFtlXDaF/_generated/favicon/favicon-32x32.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/omnia-9828cd09/ubKwIdVYRFtlXDaF/_generated/favicon/favicon.ico\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/omnia-9828cd09/ubKwIdVYRFtlXDaF/_generated/favicon-dark/android-chrome-192x192.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/omnia-9828cd09/ubKwIdVYRFtlXDaF/_generated/favicon-dark/favicon-16x16.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/omnia-9828cd09/ubKwIdVYRFtlXDaF/_generated/favicon-dark/favicon-32x32.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/omnia-9828cd09/ubKwIdVYRFtlXDaF/_generated/favicon-dark/favicon.ico\" media=\"(prefers-color-scheme: dark)\">\u003cmeta data-mint-head property=\"og:title\" content=\"Align a criterion against your labels - errorbar\">\u003cmeta data-mint-head property=\"og:description\" content=\"Judge every labeled trace with this criterion and measure agreement: TPR and TNR with 95% Wilson intervals, plus Cohen\u0026#x27;s κ — never raw agreement (class imbalance inflates it). Needs at least 30 labels (aim for 100). Billed as judging; the confusion matrix is stored row by row so every disagreement stays auditable. This number is what turns \u0026quot;the judge said 80%\u0026quot; into \u0026quot;the judge says 80%, agrees with YOU 91% of the time, so the corrected rate is 85% [79–90%]\u0026quot;.\">\u003cmeta data-mint-head property=\"og:site_name\" content=\"errorbar\">\u003cmeta data-mint-head property=\"og:url\" content=\"https://docs.errorbar.ai/api-reference/criteria/align-a-criterion-against-your-labels\">\u003cmeta data-mint-head property=\"og:type\" content=\"website\">\u003cmeta data-mint-head property=\"og:image\" content=\"https://omnia-9828cd09.mintlify.app/_mintlify/api/og/v2?division=Judges+%28criteria%29+API\u0026amp;title=Align+a+criterion+against+your+labels\u0026amp;description=Judge+every+labeled+trace+with+this+criterion+and+measure+agreement%3A+TPR+and+TNR+with+95%25+Wilson+intervals%2C+plus+Cohen%27s+%CE%BA+%E2%80%94+never+raw+agreement+%28class+imbalanc\u0026amp;theme=4890a253571116e8d6e1ff75\u0026amp;signature=c0eb707d57858625ee5f2968c30f92b27c63a79b795a1f505f375e3b2ce371b9\">\u003cmeta data-mint-head property=\"og:image:width\" content=\"1200\">\u003cmeta data-mint-head property=\"og:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"twitter:card\" content=\"summary_large_image\">\u003cmeta data-mint-head name=\"twitter:title\" content=\"Align a criterion against your labels - errorbar\">\u003cmeta data-mint-head name=\"twitter:description\" content=\"Judge every labeled trace with this criterion and measure agreement: TPR and TNR with 95% Wilson intervals, plus Cohen\u0026#x27;s κ — never raw agreement (class imbalance inflates it). Needs at least 30 labels (aim for 100). Billed as judging; the confusion matrix is stored row by row so every disagreement stays auditable. This number is what turns \u0026quot;the judge said 80%\u0026quot; into \u0026quot;the judge says 80%, agrees with YOU 91% of the time, so the corrected rate is 85% [79–90%]\u0026quot;.\">\u003cmeta data-mint-head name=\"twitter:image\" content=\"https://omnia-9828cd09.mintlify.app/_mintlify/api/og/v2?division=Judges+%28criteria%29+API\u0026amp;title=Align+a+criterion+against+your+labels\u0026amp;description=Judge+every+labeled+trace+with+this+criterion+and+measure+agreement%3A+TPR+and+TNR+with+95%25+Wilson+intervals%2C+plus+Cohen%27s+%CE%BA+%E2%80%94+never+raw+agreement+%28class+imbalanc\u0026amp;theme=4890a253571116e8d6e1ff75\u0026amp;signature=c0eb707d57858625ee5f2968c30f92b27c63a79b795a1f505f375e3b2ce371b9\">\u003cmeta data-mint-head name=\"twitter:image:width\" content=\"1200\">\u003cmeta data-mint-head name=\"twitter:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"msapplication-config\" content=\"/mintlify-assets/_mintlify/favicons/omnia-9828cd09/ubKwIdVYRFtlXDaF/_generated/favicon/browserconfig.xml\">\u003cmeta data-mint-head name=\"apple-mobile-web-app-title\" content=\"errorbar\">\u003cmeta data-mint-head name=\"msapplication-TileColor\" content=\"#3f7d39\">\u003clink data-mint-head rel=\"canonical\" href=\"https://docs.errorbar.ai/api-reference/criteria/align-a-criterion-against-your-labels\">\u003clink data-mint-head rel=\"alternate\" type=\"application/xml\" href=\"/sitemap.xml\">\u003clink data-mint-head rel=\"alternate\" type=\"text/markdown\" href=\"/api-reference/criteria/align-a-criterion-against-your-labels.md\">\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"https://docs.errorbar.ai/#organization\",\"name\":\"errorbar\",\"url\":\"https://docs.errorbar.ai\",\"logo\":{\"@type\":\"ImageObject\",\"url\":\"https://mintcdn.com/omnia-9828cd09/dR-r1msnM4tGAtp4/logo/light.svg?fit=max\u0026auto=format\u0026n=dR-r1msnM4tGAtp4\u0026q=85\u0026s=0a8e72505cf8b9297d6df70ae03e61f1\"}},{\"@type\":\"WebSite\",\"@id\":\"https://docs.errorbar.ai#website\",\"name\":\"errorbar\",\"url\":\"https://docs.errorbar.ai\",\"publisher\":{\"@id\":\"https://docs.errorbar.ai/#organization\"}},{\"@type\":\"WebPage\",\"@id\":\"https://docs.errorbar.ai/api-reference/criteria/align-a-criterion-against-your-labels#webpage\",\"url\":\"https://docs.errorbar.ai/api-reference/criteria/align-a-criterion-against-your-labels\",\"name\":\"Align a criterion against your labels\",\"description\":\"Judge every labeled trace with this criterion and measure agreement: TPR and TNR with 95% Wilson intervals, plus Cohen's κ — never raw agreement (class imbalance inflates it). Needs at least 30 labels (aim for 100). Billed as judging; the confusion matrix is stored row by row so every disagreement stays auditable. This number is what turns \\\"the judge said 80%\\\" into \\\"the judge says 80%, agrees with YOU 91% of the time, so the corrected rate is 85% [79–90%]\\\".\",\"dateModified\":\"2026-09-13T09:14:06.786Z\",\"isPartOf\":{\"@id\":\"https://docs.errorbar.ai#website\"},\"breadcrumb\":{\"@id\":\"https://docs.errorbar.ai/api-reference/criteria/align-a-criterion-against-your-labels#breadcrumb\"}},{\"@type\":\"BreadcrumbList\",\"@id\":\"https://docs.errorbar.ai/api-reference/criteria/align-a-criterion-against-your-labels#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Judges (criteria) API\",\"item\":\"https://docs.errorbar.ai/api-reference/criteria/list-criteria\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Align a criterion against your labels\",\"item\":\"https://docs.errorbar.ai/api-reference/criteria/align-a-criterion-against-your-labels\"}]},{\"@type\":[\"TechArticle\",\"APIReference\"],\"@id\":\"https://docs.errorbar.ai/api-reference/criteria/align-a-criterion-against-your-labels#apireference\",\"headline\":\"Align a criterion against your labels\",\"name\":\"Align a criterion against your labels\",\"description\":\"Judge every labeled trace with this criterion and measure agreement: TPR and TNR with 95% Wilson intervals, plus Cohen's κ — never raw agreement (class imbalance inflates it). Needs at least 30 labels (aim for 100). Billed as judging; the confusion matrix is stored row by row so every disagreement stays auditable. This number is what turns \\\"the judge said 80%\\\" into \\\"the judge says 80%, agrees with YOU 91% of the time, so the corrected rate is 85% [79–90%]\\\".\",\"url\":\"https://docs.errorbar.ai/api-reference/criteria/align-a-criterion-against-your-labels\",\"mainEntityOfPage\":{\"@id\":\"https://docs.errorbar.ai/api-reference/criteria/align-a-criterion-against-your-labels#webpage\"},\"dateModified\":\"2026-09-13T09:14:06.786Z\",\"publisher\":{\"@id\":\"https://docs.errorbar.ai/#organization\"},\"isPartOf\":{\"@id\":\"https://docs.errorbar.ai#website\"}}]}\u003c/script>\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@type\":\"WebSite\",\"name\":\"errorbar\",\"creator\":{\"@type\":\"Organization\",\"name\":\"Mintlify\",\"url\":\"https://mintlify.com\"}}\u003c/script>","siteName":"errorbar","nav":{"global":null,"tabs":[{"tab":"Documentation","pages":[{"title":"Documentation","href":"/introduction"}]},{"tab":"SDKs","pages":[{"title":"SDKs","href":"/sdks/overview"}]},{"tab":"API Reference","groups":[{"group":"Overview","pages":[{"title":"API overview","href":"/api-reference/introduction"}]},{"group":"Inference API","pages":[{"title":"Create chat completion","deprecated":null,"version":null,"openapi":"/openapi-inference.json post /chat/completions","href":"/api-reference/chat/create-chat-completion"},{"title":"Create embeddings","deprecated":null,"version":null,"openapi":"/openapi-inference.json post /embeddings","href":"/api-reference/embeddings/create-embeddings"},{"title":"Rerank documents","deprecated":null,"version":null,"openapi":"/openapi-inference.json post /rerank","href":"/api-reference/rerank/rerank-documents"},{"title":"Create a response","deprecated":null,"version":null,"openapi":"/openapi-inference.json post /responses","href":"/api-reference/responses/create-a-response"},{"title":"List models","deprecated":null,"version":null,"openapi":"/openapi-inference.json get /models","href":"/api-reference/models/list-models"}]},{"group":"Capture \u0026 setup API","pages":[{"title":"OTLP trace ingest","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/traces","href":"/api-reference/otlp-trace-ingest"},{"title":"Setup status","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/setup/status","href":"/api-reference/setup-status"}]},{"group":"Logs \u0026 datasets API","pages":[{"title":"List logged exchanges","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/logs","href":"/api-reference/logs/list-logged-exchanges"},{"title":"Export logged exchanges as JSONL","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/logs/export","href":"/api-reference/logs/export-logged-exchanges-as-jsonl"},{"title":"Import graded traffic","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/logs/import","href":"/api-reference/logs/import-graded-traffic"},{"title":"Get an agent run / grouped conversation","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/traces/{trace_id}","href":"/api-reference/get-an-agent-run-grouped-conversation"},{"title":"Create a dataset from logs","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/datasets/from_logs","href":"/api-reference/logs/create-a-dataset-from-logs"},{"title":"Check texts for benchmark contamination","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/datasets/decontaminate","href":"/api-reference/logs/check-texts-for-benchmark-contamination"}]},{"group":"Grades (labels) API","pages":[{"title":"List trace labels","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/labels","href":"/api-reference/labels/list-trace-labels"},{"title":"Label a trace","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/labels","href":"/api-reference/labels/label-a-trace"},{"title":"List label sets","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/label_sets","href":"/api-reference/labels/list-label-sets"},{"title":"Create a label set","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/label_sets","href":"/api-reference/labels/create-a-label-set"},{"title":"Attach a label set","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/label_sets/{id}/attach","href":"/api-reference/labels/attach-a-label-set"},{"title":"Freeze a label set","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/label_sets/{id}/freeze","href":"/api-reference/labels/freeze-a-label-set"}]},{"group":"Judges (criteria) API","pages":[{"title":"List criteria","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/criteria","href":"/api-reference/criteria/list-criteria"},{"title":"Create a criterion","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/criteria","href":"/api-reference/criteria/create-a-criterion"},{"title":"List criterion templates","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/criteria/templates","href":"/api-reference/list-criterion-templates"},{"title":"Get a criterion","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/criteria/{id}","href":"/api-reference/get-a-criterion"},{"title":"Update a criterion","deprecated":null,"version":null,"openapi":"/openapi.json patch /v1/criteria/{id}","href":"/api-reference/update-a-criterion"},{"title":"Delete a criterion","description":null,"deprecated":null,"version":null,"openapi":"/openapi.json delete /v1/criteria/{id}","href":"/api-reference/delete-a-criterion"},{"title":"Align a criterion against your labels","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/criteria/{id}/align","href":"/api-reference/criteria/align-a-criterion-against-your-labels"},{"title":"Get the persistent alignment report","description":"Reconstructed from the stored confusion rows: metrics with CIs, population breakdown, and every judge/human disagreement with the human critique and response excerpt. Free — reads the last run; POST /align re-judges and spends.","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/criteria/{id}/alignment","href":"/api-reference/get-the-persistent-alignment-report"},{"title":"Scan unlabeled traffic for suspected failures","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/criteria/{id}/scan","href":"/api-reference/criteria/scan-unlabeled-traffic-for-suspected-failures"},{"title":"Suggest judges from your failure critiques","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/criteria/suggest","href":"/api-reference/criteria/suggest-judges-from-your-failure-critiques"},{"title":"Run one auto-improve iteration on a judge","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/criteria/{id}/auto_improve","href":"/api-reference/criteria/run-one-auto-improve-iteration-on-a-judge"},{"title":"Get a calibration certificate","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/criteria/{id}/certificate","href":"/api-reference/criteria/get-a-calibration-certificate"},{"title":"The certificate before and after adjudication","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/criteria/{id}/adjudication","href":"/api-reference/judges/the-certificate-before-and-after-adjudication"},{"title":"Adjudicate a judge/human disagreement","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/labels/{id}/adjudicate","href":"/api-reference/judges/adjudicate-a-judgehuman-disagreement"},{"title":"Get judge settings","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/settings/judge","href":"/api-reference/criteria/get-judge-settings"},{"title":"Update judge settings","deprecated":null,"version":null,"openapi":"/openapi.json put /v1/settings/judge","href":"/api-reference/criteria/update-judge-settings"}]},{"group":"Evals API","pages":[{"title":"List eval runs","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/evals","href":"/api-reference/evals/list-eval-runs"},{"title":"Queue an eval run","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/evals","href":"/api-reference/evals/queue-an-eval-run"},{"title":"Get an eval run","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/evals/{id}","href":"/api-reference/evals/get-an-eval-run"},{"title":"Delete an eval run","deprecated":null,"version":null,"openapi":"/openapi.json delete /v1/evals/{id}","href":"/api-reference/evals/delete-an-eval-run"},{"title":"Cancel an eval run","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/evals/{id}/cancel","href":"/api-reference/evals/cancel-an-eval-run"},{"title":"Get an eval run's test cases","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/evals/{id}/samples","href":"/api-reference/evals/get-an-eval-runs-test-cases"},{"title":"Gate a deploy on an eval run","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/evals/{id}/gate","href":"/api-reference/evals/gate-a-deploy-on-an-eval-run"},{"title":"Compare two eval runs","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/evals/compare","href":"/api-reference/evals/compare-two-eval-runs"},{"title":"List failure clusters","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/evals/failure_clusters","href":"/api-reference/evals/list-failure-clusters"},{"title":"Get an evidence bundle","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/evals/{id}/evidence","href":"/api-reference/evals/get-an-evidence-bundle"},{"title":"List registered endpoints","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/endpoints","href":"/api-reference/evals/list-registered-endpoints"},{"title":"Register an endpoint","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/endpoints","href":"/api-reference/evals/register-an-endpoint"},{"title":"Remove an endpoint","deprecated":null,"version":null,"openapi":"/openapi.json delete /v1/endpoints/{id}","href":"/api-reference/evals/remove-an-endpoint"}]},{"group":"Alerts API","pages":[{"title":"List fired alerts","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/alerts","href":"/api-reference/alerts/list-fired-alerts"}]},{"group":"Fine-tuning API","pages":[{"title":"List training files","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/fine_tuning/files","href":"/api-reference/fine-tuning/list-training-files"},{"title":"Upload training file","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/fine_tuning/files","href":"/api-reference/fine-tuning/upload-training-file"},{"title":"List fine-tuning jobs","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/fine_tuning/jobs","href":"/api-reference/fine-tuning/list-fine-tuning-jobs"},{"title":"Create fine-tuning job","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/fine_tuning/jobs","href":"/api-reference/fine-tuning/create-fine-tuning-job"},{"title":"Get fine-tuning job","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/fine_tuning/jobs/{id}","href":"/api-reference/fine-tuning/get-fine-tuning-job"},{"title":"Cancel fine-tuning job","deprecated":null,"version":null,"openapi":"/openapi.json delete /v1/fine_tuning/jobs/{id}","href":"/api-reference/fine-tuning/cancel-fine-tuning-job"},{"title":"Start a bake-off","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/fine_tuning/jobs/{id}/bakeoff","href":"/api-reference/fine-tuning/start-a-bake-off"},{"title":"Get bake-off state","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/fine_tuning/jobs/{id}/bakeoff","href":"/api-reference/fine-tuning/get-bake-off-state"}]},{"group":"Certified reward API","pages":[{"title":"Create a reward session","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/reward/sessions","href":"/api-reference/certified-reward/create-a-reward-session"},{"title":"List reward sessions","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/reward/sessions","href":"/api-reference/certified-reward/list-reward-sessions"},{"title":"Get a reward session","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/reward/sessions/{id}","href":"/api-reference/certified-reward/get-a-reward-session"},{"title":"Stop a reward session","deprecated":null,"version":null,"openapi":"/openapi.json delete /v1/reward/sessions/{id}","href":"/api-reference/certified-reward/stop-a-reward-session"},{"title":"Score a rollout group","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/reward/score","href":"/api-reference/certified-reward/score-a-rollout-group"},{"title":"Mid-run anchor check","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/reward/sessions/{id}/anchor","href":"/api-reference/certified-reward/mid-run-anchor-check"},{"title":"The anchor report","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/reward/sessions/{id}/anchor","href":"/api-reference/certified-reward/the-anchor-report"},{"title":"Resume a held session","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/reward/sessions/{id}/resume","href":"/api-reference/certified-reward/resume-a-held-session"},{"title":"Export the graded trajectories","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/reward/sessions/{id}/export","href":"/api-reference/certified-reward/export-the-graded-trajectories"},{"title":"Export the environment bundle","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/reward/sessions/{id}/environment","href":"/api-reference/certified-reward/export-the-environment-bundle"}]},{"group":"Reinforcement learning API","pages":[{"title":"List GRPO runs","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/grpo/runs","href":"/api-reference/reinforcement-learning/list-grpo-runs"},{"title":"Start a GRPO run","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/grpo/runs","href":"/api-reference/reinforcement-learning/start-a-grpo-run"},{"title":"Get a GRPO run","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/grpo/runs/{id}","href":"/api-reference/reinforcement-learning/get-a-grpo-run"},{"title":"Stop a GRPO run","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/grpo/runs/{id}/stop","href":"/api-reference/reinforcement-learning/stop-a-grpo-run"},{"title":"Download trained adapter weights","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/grpo/runs/{id}/weights","href":"/api-reference/reinforcement-learning/download-trained-adapter-weights"},{"title":"List RAFT rounds","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/raft/rounds","href":"/api-reference/reinforcement-learning/list-raft-rounds"},{"title":"List declared agent tools","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/env/tools","href":"/api-reference/reinforcement-learning/list-declared-agent-tools"},{"title":"Register or update a tool","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/env/tools","href":"/api-reference/reinforcement-learning/register-or-update-a-tool"},{"title":"Revoke a tool registration","deprecated":null,"version":null,"openapi":"/openapi.json delete /v1/env/tools/{id}","href":"/api-reference/reinforcement-learning/revoke-a-tool-registration"}]},{"group":"Dedicated endpoints API","pages":[{"title":"List dedicated endpoints","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/dedicated","href":"/api-reference/dedicated-endpoints/list-dedicated-endpoints"},{"title":"List deploy templates","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/dedicated/templates","href":"/api-reference/dedicated-endpoints/list-deploy-templates"},{"title":"Create dedicated endpoint","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/dedicated","href":"/api-reference/dedicated-endpoints/create-dedicated-endpoint"},{"title":"Get dedicated endpoint","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/dedicated/{id}","href":"/api-reference/dedicated-endpoints/get-dedicated-endpoint"},{"title":"Update dedicated endpoint","deprecated":null,"version":null,"openapi":"/openapi.json patch /v1/dedicated/{id}","href":"/api-reference/dedicated-endpoints/update-dedicated-endpoint"},{"title":"Delete dedicated endpoint","deprecated":null,"version":null,"openapi":"/openapi.json delete /v1/dedicated/{id}","href":"/api-reference/dedicated-endpoints/delete-dedicated-endpoint"}]},{"group":"Batches API","pages":[{"title":"List batches","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/batches","href":"/api-reference/batches/list-batches"},{"title":"Create a batch","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/batches","href":"/api-reference/batches/create-a-batch"},{"title":"Retrieve a batch","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/batches/{id}","href":"/api-reference/batches/retrieve-a-batch"},{"title":"Cancel a batch","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/batches/{id}/cancel","href":"/api-reference/batches/cancel-a-batch"}]},{"group":"Model aliases API","pages":[{"title":"List aliases","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/aliases","href":"/api-reference/model-aliases/list-aliases"},{"title":"Create or repoint an alias","deprecated":null,"version":null,"openapi":"/openapi.json put /v1/aliases","href":"/api-reference/model-aliases/create-or-repoint-an-alias"},{"title":"Delete an alias","deprecated":null,"version":null,"openapi":"/openapi.json delete /v1/aliases/{id}","href":"/api-reference/model-aliases/delete-an-alias"}]},{"group":"Model versions API","pages":[{"title":"List model versions","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/model_versions","href":"/api-reference/list-model-versions"},{"title":"Get a model version","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/model_versions/{id}","href":"/api-reference/get-a-model-version"},{"title":"Adopt a model version (or roll back)","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/model_versions/{id}/adopt","href":"/api-reference/model-versions/adopt-a-model-version-or-roll-back"}]},{"group":"Audit \u0026 proving API","pages":[{"title":"Export the audit log","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/audit/export","href":"/api-reference/audit-\u0026-proving/export-the-audit-log"},{"title":"Verify the audit chain","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/audit/verify","href":"/api-reference/audit-\u0026-proving/verify-the-audit-chain"},{"title":"List deletion tombstones","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/audit/tombstones","href":"/api-reference/audit-\u0026-proving/list-deletion-tombstones"},{"title":"Record a deletion tombstone","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/audit/tombstones","href":"/api-reference/audit-\u0026-proving/record-a-deletion-tombstone"},{"title":"List enforcement refusals","deprecated":null,"version":null,"openapi":"/openapi.json get /v1/enforcement/refusals","href":"/api-reference/audit-\u0026-proving/list-enforcement-refusals"},{"title":"Verify a signed document","deprecated":null,"version":null,"openapi":"/openapi.json post /v1/verify","href":"/api-reference/audit-\u0026-proving/verify-a-signed-document"}]}]}]},"title":"Align a criterion against your labels - errorbar","buildId":"dpl_849ArpqvBiCczQiT1Y1stNGpbqHT","deploymentId":"dpl_849ArpqvBiCczQiT1Y1stNGpbqHT","dynamics":["c2aa737661","2b790eda92","b0f84f53ff","a9bb46feb3"],"chunks":["OpenApiReferenceProvider-Bbp7I60D.js","utils-C3RIvJha.js","cloneDeep-izs3Hzxb.js","_Uint8Array-BVIqJ90l.js","isObjectLike-C8vmzPEN.js","isArray-Dcu1dQ5t.js","utils-DqD-vLt8.js","toString-BX-iyh8G.js","isSymbol-CSH9U2Gm.js","addUniqueKeysToSchema-B31Loprg.js","snippetPresets-DKpk-b8t.js","style-to-object-BY3JEAdk.js","v4-DIgz8zy6.js","ApiExamples-Cj65AhOR.js","isEqual-BfaiBbcl.js","usePlaygroundInputsStore-BXtn5gcC.js","shallow-RY2b03Cp.js","getApiPlaygroundDisplay-B0jeiPk3.js","CodeSnippet-CYgBjjwn.js","CodeSnippets-DOngDrYB.js","Page-n7I5XbKb.js","getPastedJson-DRQbJx-T.js","icon-CQiQCicY.js","MethodPill-BKTLdNtu.js","DuplicateIcon-B9jpAdat.js","createIcon-DCddRrXv.js","DynamicMarkdown-a8hUKrw6.js","SectionHeading-eECVV2eH.js","axios-vUo_bRtp.js","Markdown--GF7ZIpf.js","clsx-DZLdJUrt.js","lib--scc4DXi.js","lib-B8HbPgcC.js","html-void-elements-mCSPHezU.js","dist-2LoAhjKH.js","katex-YMM-FMeD.js","marked.esm-CZWythHk.js","AssistantPageLink-Zwc0ZncM.js"]}