Datasets & formats

Fine-tuning trains on your data. You provide it either as a training file (JSONL) or a managed dataset, mapped into one of the supported training formats. Files and datasets are managed via the API at https://gateway.errorbar.ai/v1 or the dashboard’s fine-tuning wizard. You can also curate a managed dataset straight from your logged traffic; filtered by task, model, or auto-detected segment, with a disjoint eval holdout. A dataset is a frozen snapshot: logs age out on your retention window (7/30/90 days), but a dataset curated from them does not: curate what matters before it expires.

Scope a dataset to one task. A dataset built from mixed traffic teaches a muddle of behaviours; one scoped to a single objective teaches that objective well. Send a task label on every call with the X-Errorbar-Tag header (e.g. document-summary), then filter by it when you curate. Because the task label is independent of the model, the same task spans every model you route it to; swap models or rewrite prompts, and the task identity (and its dataset) persists. Calls with no tag fall under untagged traffic.

curl https://gateway.errorbar.ai/v1/chat/completions \
  -H "X-Errorbar-Tag: document-summary" \
  -H "Authorization: Bearer $ERRORBAR_API_KEY" \
  -d '{ "model": "...", "messages": [...] }'

Curation isn’t just a filter; it cleans as it builds. Only successful responses are included; byte-identical duplicates (cache-replay collapse), truncated answers (finish_reason=length), and empty replies are dropped, and multi-turn conversations are reconstructed and folded so early turns aren’t over-trained. Every build reports exactly what it kept and why (“kept 4,812 of 6,003: 891 duplicates, 300 truncated…”).

Curating from logs

POST /v1/datasets/from_logs turns logged traffic into a managed dataset. It needs an owner/admin key and a name (80 characters or fewer); everything else narrows what feeds the build.

curl -X POST https://gateway.errorbar.ai/v1/datasets/from_logs \
  -H "Authorization: Bearer $ERRORBAR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "support-bot-v1",
    "holdout_pct": 10,
    "filters": { "tag": "support-bot", "cache_hit": false },
    "quality": { "mode": "judge", "criterion_id": "crit_..." }
  }'

Choosing the traffic

Field	What it selects
`filters.tag`	The task label sent as `X-Errorbar-Tag` — the sharpest filter you have
`filters.segment`	One auto-detected segment, for traffic you never tagged
`filters.model`	Exchanges served by one model
`filters.finish_reason`, `filters.cache_hit`	Narrow by how the response ended, or exclude cache replays
`filters.start`, `filters.end`	Unix seconds; `start` inclusive, `end` exclusive
`sources.models`	An include-list, when the same task spans several models and you want only some

The build reads the 50,000 most recent matching rows. Filter deliberately on high-volume traffic rather than assuming a bare name sweeps everything.

The quality ladder

quality.mode decides how high the bar sits. This is the part that makes a dataset yours rather than merely clean:

cleaned — mechanical curation only (default)

Drops errored, truncated, empty, and duplicate exchanges. No judgement of whether the answers were any good.

graded — only what a human passed

Keeps only exchanges a person graded pass. Free, and the highest-precision option you have, bounded by how much you’ve graded. The response reports ungraded_excluded so you can see what the bar cost you.

judge — only what a calibrated judge passed

Applies a calibrated judge at each conversation’s terminal turn, which is how you get graded-quality selection at a volume no person could grade by hand. criterion_id is required, and the criterion must be calibrated to the aligned tier, request-unit, and free of a drift flag; a segment-bound judge only works on a build scoped to its segment. Each refusal is written to the refusal ledger rather than silently downgrading the build.

In every mode, an exchange a human graded FAIL never enters a dataset — nor does one sharing an agent run with a trace-scoped FAIL. The ladder raises the bar; it never lowers it below your own grades.

judge mode spends. It is gated up front at roughly $0.02 per conversation judged and refuses with 402 before spending anything if the wallet can’t cover the estimate; each judge call is then metered as usage. A scoring failure partway through fails the whole build — no dataset is created — but rows already judged were billed. Retries are idempotent and don’t re-bill.

Decontamination

decontaminate defaults to true: rows whose prompt shares a 13-word shingle with a public benchmark’s test split are dropped, so a later benchmark score measures capability rather than a memorised answer key. Pass false to keep them. If the contamination index is unavailable the provenance records “not checked” — never “clean”.

The eval split

holdout_pct (0–50) carves a second, disjoint {name}-eval dataset out of the curated lines. No example appears in both, which is exactly what makes it usable as an eval source: pass its id as sample_filters.dataset_id when queueing an eval.

Carve the holdout at build time. It’s the difference between an eval set and the model’s own homework, and you can’t reconstruct disjointness later by splitting a dataset you already trained on.

What the response tells you

A 201 returns training_name / training_count, the eval pair when you asked for a split, and two accounting objects worth reading rather than skipping:

summary — total, kept, and dropped broken out by reason (unparseable, errored, truncated, empty, duplicate, human_failed, contaminated), plus the multi-turn folding counts. A dataset is never silently thinner than you think.
quality — what the gate actually did: the mode, the judge’s kappa at build time, human_pass_kept, ungraded_excluded, and in judge mode judged, judge_passed, judge_failed, judge_unparsed, and judge_spend_usd. A dataset never claims more than what happened.

If curation leaves nothing usable, the call returns 400 and creates nothing.

Upload methods

Direct file

Upload a JSONL file directly as a training file.

Object storage (S3)

Use an S3-compatible bucket as a dataset source with your credentials.

Chunked upload

Large files upload in parts (chunked/multipart) so big datasets upload reliably.

Uploading a training file

Upload a JSONL training file with POST /v1/fine_tuning/files, and list your files with GET /v1/fine_tuning/files.

# Upload
curl https://gateway.errorbar.ai/v1/fine_tuning/files \
  -H "Authorization: Bearer $ERRORBAR_API_KEY" \
  -F "file=@training.jsonl"

# List
curl https://gateway.errorbar.ai/v1/fine_tuning/files \
  -H "Authorization: Bearer $ERRORBAR_API_KEY"

import requests

BASE = "https://gateway.errorbar.ai/v1"
headers = {"Authorization": f"Bearer {ERRORBAR_API_KEY}"}

# Upload
with open("training.jsonl", "rb") as f:
    file = requests.post(
        f"{BASE}/fine_tuning/files",
        headers=headers,
        files={"file": f},
    ).json()

# List
files = requests.get(f"{BASE}/fine_tuning/files", headers=headers).json()

The response includes the file’s id, which you reference as trainingFileId (or validationFileId) when creating a job.

Training formats

When you use a managed dataset, you map its columns into one of four training formats. Pick the format that matches your task:

Text — raw text completion

Plain text. Map a single column to the text field. Use for continued pretraining or simple completion.

{ "type": "text", "text": "<column>" }

Prompts — prompt / completion pairs

Prompt/completion pairs. Map one column to the prompt and one to the completion. Use for instruction tuning.

{ "type": "prompts", "prompt": "<column>", "completion": "<column>" }

Messages — chat format (role / content)

OpenAI chat format: role/content conversations. Map the column that holds the conversation. Use for chat/assistant fine-tuning.

{ "type": "messages", "messages": "<column>" }

Pretokenized — token-id arrays

Already-tokenized data as token-id arrays. Map input_ids (and optionally labels, attention_mask). Use when you control tokenization.

{ "type": "pretokenized", "input_ids": "<column>", "labels": "<column>" }

Validation

JSONL is validated before a job starts. Line-level JSONL errors and format problems are surfaced with the exact reason, so you fix them before spending on a job rather than after. You can also preview dataset rows in the dashboard before training.

Using a dataset in a job

Once your file or dataset is ready, reference it when creating a fine-tuning job. A managed dataset is converted to a training_file automatically, using the column-mapping format you chose; you then pass the resulting file id as trainingFileId.

Create a job

Start training with your prepared data.

Deploy the result

Serve a trained model on a dedicated endpoint.

Get started

Capture

Observe

Grade & Judge

Evaluate

Improve

Run at scale

Platform

Datasets & formats

Curating from logs

Choosing the traffic

The quality ladder

Decontamination

The eval split

What the response tells you

Upload methods

Direct file

Object storage (S3)

Chunked upload

Uploading a training file

Training formats

Validation

Using a dataset in a job

Create a job

Deploy the result

​Curating from logs

​Choosing the traffic

​The quality ladder

​Decontamination

​The eval split

​What the response tells you

​Upload methods

Direct file

Object storage (S3)

Chunked upload

​Uploading a training file

​Training formats

​Validation

​Using a dataset in a job

Create a job

Deploy the result

Curating from logs

Choosing the traffic

The quality ladder

Decontamination

The eval split

What the response tells you

Upload methods

Uploading a training file

Training formats

Validation

Using a dataset in a job