Skip to content

Upload an evaluation dataset ​

In a new Exact or Judge workflow, open Dataset → Upload a dataset. Drag and drop one file or choose Browse files, then Add dataset. This option is available before a dataset source or models are connected. Human evaluation continues to build its dataset from reviewed cases.

Formats and validation ​

FormatRequired structure
JSONAn array of example objects.
JSONLOne example object per nonblank line.
CSVA comma-separated table with unique, nonempty headers; quoted commas, escaped quotes and multiline fields are supported.

Use UTF-8 and a file of at most 20 MiB, containing 100–100,000 examples. The 100-row minimum follows the existing model-selection requirement. The normalized dataset must not exceed 50 MiB.

Each example needs an input and a reference answer or label:

json
{"input":"How can I change my subscription?","expected":"billing"}

Use input and expected as column names. Input aliases are prompt, question, and query; reference aliases are output, answer, label, response, and completion. Canonical names take precedence. Otherwise, multiple matching aliases are rejected as ambiguous. The selected columns must contain usable values on every row. Numeric zero and boolean false are valid values.

Combine multiple input fields into an input object in JSON/JSONL. Keep long IDs and codes with leading zeros as strings. Optional rubric must be text and metadata must be an object.

The server checks the entire file, rejecting malformed records, missing inputs/references, inconsistent CSV column counts, invalid numbers, and overly deep nesting. A short error identifies the row or parsing location where possible. Structural validation cannot verify that the reference answers are factually correct or that the data represents production traffic.

Storage and duplicates ​

The app saves normalized JSONL privately, with version snapshots for reproducible evaluations. Rows contain generated unique IDs, text inputs, references, optional rubric and metadata. Object inputs become JSON text; original IDs are retained as metadata.source_id. Other top-level fields are excluded, so put required context inside input, rubric, or metadata.

Row order and duplicates are preserved. The Dataset panel reports duplicate input/reference/rubric combinations. Uploading does not rebalance or silently cap the file at the live-ingestion default of 1,000 rows. Choose a sampling algorithm separately when running a ranking.

An uploaded file is a fixed source. Create another workflow to use a different dataset file; live push ingestion and token creation are disabled for uploaded sources. Identical retries return the already-saved upload.

A dataset is different from a reference attachment: the dataset supplies evaluation examples; attachments guide candidate answers across those examples. Shared attachments can change without replacing the uploaded dataset.

See Dataset and attachment APIs for authentication and request formats.