Human evaluation

Compare models using your reviewers’ judgments, even before you have a dataset. Enter a case, generate candidate answers, and mark the correct responses. Reviewed cases build a reusable dataset as you work.

Start with models

1. Save your OpenRouter integration in Settings → Integrations.
2. Create a leaderboard, select Human evaluation, and enter its name and default system prompt. You do not need a Hugging Face dataset or PUSH rows.
3. In Models, add 2–8 active production candidates. Browse the catalog or use Auto-select models with cost, latency and open-source preferences. Auto-selection without dataset rows is available specifically for Human Eval; normal plan limits and fees still apply.
4. Open Rankings → Evaluate a new case, or Dataset → Add a case.

A pre-production model does not count toward the minimum. If case creation is disabled, follow Choose models and check the production candidate count.

Create and generate a case

Enter Input and optional Context. Keep Use leaderboard prompt enabled to inherit the default, or turn it off and enter a Prompt for this case. The same saved prompt and case input are used for every compatible candidate.

You can save a case draft before generation. On a saved draft, click Save case prompt after editing the prompt; unsaved prompt changes disable generation. Once responses have been generated, the prompt is immutable. Use Re-evaluate with current models to create a new case instead.

Click Generate responses to send one request per candidate. The progress card shows elapsed time and the number of processed responses, including failures. A failed provider does not discard the other models’ answers. Requests incur normal provider usage charges.

Cases are saved before generation and can be reopened. There are no automatic paid retries: re-evaluation explicitly creates a new case and generates new responses. If generation was interrupted, reopen the case; unresolved responses are marked as technical failures after the recovery window (approximately three minutes).

Attach documents

Choose the scope when uploading:

Scope	Use
This case only	Include the document in the case you are creating.
All future cases in this leaderboard	Share reference material with every subsequently created case. Manage these files in Dataset, where they are labeled All future cases.

New cases snapshot the active shared files plus their case-only files. Adding or removing a shared attachment does not change saved cases. Removing a shared file archives it for future use; existing cases keep their original reference. Re-evaluation retains case-only attachments and uses the current shared files.

Formats and limits

• UTF-8 text: .txt, .md, .markdown, .csv, .tsv, .json, .yaml, .yml, .xml, .html, .htm.
• .docx: extracted as plain text; images and layout are not preserved.
• .pdf: supported through OpenRouter as described below.
• Other office or legacy formats: export to PDF or a supported text format first.
• Maximum 5 MiB per file, 8 files and 10 MiB combined per case, counting shared and case-only files together.
• Extracted text must contain at most 200,000 characters per file. Oversized text is rejected, not silently truncated. Provider context limits still apply.

What each model receives

Attachment	OpenRouter candidates	Custom, Hugging Face and Dr.Gero candidates
Text or DOCX	Text appended to the user input.	Text appended to the user input.
PDF	File content parts with private base64 data URLs and OpenRouter’s file-parser plugin. Native PDF handling or a parsing fallback is selected by OpenRouter.	Explicit unsupported-file technical failure before a provider request; convert to text to compare these candidates.

PDFs are never silently omitted. Compatible candidates continue even if another candidate cannot process the attachment. PDF parsing can add provider charges. See OpenRouter’s PDF documentation.

Files are stored privately and accessed through workspace-authorized routes. Generating responses sends their contents to the selected compatible model providers. Dataset exports contain attachment metadata and hashes, not the files themselves.

Review answers

Responses initially appear as A, B, C… in a saved random order, without model names. For every successful answer choose:

• Correct — counts as a correct answer.
• Incorrect — counts as a reviewed but incorrect answer.
• Cannot evaluate — abstains; excluded from accuracy.

Several answers may be correct. None of the answers is correct marks all successful answers incorrect; technical failures remain separate and need no verdict.

Optionally write a reference answer, choose Use as reference, and add reviewer notes. A reference answer never awards a model points or changes its original response.

Use Save draft to pause, or Save evaluation and next case to confirm the review, update the ranking and open the next input. Confirmation requires a verdict for every successful response. Model names are revealed after confirmation. Saving a revised draft of a confirmed case withdraws its scores until you confirm again. Review history retains revisions; if another reviewer saves first, reload before resubmitting.

Save edits before switching tabs or cases. Once at least one case exists, opening Rankings shows the ranking and case list; the new-case form opens only when requested, except when continuing directly after saving an evaluation.

Read the ranking

Each model is scored on its own confirmed Correct/Incorrect answers in the current configuration:

Accuracy = correct answers / reviewed answers

Pending reviews, abstentions and technical failures do not enter that denominator. For example, if three models are correct, one incorrect and one fails on a confirmed case, the rows show 1/1, 1/1, 1/1, 0/1 and 0/0. The failed model is Not evaluated, with a separate technical error.

The ranking is labeled Provisional because sample counts can differ. Tied scores share a rank. Common cases are confirmed cases with a successful Correct/Incorrect verdict for every current candidate.

Avg. cost / response and Total cost (USD) include recorded provider costs from all processed responses in the current configuration, including unreviewed cases. Unknown costs are not zero: partial averages use only responses with recorded costs. An explicit zero can represent a free response. These columns are provider costs, not additional platform fees. Older stored zero values may not distinguish free usage from missing cost data.

Changing the leaderboard default prompt, candidate set, endpoint identity, or managed deployment/version starts a separate comparison. Historical cases remain accessible. Per-case prompt overrides and attachments are evaluation inputs and do not split the comparison. Use a separate leaderboard if you want to compare a different task independently.

Use the familiar tabs

Tab	Human Eval behavior
Overview	Models, Dataset, Ranking and Serve & Automate resource cards, readiness and a recommended next action. Pending current-configuration cases take priority over starting another.
Dataset	Shared documents, cases and Export validated dataset.
Models	Browse, auto-select and manage production candidates before collecting cases.
Rankings	Provisional scores, reviewed counts, costs, technical errors and case review.
Serve & Automate	The same endpoint, token, copy and test controls, with human-comparison readiness. Automatic schedules and manual model pinning are unavailable.
Activity	Timeline, Case reviews in place of automatic Run logs, and Inference logs. Open cases to inspect review history.

Serve the reviewed winner

Serving requires at least one common case in the current configuration and an API token with leaderboards:inference. A provisional ranking alone is not enough.

The existing inference endpoint selects the winner of a separate ranking computed on equal common cases. It can differ from the provisional leader, whose score uses all of its reviewed answers. Human Eval follows this common-case winner; Best Balanced and manual pinning are unavailable. Automatic runs, dataset-improvement runs and evaluation schedules are also unavailable: reviewers must generate and review cases explicitly.

Inference uses the leaderboard default prompt and the caller’s input. It does not inherit an arbitrary evaluation case’s prompt override or attachments.

Export and reuse the dataset

Dataset → Export validated dataset downloads JSON for confirmed cases with a reference or at least one accepted answer. Pending reviews are excluded. An all-incorrect case needs a manually written reference to produce an export row.

Each row includes case_id, input, context, saved prompt, output, accepted_outputs, attachment metadata, cohort and review_revision. The export can include historical configurations; filter by cohort when needed. Document bytes are not included.

To reuse this material in automatic evaluation or fine-tuning, prepare it in the destination’s required dataset format and include any needed document content explicitly. Existing training-data requirements still apply. Once human cases exist, changing that leaderboard’s evaluation method is blocked; create a separate leaderboard for another method.

For session-authenticated case and document operations, see the Human evaluation API.