Human evaluation API

These are signed-in session endpoints, not Dr.Gero API-token automation endpoints. Every request requires a Supabase session bearer token and membership in the leaderboard’s workspace. The leaderboard must use eval_type: human. Create it through the UI’s Human evaluation workflow, then add 2–8 production candidates.

export HUMAN_BASE="$API_BASE/api/leaderboards/$LEADERBOARD_ID/human"
# SUPABASE_ACCESS_TOKEN is the current signed-in user's session token.

Never substitute a service-role key. Standard API tokens remain the credential for production inference.

Routes

Paths below are relative to $HUMAN_BASE. Bodies and responses are JSON.

Method	Path	Behavior
GET	/	Current configuration, ranking, pending counts and case summaries; offset paginates 50 cases at a time.
POST	/cases	Create an idempotent case draft.
GET	/cases/{case_id}	Saved prompt, attachments, responses, review, revision and up to 100 review-history entries.
PUT	/cases/{case_id}/config	Save a draft’s prompt override with a revision check.
POST	/cases/{case_id}/generate	Generate responses; poll case GET to display progress.
PUT	/cases/{case_id}/review	Save draft or confirm a review with a revision check.
GET	/documents	Active shared attachment metadata.
POST	/documents	Upload a shared or case-only document.
DELETE	/documents/{document_id}	Archive a shared document for future cases.
GET	/export	Download validated reference rows as a JSON array.

Upload and create a case

Upload a document first with a client-generated UUID, filename, scope and raw base64 content (no data-URL prefix):

{
  "id": "c18023ef-40f5-4c03-aeab-5943e7f46e13",
  "name": "reference.txt",
  "scope": "case",
  "data": "UmVmZXJlbmNlIHRleHQu"
}

scope is case or shared. The response is metadata: id, name, scope, media_type, size_bytes, sha256. Uploads return 201; identical retries return 200. Reusing an ID for different content or an archived document returns 409. See supported formats, limits and model compatibility.

Create a case with a different client-generated UUID:

curl -sS -X POST "$HUMAN_BASE/cases" \
  -H "Authorization: Bearer $SUPABASE_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "39d79e13-ed6c-4669-b866-07bb482d5e09",
    "input": "Summarize the reference.",
    "context": "For a non-specialist reader.",
    "prompt_override": "Answer using the supplied document.",
    "attachment_ids": ["c18023ef-40f5-4c03-aeab-5943e7f46e13"]
  }'

input is required and nonblank. Input, context and prompt each allow up to 32,000 characters. Omit prompt_override or send null to inherit; an empty string is an intentional empty prompt. attachment_ids contains only case-scope documents owned by this leaderboard. Active shared files are added automatically and snapshot with the draft. Combined limits include both scopes.

Creation returns 201, or 200 for an identical ID/payload retry. A changed payload with the same ID returns 409. Use a fresh UUID for a new case.

Edit, generate and review

Save a prompt before generation with PUT /cases/{case_id}/config:

{"revision": 0, "prompt_override": "Answer concisely using the reference."}

Always read and send the actual current revision; successful updates increment it. null restores the saved base prompt. Only drafts can be edited. Generated cases are immutable: re-evaluation creates a new case, retaining case-only documents and snapshotting current shared documents.

Send POST /cases/{case_id}/generate with {}. The request remains open until generation completes. Poll GET /cases/{case_id} while it runs; count succeeded plus failed responses against the total. An atomic generation claim prevents repeat calls for the same case from generating/billing twice. A draft whose leaderboard configuration changed returns 409; create a new case for the current models. Normal inference entitlements apply.

Responses are blind until confirmation. Use response IDs, not model IDs or display labels, when saving a review:

{
  "revision": 1,
  "confirm": true,
  "review": {
    "verdicts": {
      "RESPONSE_UUID_1": "correct",
      "RESPONSE_UUID_2": "incorrect"
    },
    "reference": "An optional expert answer.",
    "notes": "Optional reviewer notes."
  }
}

Allowed verdicts: correct, incorrect, abstain. Confirmation requires every successful response to have a verdict; failed responses must not receive one. reference and notes are required strings (empty is allowed), limited to 32,000 and 8,000 characters respectively. confirm: false saves a draft, withdrawing any previous score until reconfirmed. Both config and review writes use optimistic revision checks; on 409, reload and reconcile edits.

Summary, costs and export

Summary fields include default_prompt, cohort, candidate_count, ranking, total, pending, current_pending, next_pending_case, cases, next_offset. Follow next_offset until null. Pending counts cover all pages.

ranking.rows scores each candidate’s confirmed correct/incorrect answers. ranking.common_rows uses only equally reviewed common cases and supplies the serving winner. ranking.confirmed and ranking.comparable count confirmed and common cases. Rows include correct, total, score, rank, average_cost_usd, total_cost_usd, cost_samples, cost_missing, and technical_errors. Scores are fractions (0–1); scores/ranks are null without reviewed answers. Costs include processed, unreviewed responses; see ranking semantics.

GET /export returns reference rows from confirmed cases across configurations, including saved prompts and attachment metadata/hashes, never document bytes. Cases without any reference or accepted answer produce no row. The file is JSON, not JSONL.

Errors

• 400: invalid case, verdict, file or attachment reference.
• 401/403: missing/invalid session or workspace access; generation can also be denied by entitlements.
• 404: leaderboard, case or document unavailable to this request.
• 409: wrong evaluation mode, invalid candidate count, changed configuration, immutable case, stale revision or reused ID.
• 413: upload, extracted text or combined document limits exceeded.

A provider failure is stored on its response and does not discard successful answers. No automatic paid retries occur. Use a new case for an explicit retry. See troubleshooting.