Appearance
Leaderboards
Leaderboards are the main Dr.Gero workflow. They define a task, dataset, evaluation method, candidate models, ranking runs, and the production inference target.
For Human evaluation, choose models first and build the dataset from reviewed cases. Follow the Human evaluation guide for per-case prompts, shared/case attachments, scoring, costs and export. The GET/PUSH and Run instructions below describe automatic evaluation.
Create Leaderboard wizard
Open Leaderboards → Create Leaderboard.
1. Name
The name is used for the leaderboard and the associated challenge.
2. System prompt
The prompt defines the task. It can include a placeholder:
text
Answer the following task as clearly and concisely as possible.
Input:
{input}Supported placeholders include {input}, {question}, and {query}. If no placeholder exists, Dr.Gero appends the input to the prompt.
3. Dataset
Choose one mode:
| Dataset mode | UI fields | Behavior |
|---|---|---|
| Upload a dataset (Dataset tab) | JSONL, JSON or CSV file | Validate every row and save privately; 100–100,000 examples, up to 20 MiB. |
| Get dataset | Hugging Face URL | Dr.Gero reads a .jsonl or .jsonl.gz dataset. The UI validates the URL before continuing. |
| Push dataset | max samples, daily/monthly limits, consolidation cadence, optional end date | Dr.Gero creates a webhook-style dataset that your app can append to. |
See file uploads for schema and storage details. Uploaded datasets are fixed sources; they cannot receive live push rows.
For automatic evaluation, push datasets require enough accepted rows before you can add models. The UI guidance uses 100 rows as the minimum before model onboarding.
4. Evaluation type
| Type | Description |
|---|---|
| Exact | Deterministic output comparison. |
| Judge | A judge provider/model scores candidate outputs. Defaults are based on OpenRouter. |
| Human | Human-reviewed evaluation workflow. |
Leaderboard detail view
After you select a leaderboard, the UI organizes the workflow into Overview, Dataset, Models, Rankings, Serve & Automate, and Activity.
Human Eval keeps these same tabs and Overview resource cards. Dataset contains cases and shared documents; Rankings contains human reviews; Activity uses Case reviews instead of Run logs. Serve & Automate uses the standard endpoint/token controls with reviewed-candidate routing, Gero-0 or a pinned production candidate. Human schedules refresh saved reviews without generating new answers.
Rankings and serving (Exact/Judge)
The Rankings area shows immutable ranking history. Serve & Automate shows the stable inference endpoint, production-routing choice, and schedule.
Model selection can be:
- Best Balanced · Gero-0: use a cost/quality routing mix when the ranking contains an eligible complementary model pair. This is the default.
- Ranking winner: automatically use the current top-ranked model.
- Manual: pin a chosen leaderboard model.
Dataset and models
These areas show:
- Candidate model list.
- System prompt.
- Dataset configuration.
- Dataset path or push webhook metadata.
- Evaluation config.
- Traces URL.
For live PUSH leaderboards, the Dataset area also lets you create a dataset token and shows example webhook usage. Uploaded files use managed storage but do not expose these ingestion controls.
All evaluation modes support shared reference attachments in Dataset. Adding, removing or replacing shared files updates evaluation identity. Established Exact/Judge workflows advance versions and mark prior rankings stale; Human evaluation tracks an attachment revision and separates comparison groups. Shared-file edits are blocked during active automatic rankings.
Activity
For Exact/Judge, Activity includes a timeline, run logs, and inference logs. Run logs show:
- Source: manual, schedule, or dataset improvement.
- Started/finished time.
- Execution time.
- Total cost.
- Ranking summary.
- Leaderboard configuration JSON.
- Model configuration JSON.
Retained run artifacts can be downloaded from the run log. Run and inference webhook settings are managed independently from their respective log views.
Add models
Click Add Model from the leaderboard detail view.
Auto-select models
Auto-select chooses OpenRouter models using constraints:
- Number of models.
- Optional input/output cost limits per million tokens.
- Optional P95/P99 latency limits.
- Optional open-source-only filter.
For Human Eval, auto-selection works before any case exists. Keep 2–8 active production candidates to generate responses. This dataset-free exception does not change Exact/Judge prerequisites.
Manual model
Manual add supports:
| Platform | Required fields | Notes |
|---|---|---|
| OpenRouter | Model ID | Uses the workspace OpenRouter integration. |
| Custom | API endpoint, optional auth method/token | Your endpoint should accept a POST request and return OpenAI-compatible chat-completions output or JSON/text. |
| Hugging Face | Endpoint URL and optional token | Available through API; the UI may mark it unavailable depending on deployment. |
| Dr.Gero | Deployed Dr.Gero model | Lets you evaluate a model created in the Models area. |
Run a leaderboard (Exact/Judge)
Click Run after at least two models are attached. The app estimates runtime based on visible model count and dataset row count. While a run is active, the UI shows status and disables conflicting actions.
Limit size and label coverage
In New ranking run → Limit size, turn Auto off to choose an algorithm. Alongside Last N, Random shuffle and Weighted shuffle, classification datasets can use Balanced labels (deduplicated) or Proportional + label coverage (deduplicated). The former emphasizes category coverage; the latter keeps original label percentages close while including at least one example of every observed label. Both require reference labels and sample without repeating input/label pairs.
See Dataset sampling for duplicate rules, minimum sizes, reproducibility and how to interpret the resulting score.
Repeated reference prompts
OpenRouter candidates can automatically reuse sufficiently large shared text prefixes when provider caching is available. Put instructions and embedded reference text before {input}. Shared text attachments in Exact, Judge and Human evaluation are also eligible; binary PDF size is not treated as a text-token count. Cache savings are best-effort and must be confirmed through provider usage. See Automatic prompt caching.
Deleting leaderboards
Paid plans can delete leaderboards. The free plan may lock deletion and limit workspaces to a small number of leaderboards.
Optimize an Exact-mode prompt
After a completed ranking, choose Optimize prompt in Rankings or beside the Overview input-prompt viewer. The system default model proposes compact edits, while the ranking winner tests them on 100–1,000 distinct examples.
Review the estimate, spending cap and editable guidance, then start the background experiment. Follow Measure baseline → Test improvements → Verify result; you can leave and return or cancel. Original documents stay protected by default.
Review verified before/after accuracy, fixed and regressed examples, estimated prompt-length change and expandable diffs. Apply as new version is available for verified improvements; inconclusive candidates remain reviewable. Apply is reversible and prompts you to rerun the ranking to determine the next winner. See Optimize prompt for requirements and limits.