Leaderboards

Leaderboards are the main Dr.Gero workflow. They define a task, dataset, evaluation method, candidate models, ranking runs, and the production inference target.

For Human evaluation, choose models first and build the dataset from reviewed cases. Follow the Human evaluation guide for per-case prompts, shared/case attachments, scoring, costs and export. The GET/PUSH and Run instructions below describe automatic evaluation.

Create Leaderboard wizard

Open Leaderboards → Create Leaderboard.

1. Name

The name is used for the leaderboard and the associated challenge.

2. System prompt

The prompt defines the task. It can include a placeholder:

Answer the following task as clearly and concisely as possible.

Input:
{input}

Supported placeholders include {input}, {question}, and {query}. If no placeholder exists, Dr.Gero appends the input to the prompt.

3. Dataset

Choose one mode:

Dataset mode	UI fields	Behavior
Get dataset	Hugging Face URL	Dr.Gero reads a .jsonl or .jsonl.gz dataset. The UI validates the URL before continuing.
Push dataset	max samples, daily/monthly limits, consolidation cadence, optional end date	Dr.Gero creates a webhook-style dataset that your app can append to.

For automatic evaluation, push datasets require enough accepted rows before you can add models. The UI guidance uses 100 rows as the minimum before model onboarding.

4. Evaluation type

Type	Description
Exact	Deterministic output comparison.
Judge	A judge provider/model scores candidate outputs. Defaults are based on OpenRouter.
Human	Human-reviewed evaluation workflow.

Leaderboard detail view

After you select a leaderboard, the UI organizes the workflow into Overview, Dataset, Models, Rankings, Serve & Automate, and Activity.

Human Eval keeps these same tabs and Overview resource cards. Dataset contains cases and shared documents; Rankings contains human reviews; Activity uses Case reviews instead of Run logs. Serve & Automate uses the standard endpoint/token controls but requires a common human comparison and follows its winner. There are no automatic schedules or model pinning in Human Eval.

Rankings and serving (Exact/Judge)

The Rankings area shows immutable ranking history. Serve & Automate shows the stable inference endpoint, production-routing choice, and schedule.

Model selection can be:

• Best Balanced · Gero-0: use a cost/quality routing mix when the ranking contains an eligible complementary model pair. This is the default.
• Ranking winner: automatically use the current top-ranked model.
• Manual: pin a chosen leaderboard model.

Dataset and models

These areas show:

• Candidate model list.
• System prompt.
• Dataset configuration.
• Dataset path or push webhook metadata.
• Evaluation config.
• Traces URL.

For PUSH leaderboards, the Dataset area also lets you create a dataset token and shows example webhook usage.

Activity

For Exact/Judge, Activity includes a timeline, run logs, and inference logs. Run logs show:

• Source: manual, schedule, or dataset improvement.
• Started/finished time.
• Execution time.
• Total cost.
• Ranking summary.
• Leaderboard configuration JSON.
• Model configuration JSON.

Retained run artifacts can be downloaded from the run log. Run and inference webhook settings are managed independently from their respective log views.

Add models

Click Add Model from the leaderboard detail view.

Auto-select models

Auto-select chooses OpenRouter models using constraints:

• Number of models.
• Optional input/output cost limits per million tokens.
• Optional P95/P99 latency limits.
• Optional open-source-only filter.

For Human Eval, auto-selection works before any case exists. Keep 2–8 active production candidates to generate responses. This dataset-free exception does not change Exact/Judge prerequisites.

Manual model

Manual add supports:

Platform	Required fields	Notes
OpenRouter	Model ID	Uses the workspace OpenRouter integration.
Custom	API endpoint, optional auth method/token	Your endpoint should accept a POST request and return OpenAI-compatible chat-completions output or JSON/text.
Hugging Face	Endpoint URL and optional token	Available through API; the UI may mark it unavailable depending on deployment.
Dr.Gero	Deployed Dr.Gero model	Lets you evaluate a model created in the Models area.

Run a leaderboard (Exact/Judge)

Click Run after at least two models are attached. The app estimates runtime based on visible model count and dataset row count. While a run is active, the UI shows status and disables conflicting actions.

Deleting leaderboards

Paid plans can delete leaderboards. The free plan may lock deletion and limit workspaces to a small number of leaderboards.