Appearance
Schedules
This guide covers automatic evaluation. Human evaluation uses explicit case generation and review. Its schedules refresh rankings from saved reviews without generating responses or invoking an automatic judge. See Human evaluation.
Schedules automate leaderboard evaluation and fine-tuning.
Leaderboard schedule triggers
Leaderboards support three trigger types.
New model version
Run when linked Dr.Gero models publish new versions/resolutions. Cadence options:
DAILYWEEKLYMONTHLY
New data
Run when PUSH datasets accumulate enough new events. Configure:
- Check cadence:
HOURLY,DAILY,WEEKLY, orMONTHLY. every_new_events: minimum number of new accepted rows before running.
Cron
Use presets:
DAILY:0 0 * * *UTCWEEKLY:0 0 * * 0UTCMONTHLY:0 0 1 * *UTC
Or provide a custom five-field cron expression.
Dataset size for scheduled runs
json
{
"mode": "LIMIT",
"limit": {
"auto": false,
"rows": 5000,
"algorithm": "LAST_N"
}
}Algorithms:
LAST_NRANDOM_SHUFFLEWEIGHTED_SHUFFLEBALANCED_UNIQUEPROPORTIONAL_UNIQUE
The two label-aware options require scalar reference labels and deduplicate input/label pairs. PROPORTIONAL_UNIQUE also requires enough rows to include every observed label. Set auto: false to select an algorithm. See Dataset sampling for their behavior and score meaning. These additional algorithms apply to leaderboard evaluation, not fine-tuning.
Full leaderboard schedule example
json
{
"version": 2,
"triggers": {
"model_version": {"enabled": true, "cadence": "WEEKLY"},
"new_data": {"enabled": true, "check_cadence": "DAILY", "every_new_events": 500},
"cron": {"enabled": true, "preset": "CUSTOM", "expression": "0 6 * * 1"}
},
"dataset": {
"mode": "LIMIT",
"limit": {"auto": false, "rows": 5000, "algorithm": "LAST_N"}
}
}At least one trigger must be enabled.
Fine-tune schedule
Fine-tune schedules are saved on model resources:
json
{
"run": {"type": "WEEKLY"},
"dataset": {
"mode": "LIMIT",
"limit": {"auto": false, "rows": 2000, "algorithm": "WEIGHTED_SHUFFLE"}
}
}Available run types are REAL_TIME, DAILY, WEEKLY, and MONTHLY.
After a schedule is saved, Dr.Gero evaluates its triggers automatically. No additional customer API call is required.