Skip to content

Schedules ​

This guide covers automatic evaluation. Human evaluation uses explicit case generation and review. Its schedules refresh rankings from saved reviews without generating responses or invoking an automatic judge. See Human evaluation.

Schedules automate leaderboard evaluation and fine-tuning.

Leaderboard schedule triggers ​

Leaderboards support three trigger types.

New model version ​

Run when linked Dr.Gero models publish new versions/resolutions. Cadence options:

  • DAILY
  • WEEKLY
  • MONTHLY

New data ​

Run when PUSH datasets accumulate enough new events. Configure:

  • Check cadence: HOURLY, DAILY, WEEKLY, or MONTHLY.
  • every_new_events: minimum number of new accepted rows before running.

Cron ​

Use presets:

  • DAILY: 0 0 * * * UTC
  • WEEKLY: 0 0 * * 0 UTC
  • MONTHLY: 0 0 1 * * UTC

Or provide a custom five-field cron expression.

Dataset size for scheduled runs ​

json
{
  "mode": "LIMIT",
  "limit": {
    "auto": false,
    "rows": 5000,
    "algorithm": "LAST_N"
  }
}

Algorithms:

  • LAST_N
  • RANDOM_SHUFFLE
  • WEIGHTED_SHUFFLE
  • BALANCED_UNIQUE
  • PROPORTIONAL_UNIQUE

The two label-aware options require scalar reference labels and deduplicate input/label pairs. PROPORTIONAL_UNIQUE also requires enough rows to include every observed label. Set auto: false to select an algorithm. See Dataset sampling for their behavior and score meaning. These additional algorithms apply to leaderboard evaluation, not fine-tuning.

Full leaderboard schedule example ​

json
{
  "version": 2,
  "triggers": {
    "model_version": {"enabled": true, "cadence": "WEEKLY"},
    "new_data": {"enabled": true, "check_cadence": "DAILY", "every_new_events": 500},
    "cron": {"enabled": true, "preset": "CUSTOM", "expression": "0 6 * * 1"}
  },
  "dataset": {
    "mode": "LIMIT",
    "limit": {"auto": false, "rows": 5000, "algorithm": "LAST_N"}
  }
}

At least one trigger must be enabled.

Fine-tune schedule ​

Fine-tune schedules are saved on model resources:

json
{
  "run": {"type": "WEEKLY"},
  "dataset": {
    "mode": "LIMIT",
    "limit": {"auto": false, "rows": 2000, "algorithm": "WEIGHTED_SHUFFLE"}
  }
}

Available run types are REAL_TIME, DAILY, WEEKLY, and MONTHLY.

After a schedule is saved, Dr.Gero evaluates its triggers automatically. No additional customer API call is required.