Appearance
Sampling datasets for ranking runs
Use a limited dataset to compare candidates at a manageable cost. Choose a sampler based on the question you want the ranking to answer: common production cases, category coverage, or a compromise between them.
Open New ranking run → Limit size, turn Auto off, enter the row limit and choose Algorithm. The same choices are available for automatic leaderboard schedules. These options do not change fine-tuning samplers or Human evaluation, which ranks reviewed cases instead of sampling automatic runs.
Choose an algorithm
| UI option | API value | Use it for |
|---|---|---|
| Last N | LAST_N | Recent rows in dataset order. |
| Random shuffle | RANDOM_SHUFFLE | A simple random subset retaining label frequencies in expectation; rare labels may be absent. |
| Weighted shuffle (latest > older) | WEIGHTED_SHUFFLE | Favoring later rows in dataset order. |
| Balanced labels (deduplicated) | BALANCED_UNIQUE | Broad category coverage, with approximately equal allocations where capacity permits. |
| Proportional + label coverage (deduplicated) | PROPORTIONAL_UNIQUE | Staying close to original label percentages while including every observed label. |
The API also accepts FIRST_N; it is not a choice in the ranking dropdown. The new options do not change the default algorithm or the Auto setting.
Shared rules for the two label-aware samplers
Both require a non-empty scalar reference label on every source row, selected through the existing expected/output mapping. Missing labels or structured answers fail validation before candidate generation. These samplers are intended for classification, not free-form answers where every answer may be a distinct class.
Duplicates are canonical input + label pairs:
- Structured inputs are compared by their content.
- Recognized top-level row identifiers are excluded from duplicate comparisons unless they are the only input fields. Nested identifiers remain significant. The selection report lists the identifier fields excluded from comparison.
- Keep descriptions, concepts, amounts and other semantic values. Numeric
1and1.0compare equal, but numeric strings, text case and whitespace remain distinct. - Keep different labels on the same input separately and report those conflicts; never automatically relabel them. No fuzzy duplicate detection is performed.
Unique pairs are shuffled within each label using the saved run seed. Each pair is selected at most once. If unique examples run out, the sampler returns fewer rows than requested rather than padding duplicates. Selected rows retain their original inputs, labels and identifiers and are returned in source order.
All candidate models in a run use the same selection. Saved selection provenance includes the seed; explicitly reusing a ranking's selection preserves it. Different seeds can produce different examples and should not be mistaken for an identical benchmark. These samplers do not remove training/test leakage.
Balanced labels
Allocate one example at a time to a least-represented label that still has unique examples. Rare labels may run out early; their remaining slots go to other labels. If the row budget is smaller than the number of labels, the seed chooses a subset of labels. Full coverage is only possible with a sufficient budget.
Use this option to expose weaknesses hidden by a dominant class. Its accuracy reflects the selected sample, not production frequencies. When rare classes have fewer examples, it is not exactly macro-average accuracy either.
Proportional + label coverage
This is proportional stratified sampling with a one-example-per-label floor:
- Count original label frequencies before deduplication.
- Reserve at least one unique example for every observed label.
- Allocate the remaining slots to stay as close as possible to the original label percentages, subject to available unique examples and integer row counts.
The specific objective minimizes the sum of squared differences between selected and original label percentages. Ties use the run seed. For actual sample size M, original class count C_i, source size N and unique class capacity U_i, quotas satisfy sum(q_i) = M and 1 <= q_i <= U_i, minimizing sum((q_i - M * C_i / N)^2). The actual size is capped at the total unique capacity.
If the row limit is below the number of observed labels, the run fails validation with the minimum required count. Increase the limit or choose another algorithm.
This is a gentler adjustment than equal balancing. It does not preserve original frequencies perfectly: coverage floors and limited unique capacity can force a change. Uniform sampling of unique pairs also changes duplicate frequencies within each label. No automatic score weighting is added.
Coverage is not confidence
One example ensures that a label appears, but cannot establish reliable accuracy for that label. Use a larger coverage set and collect more reviewed examples when rare categories matter to the business.
Example: an imbalanced classification dataset
Consider a fictional support dataset with three labels: Billing (80%), Technical (15%), and Account (5%). For a 60-row sample, assuming enough unique examples in each label:
| Sampler | Billing | Technical | Account |
|---|---|---|---|
| Balanced labels | 20 | 20 | 20 |
| Proportional + label coverage | 48 | 9 | 3 |
The balanced sample gives each category equal representation. The proportional sample preserves these example frequencies while covering every label. Actual allocations depend on the row budget and available unique examples.
Audit a selection
For the new samplers, retained run_config.json includes sampling_report with:
- Source rows, unique input/label pairs and duplicate rows removed.
- Conflicting unique inputs, requested/selected rows and labels covered.
- Source, unique and selected counts for each label.
- Algorithm name, sampler version and ignored identifier fields.
The proportional sampler additionally records its allocation objective, minimum per label, source/selected fractions and total variation distance between label distributions. This is a sampling audit, not a per-class model-accuracy report.
For production selection, pair overall representative performance with per-label checks and requirements for critical categories. Neither sampler alone guarantees that the ranking winner meets every business requirement.
See Run API, Schedules and Automatic prompt caching.
Optimization samples
Prompt optimization has a separate sampling contract: 100–1,000 distinct, non-conflicting examples, a frozen approximately 60/20/20 development/validation/holdout split, and optional related-example grouping. At least 100 usable examples must remain after deduplication. Its API accepts proportional or random, not the ranking sampler enum names above. Small or rare classes may be absent from a partition; inspect the coverage report before drawing per-class conclusions. Previously used optimization samples can make verification inconclusive.