Appearance
Troubleshooting
Create Leaderboard is disabled
Add and validate an OpenRouter token in Settings → Integrations.
Hugging Face dataset cannot be checked
Confirm the URL is a Hugging Face .jsonl or .jsonl.gz file. Add a Hugging Face integration for private or gated datasets.
bash
curl -sS -X POST "$API_BASE/api/datasets/huggingface/check" \
-H "Authorization: Bearer $DRGERO_TOKEN" \
-H "Content-Type: application/json" \
-d '{"dataset_url":"https://huggingface.co/datasets/.../resolve/main/eval.jsonl"}' | jqInference returns 409
For Exact/Judge, add at least two candidates and complete a run. For Human Eval, confirm at least one current case where every candidate has a successful Correct/Incorrect verdict; a provisional ranking alone does not enable inference.
Inference returns 403
Check token scope and plan entitlement. The token needs leaderboards:inference.
Traces are empty
Check:
- You are querying the correct leaderboard ID.
sourcefilter is not too narrow.limit,offset, andbatch_limitare appropriate.- The workspace plan includes trace access.
Push dataset accepts zero rows
Check daily/monthly quotas, max rows per request, row validity, and whether the leaderboard is actually PUSH type.
Custom model produces invalid output
Make the endpoint return OpenAI-compatible chat-completions JSON. Include choices[0].message.content when possible.
Token budget exhausted
Create a new token with a higher budget, wait for the reset interval, or lower usage. Inspect budget headers on inference responses.
Free plan limitations
The free tier may limit leaderboard count, deletion, auto-selection, token spend, and request volume. Upgrade or use a paid workspace for production automation.
Human evaluation
| Symptom | Check or action |
|---|---|
| Add a case / Generate responses disabled | Use 2–8 active production models, enter input, and save any edited case prompt. No dataset rows are required. |
| Ranking still shows 0/0 | Use Save evaluation and next case, not Save draft. Correct/Incorrect answers count; abstentions, failures and old configurations do not. |
| A model failed but other answers succeeded | Review the successful answers. Only the failed response is excluded from accuracy. Retry explicitly with a new case if needed. |
| Scores exist but inference is unavailable | Automatic routing needs current confirmed human verdicts; Gero-0 also needs eligible cost data. Alternatively pin a production candidate. An inference token is required. |
| PDF fails for a candidate | PDF support is implemented through OpenRouter. Convert to text for non-OpenRouter candidates; PDF files are never silently omitted. |
| Cost is partial or unavailable | The provider did not report every cost. Missing cost is not treated as zero. |
| Uploaded shared file does not appear in an existing case | Shared files apply to future cases. Existing cases preserve their saved attachment snapshot. |
| Prompt cannot be edited | Save draft prompt changes before generation. Generated cases are immutable; re-evaluate to create a new case. |
| Save returns a revision conflict | Another write changed the case. Reload its latest revision and reconcile your review before saving again. |
| Generation appears stuck | Reopen the saved case. Interrupted pending responses recover as technical failures after approximately three minutes; no automatic paid retry is made. |
See the complete Human evaluation workflow.
Shared prompts are not producing cache hits
Check that shared text precedes the changing input, the reusable prefix is large enough, and the target is OpenRouter. PDF byte size does not establish eligibility. Explicit caller routing/cache controls suppress Dr.Gero's automatic additions. Metadata lookup failures do not stop generation; missing cache usage is unknown. Inspect provider usage and actual cost using the caching guide.
A label-aware sample fails or contains fewer rows
Both new classification samplers need scalar reference labels on every row. Check the expected/output mapping. Proportional + label coverage additionally needs a row limit at least as large as the number of observed labels. Both can return fewer rows when unique input/label pairs run out; duplicates are not used to fill the gap. Review run_config.json → sampling_report and the sampling guide.
Dataset files and attachment versions
| Symptom | Check or action |
|---|---|
| File upload rejected | Use UTF-8 JSONL, a JSON array, or CSV with headers; supply nonempty input/reference values for every row. Limits are 20 MiB and 100–100,000 rows. |
| Dataset source cannot be replaced | File upload is for a new automatic workflow before a source/models are connected. Create another workflow for a different dataset file. |
| Attachment changes do not advance an established automatic version | Refresh the leaderboard. If the version still does not reflect the change, contact support with the leaderboard ID and time of the change. |
| No numbered version during initial setup | Pre-v0 lasts until the first successful automatic ranking; shared attachment revision still tracks changes. |
| Attachment edit returns a wait/cancel message | An automated ranking is active. Finish or cancel it before changing shared references. |
| Ranking start reports changed attachments | References changed during run setup. Retry against the current configuration. |
| Human scores change after editing shared files | Only reviews with a matching prompt, candidate set and shared references enter the current comparison. Earlier cases remain available. |
See attachment versions and dataset validation.
Prompt optimization
| Symptom | What to check |
|---|---|
| Optimize prompt is unavailable | Use Exact mode and complete a ranking. The application backend and worker feature must be deployed; this docs repo does not enable it. |
| Worker offline or system optimizer unavailable | Ask the service operator to check worker heartbeat, always-on execution and the system OpenRouter credential/default model. See deployment prerequisites. |
| Quote rejects the target or references | The winner must be an OpenRouter model with supported token pricing. Convert native PDF/image guidance to text; custom and managed endpoints are currently unsupported. |
| Too few examples | Supply at least 100 distinct, non-conflicting string input/expected pairs after deduplication. Materialize transforms first and check field mapping/grouping. |
| Positive gain is inconclusive | Small holdouts, repeated sample exposure, interrupted calls or model-version drift can prevent verification. Inspect the result note and use fresh representative data. |
| Spending cap reached | Review retained spend and partial results. Reduce rounds or sample size, or deliberately start a new experiment with a higher cap. Incomplete verification cannot be applied. |
| Cancel still shows running | Cancellation takes effect after the current call. Closing the panel does not cancel. Poll until status is terminal. |
| Apply or restore is rejected | Check whether workflow inputs changed or another run is active. Restore refuses to overwrite newer edits; reranking alone is allowed. Inspect the error message, even if returned as a server error. |
| 401 or 403 | Use a valid Supabase user session. Reads require membership; POST actions require owner/admin, and paid starts also check entitlement. API and push tokens are not accepted. |
| Start response was lost | Look up the original job UUID and retry with the same ID and controls. Do not automatically generate a new UUID for a paid retry. |
See the workflow and API recovery guidance.