Skip to content

Troubleshooting ​

Create Leaderboard is disabled ​

Add and validate an OpenRouter token in Settings → Integrations.

Hugging Face dataset cannot be checked ​

Confirm the URL is a Hugging Face .jsonl or .jsonl.gz file. Add a Hugging Face integration for private or gated datasets.

bash
curl -sS -X POST "$API_BASE/api/datasets/huggingface/check" \
  -H "Authorization: Bearer $DRGERO_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"dataset_url":"https://huggingface.co/datasets/.../resolve/main/eval.jsonl"}' | jq

Inference returns 409 ​

For Exact/Judge, add at least two candidates and complete a run. For Human Eval, confirm at least one current case where every candidate has a successful Correct/Incorrect verdict; a provisional ranking alone does not enable inference.

Inference returns 403 ​

Check token scope and plan entitlement. The token needs leaderboards:inference.

Traces are empty ​

Check:

  • You are querying the correct leaderboard ID.
  • source filter is not too narrow.
  • limit, offset, and batch_limit are appropriate.
  • The workspace plan includes trace access.

Push dataset accepts zero rows ​

Check daily/monthly quotas, max rows per request, row validity, and whether the leaderboard is actually PUSH type.

Custom model produces invalid output ​

Make the endpoint return OpenAI-compatible chat-completions JSON. Include choices[0].message.content when possible.

Token budget exhausted ​

Create a new token with a higher budget, wait for the reset interval, or lower usage. Inspect budget headers on inference responses.

Free plan limitations ​

The free tier may limit leaderboard count, deletion, auto-selection, token spend, and request volume. Upgrade or use a paid workspace for production automation.

Human evaluation ​

SymptomCheck or action
Add a case / Generate responses disabledUse 2–8 active production models, enter input, and save any edited case prompt. No dataset rows are required.
Ranking still shows 0/0Use Save evaluation and next case, not Save draft. Correct/Incorrect answers count; abstentions, failures and old configurations do not.
A model failed but other answers succeededReview the successful answers. Only the failed response is excluded from accuracy. Retry explicitly with a new case if needed.
Scores exist but inference is unavailableAutomatic routing needs current confirmed human verdicts; Gero-0 also needs eligible cost data. Alternatively pin a production candidate. An inference token is required.
PDF fails for a candidatePDF support is implemented through OpenRouter. Convert to text for non-OpenRouter candidates; PDF files are never silently omitted.
Cost is partial or unavailableThe provider did not report every cost. Missing cost is not treated as zero.
Uploaded shared file does not appear in an existing caseShared files apply to future cases. Existing cases preserve their saved attachment snapshot.
Prompt cannot be editedSave draft prompt changes before generation. Generated cases are immutable; re-evaluate to create a new case.
Save returns a revision conflictAnother write changed the case. Reload its latest revision and reconcile your review before saving again.
Generation appears stuckReopen the saved case. Interrupted pending responses recover as technical failures after approximately three minutes; no automatic paid retry is made.

See the complete Human evaluation workflow.

Shared prompts are not producing cache hits ​

Check that shared text precedes the changing input, the reusable prefix is large enough, and the target is OpenRouter. PDF byte size does not establish eligibility. Explicit caller routing/cache controls suppress Dr.Gero's automatic additions. Metadata lookup failures do not stop generation; missing cache usage is unknown. Inspect provider usage and actual cost using the caching guide.

A label-aware sample fails or contains fewer rows ​

Both new classification samplers need scalar reference labels on every row. Check the expected/output mapping. Proportional + label coverage additionally needs a row limit at least as large as the number of observed labels. Both can return fewer rows when unique input/label pairs run out; duplicates are not used to fill the gap. Review run_config.json → sampling_report and the sampling guide.

Dataset files and attachment versions ​

SymptomCheck or action
File upload rejectedUse UTF-8 JSONL, a JSON array, or CSV with headers; supply nonempty input/reference values for every row. Limits are 20 MiB and 100–100,000 rows.
Dataset source cannot be replacedFile upload is for a new automatic workflow before a source/models are connected. Create another workflow for a different dataset file.
Attachment changes do not advance an established automatic versionRefresh the leaderboard. If the version still does not reflect the change, contact support with the leaderboard ID and time of the change.
No numbered version during initial setupPre-v0 lasts until the first successful automatic ranking; shared attachment revision still tracks changes.
Attachment edit returns a wait/cancel messageAn automated ranking is active. Finish or cancel it before changing shared references.
Ranking start reports changed attachmentsReferences changed during run setup. Retry against the current configuration.
Human scores change after editing shared filesOnly reviews with a matching prompt, candidate set and shared references enter the current comparison. Earlier cases remain available.

See attachment versions and dataset validation.

Prompt optimization ​

SymptomWhat to check
Optimize prompt is unavailableUse Exact mode and complete a ranking. The application backend and worker feature must be deployed; this docs repo does not enable it.
Worker offline or system optimizer unavailableAsk the service operator to check worker heartbeat, always-on execution and the system OpenRouter credential/default model. See deployment prerequisites.
Quote rejects the target or referencesThe winner must be an OpenRouter model with supported token pricing. Convert native PDF/image guidance to text; custom and managed endpoints are currently unsupported.
Too few examplesSupply at least 100 distinct, non-conflicting string input/expected pairs after deduplication. Materialize transforms first and check field mapping/grouping.
Positive gain is inconclusiveSmall holdouts, repeated sample exposure, interrupted calls or model-version drift can prevent verification. Inspect the result note and use fresh representative data.
Spending cap reachedReview retained spend and partial results. Reduce rounds or sample size, or deliberately start a new experiment with a higher cap. Incomplete verification cannot be applied.
Cancel still shows runningCancellation takes effect after the current call. Closing the panel does not cancel. Poll until status is terminal.
Apply or restore is rejectedCheck whether workflow inputs changed or another run is active. Restore refuses to overwrite newer edits; reranking alone is allowed. Inspect the error message, even if returned as a server error.
401 or 403Use a valid Supabase user session. Reads require membership; POST actions require owner/admin, and paid starts also check entitlement. API and push tokens are not accepted.
Start response was lostLook up the original job UUID and retry with the same ID and controls. Do not automatically generate a new UUID for a paid retry.

See the workflow and API recovery guidance.