Troubleshooting

Create Leaderboard is disabled

Add and validate an OpenRouter token in Settings → Integrations.

Hugging Face dataset cannot be checked

Confirm the URL is a Hugging Face .jsonl or .jsonl.gz file. Add a Hugging Face integration for private or gated datasets.

curl -sS -X POST "$API_BASE/api/datasets/huggingface/check" \
  -H "Authorization: Bearer $DRGERO_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"dataset_url":"https://huggingface.co/datasets/.../resolve/main/eval.jsonl"}' | jq

Inference returns 409

For Exact/Judge, add at least two candidates and complete a run. For Human Eval, confirm at least one current case where every candidate has a successful Correct/Incorrect verdict; a provisional ranking alone does not enable inference.

Inference returns 403

Check token scope and plan entitlement. The token needs leaderboards:inference.

Traces are empty

Check:

• You are querying the correct leaderboard ID.
• source filter is not too narrow.
• limit, offset, and batch_limit are appropriate.
• The workspace plan includes trace access.

Push dataset accepts zero rows

Check daily/monthly quotas, max rows per request, row validity, and whether the leaderboard is actually PUSH type.

Custom model produces invalid output

Make the endpoint return OpenAI-compatible chat-completions JSON. Include choices[0].message.content when possible.

Token budget exhausted

Create a new token with a higher budget, wait for the reset interval, or lower usage. Inspect budget headers on inference responses.

Free plan limitations

The free tier may limit leaderboard count, deletion, auto-selection, token spend, and request volume. Upgrade or use a paid workspace for production automation.

Human evaluation

Symptom	Check or action
Add a case / Generate responses disabled	Use 2–8 active production models, enter input, and save any edited case prompt. No dataset rows are required.
Ranking still shows 0/0	Use Save evaluation and next case, not Save draft. Correct/Incorrect answers count; abstentions, failures and old configurations do not.
A model failed but other answers succeeded	Review the successful answers. Only the failed response is excluded from accuracy. Retry explicitly with a new case if needed.
Scores exist but inference is unavailable	Serving requires a common confirmed case with Correct/Incorrect answers for every current candidate, plus an inference token.
PDF fails for a candidate	PDF support is implemented through OpenRouter. Convert to text for non-OpenRouter candidates; PDF files are never silently omitted.
Cost is partial or unavailable	The provider did not report every cost. Missing cost is not treated as zero.
Uploaded shared file does not appear in an existing case	Shared files apply to future cases. Existing cases preserve their saved attachment snapshot.
Prompt cannot be edited	Save draft prompt changes before generation. Generated cases are immutable; re-evaluate to create a new case.
Save returns a revision conflict	Another write changed the case. Reload its latest revision and reconcile your review before saving again.
Generation appears stuck	Reopen the saved case. Interrupted pending responses recover as technical failures after approximately three minutes; no automatic paid retry is made.

See the complete Human evaluation workflow.