Skip to content

Optimize prompt ​

Optimize prompt tests compact improvements to an Exact-mode task prompt and selected reference guidance. The journey is Start → Review improvement → Apply. The system default model proposes changes, the previous ranking winner answers the examples, and the deterministic evaluator measures the results.

This feature requires a deployment with the prompt-optimization backend and an available background worker. Publishing this documentation alone does not enable it.

Before you start ​

RequirementCurrent support
EvaluationExact; Judge and Human evaluation are not supported.
RankingA completed ranking with an available winning model. Optimization targets the ranking winner, even if production routing is manual or Best Balanced.
Target providerOpenRouter models with published token pricing. Custom, Hugging Face and Dr.Gero managed endpoints are not supported by this optimizer.
CredentialsThe target uses the customer's resolved model/workspace OpenRouter credential. The system default optimizer is configured by the service operator.
DatasetAt least 100 distinct, non-conflicting examples after deduplication. Source limit: 100,000 rows / 50 MiB; upload limits still apply separately.
Valuesinput and expected must be strings. Serialize object inputs and JSON expected answers as strings; materialize evaluation transforms first.
ReferencesText content, including extracted DOCX text. Native PDF/image processing is blocked; export required guidance as text first.
AccessWorkspace members can read experiments. Starting, estimating, canceling, applying and restoring require an owner/admin session. Paid starts also check inference entitlement.

Supported sources are uploaded/managed datasets, consolidated PUSH datasets and connected Hugging Face dataset files. The optimizer uses the configured field mapping. It never changes source rows or expected answers.

Use plain task instructions or simple {input}, {question}, or {query} placeholders. Literal JSON braces are supported. Label, metadata, ID and advanced Python-format placeholders are blocked for optimization: target requests must not receive expected answers.

1. Start ​

After a ranking, open Rankings → Optimize prompt. You can also open it beside the input-prompt viewer in Overview. Without a ranking, use Run first comparison.

The side panel shows the target model and system optimizer. Choose a sample size, set a spending cap and select Estimate cost before Start optimization.

We’ll test improvements to your prompt and editable guidance while keeping original documents protected.

References are protected by default. Select only task guides you want the optimizer to revise under Editable guidance. For example, select a compact classification JSON guide while leaving an authoritative source document protected. Original uploads remain immutable, including when a revised guide is eventually applied.

Defaults and Advanced controls ​

ControlDefaultAllowed values / behavior
Sample size500100–1,000; uses all available if fewer than requested but at least 100. Never pads duplicates.
Spending cap$5$0.01–$1,000; includes optimizer and target calls.
Maximum rounds51–10. Two candidates per round; one repair attempt for an invalid proposal.
Stop after no acceptance2 roundsFixed patience.
Editable context growth10%, capped at 500 estimated tokensThe smaller allowance applies against the original bundle throughout the job. Advanced limits: 0–25% and 0–1,000 tokens.
SamplingProportional with label coverageFalls back to seeded random selection for non-categorical outputs. Random selection can also be chosen explicitly.
Related-example groupingNoneOptional dotted field path, such as metadata.invoice_id.
Maximum duration120 minutes1–360.
Output limit2,048 tokens128–8,192; fixed before the experiment starts.

Token counts displayed for prompt length are estimates based on UTF-8 bytes divided by three, not the provider's tokenizer. The growth allowance covers the task prompt and editable guidance; the result's total context length also includes protected text references. Runtime request bounds account for the full rendered input.

The estimate is not a charge or a guarantee that the full search will fit the cap. For 500 examples, five rounds and two candidates per round, up to 4,600 target row evaluations are planned, plus optimizer calls. Early stopping may reduce this. If the estimate exceeds your cap, reduce rounds or raise the cap before starting.

2. Follow progress ​

The background job has three stages:

  1. Measure baseline — evaluate the unchanged bundle on a fixed sample.
  2. Test improvements — propose, validate and compare compact edits.
  3. Verify result — compare the selected candidate against the original on the final holdout.

The panel shows completed calls, round, spend and current reservation. You can close it and return later; execution does not depend on an open browser tab. Previous experiments remain available from the panel.

Cancel experiment records a cancellation request. A queued job stops without model generation. A running job stops after the current call and preserves completed responses and incurred or conservatively reserved costs. Closing the panel does not cancel the experiment.

How spending is bounded ​

Before each paid call, the worker reserves a conservative envelope based on the rendered request, output limit and frozen prices. Provider-reported usage releases unused reservations. Unknown interrupted calls retain their full reservation and are not silently sent again. The cap includes both target and optimizer calls.

If an accepted candidate exists when search can no longer fit the cap, the worker uses reserved verification budget where possible. Without completed verification, there is no recommended Apply action. A provider reporting a charge above its price envelope stops the job; the application cannot undo third-party charges.

These optimization requests use one HTTP attempt and do not use the automatic cache-fallback path described for ordinary ranking runs. Do not assume ranking cache savings when budgeting an experiment. Provider-side cache hits can still occur.

3. Review the result ​

The result leads with final before/after accuracy, examples fixed and regressed, and estimated prompt-length change. Expand the prompt/reference comparisons and experiment history to inspect what changed.

ResultMeaningNext step
Verified improvementPositive final-holdout gain passes the verification gate and no verification invalidator was detected.Review the changes, then choose Apply as new version if useful.
InconclusiveEvidence is too weak, verification is incomplete, examples overlap earlier experiments, or an interruption/model-version change undermines the comparison.Review or download the candidate. No recommended Apply action.
No improvement foundNo candidate passed selection, or final verification had zero/negative gain.Keep the original; consider better coverage or clearer source guidance.

If no candidate reaches verification, final before/after metrics are absent. The baseline is freshly measured with the experiment's generation settings; it may differ from the historical leaderboard score. A failed or canceled job is an execution status, not a finding that the original prompt is optimal.

Download experiment includes configuration, original/candidate bundles, sample membership, inputs, expected answers, responses, usage and decisions. It is a dataset export containing workspace data, not just a prompt file.

What “verified” establishes ​

The sample is frozen and divided approximately into 60% development, 20% validation and 20% final holdout, keeping configured related-example groups together. Label coverage and partition sizes are reported; rare labels may not appear in every partition. Each partition must retain at least 10 examples; oversized related-example groups may prevent a usable split. One example of a label cannot establish reliable per-label accuracy.

Only development errors and a few correct examples reach the optimizer. Validation selects candidates without exposing its examples or labels to the optimizer. A candidate needs a validation gain of at least two examples or one percentage point, whichever is larger, and no aggregate development regression.

Final verification is performed once. A positive gain with a one-sided paired sign-test p-value at most 0.05 qualifies for Verified improvement. Smaller positive gains remain inconclusive. Interrupted holdout calls, observed provider model-version drift, or overlap with previous optimization samples force an inconclusive outcome. Previously inspected data outside Dr.Gero cannot be detected; use fresh examples when you need independent evidence. Repeated jobs on a small dataset may need fresh rows before another verified result is possible.

Scoring retains the configured strict, normalized_text, or json exact-match mode, including the existing ranking worker's generated-text preprocessing. The optimizer cannot change scoring rules. Generation failures count as incorrect and are distinguished from wrong classifications; five consecutive generation failures stop the experiment.

4. Apply and rerun the ranking ​

Apply as new version is deliberate and reversible:

  • It checks that the prompt, dataset, evaluator and references still match the experiment's starting state and that no conflicting run is active.
  • It creates a single coherent new prompt/reference version. Original documents remain available to saved experiments and historical runs.
  • It marks previous rankings stale. Optimization scores never replace leaderboard scores directly.
  • It offers Rerun ranking, using the existing ranking setup.

The measured gain applies to the tested model and sample. The next ranking compares candidates under the new bundle and determines the winner. Optimization is not a production-accuracy guarantee, a model deployment or a fine-tuning run. Reference attachments used in evaluation are still not automatically inherited by production inference; see reference attachment behavior.

Use Restore original version to restore the original bundle as another new version. A reranking alone does not prevent restoration, but subsequent changes to the prompt, dataset, evaluator or references do: restoration must not overwrite newer work. Rerun the ranking after restoration as well.

Developer integration and service availability ​

Use the prompt-optimization API for session-aware client integrations. These routes do not accept Dr.Gero API tokens or push tokens, and optimization is not triggered automatically by a ranking schedule.

For hosted workspaces, an offline-worker or missing-system-credential error needs the service operator's attention. Developers should not put service credentials in a browser, public repo or example request.

Operators deploying the application feature must apply its 20260930130000_prompt_optimization.sql migration after the reference-document migrations, deploy the matching frontend/API, and run npm run worker:prompt in the application frontend image as an always-on process. That command and migration belong to the application feature patch, not this public documentation repository. The worker needs the server's Supabase/integration configuration and DR_GERO_OPENROUTER_API_KEY; it resolves DEFAULT_LLM_MODEL, then SYSTEM_DEFAULT_LLM_MODEL, then the application fallback. An embedded worker can be enabled with PROMPT_OPTIMIZATION_WORKER=true only where the process retains CPU between requests. A request-only serverless instance is insufficient.

See troubleshooting for recovery steps.