Skip to content

Prompt optimization API ​

Session-aware clients can estimate, start, monitor, review and apply Exact-mode prompt experiments. Read the workflow and evaluation contract before interpreting scores or applying a candidate.

These endpoints require the corresponding application deployment and worker; this reference does not imply that every deployment already enables the feature.

Authentication and permissions ​

Send a Supabase user session access token in Authorization: Bearer .... Dr.Gero API tokens (drgero_...), push tokens (hpd_...) and API-token scopes do not authorize these routes.

OperationRequired access
Read setup, jobs or exportSigned-in workspace member
Quote, start, cancel, apply or rollbackWorkspace owner/admin session
Start paid experimentAlso passes the existing inference-entitlement check

Do not use a Supabase service-role key in a client. Backend credentials are resolved server-side and are not returned in job snapshots.

Routes ​

All paths below begin with:

text
/api/leaderboards/{leaderboard_id}/prompt-optimizations
MethodSuffixSuccessPurpose
GET/200Defaults, winning target, optimizer, reference metadata and the newest 20 jobs. No list pagination.
POST/quote200Prepare a sample and estimate cost without paid generation or enqueueing.
POST/202; 200 on matching retryEnqueue an experiment.
GET/{job_id}200Progress, result, original bundle, sampling report and iteration history.
POST/{job_id}/cancel200Persist cancellation request; returns {"ok":true}.
POST/{job_id}/apply200Apply a verified candidate; returns {"version_id":"…"}.
POST/{job_id}/rollback200Restore the original bundle if inputs still match; returns {"version_id":"…"}.
GET/{job_id}/export200Download authorized experiment JSON, including sample rows and response records.

Use {} for the three job-action POST bodies. GET polling responses do not include held-out row-level data; export intentionally includes the full sample. There is no optimization webhook or automatic schedule in this API.

Configuration ​

Quote and start accept the same body. id is a client-generated UUID identifying the experiment. Generate it once and reuse it for a retry of the same start request.

json
{
  "id": "9c4468f0-e4c6-4a08-8df8-b5142d053b65",
  "sample_size": 500,
  "spending_cap_usd": 5,
  "max_rounds": 5,
  "token_growth": 0.1,
  "token_allowance": 500,
  "sampling": "proportional",
  "group_path": "",
  "editable_document_ids": [],
  "max_minutes": 120,
  "max_output_tokens": 2048
}
FieldTypeDefaultLimits / meaning
idUUID stringRequiredStable request identity; also the job ID.
sample_sizeinteger500100–1,000. Actual selection can be smaller, but must contain at least 100 usable distinct examples.
spending_cap_usdnumber50.01–1,000; target and optimizer spend combined.
max_roundsinteger51–10.
token_growthnumber0.10–0.25; a fraction, not a percentage integer.
token_allowanceinteger5000–1,000 additional estimated tokens. The smaller growth constraint wins.
samplingstringproportionalproportional or random; proportional selection falls back to random for non-categorical outputs.
group_pathstring""Dotted source-row field path, up to 200 characters. If set, every row must have this field.
editable_document_idsUUID array[]Active shared text documents explicitly permitted to change; everything else stays protected.
max_minutesinteger1201–360.
max_output_tokensinteger2048128–8192, frozen for target calls.

The optimizer and target are resolved by the server; clients cannot override them in this body. Two proposals per round and two-round patience are fixed. The wider ranking sampler names such as PROPORTIONAL_UNIQUE are not values for sampling on this API.

Quote, then start ​

Obtain SESSION_ACCESS_TOKEN through your application's signed-in Supabase session. The following example requires curl, jq, and uuidgen and deliberately does not use examples/env.sh, whose token is a different authentication type.

bash
export API_BASE="https://app.drgero.ai"
export LEADERBOARD_ID="YOUR_LEADERBOARD_UUID"
# Set SESSION_ACCESS_TOKEN to the current signed-in user's access token.
export JOB_ID="$(uuidgen | tr '[:upper:]' '[:lower:]')"

jq -n --arg id "$JOB_ID" \
  '{id:$id,sample_size:500,spending_cap_usd:5,max_rounds:5}' \
  > prompt-optimization-request.json

curl --fail-with-body -sS -X POST \
  "$API_BASE/api/leaderboards/$LEADERBOARD_ID/prompt-optimizations/quote" \
  -H "Authorization: Bearer $SESSION_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  --data-binary @prompt-optimization-request.json | jq

A new quote returns these fields (numeric values are illustrative):

json
{
  "estimated_usd": 1.25,
  "conservative_ceiling_usd": 4.75,
  "target_calls": 4600,
  "sample_size": 500,
  "context_tokens_estimated": 900,
  "token_count_is_estimate": true,
  "target": {"id":"TARGET_UUID","name":"Winning model","platform":"OpenRouter"},
  "optimizer": "SYSTEM_DEFAULT_MODEL_ID",
  "sampling": {"requested":500,"selected":500,"algorithm":"proportional_unique"}
}

The response includes additional target and sampling metadata. A quote is not a reservation or a persisted job. Start resolves the configuration and prices again; refresh your estimate if the workflow changes.

After reviewing the estimate and cap, explicitly start:

bash
curl --fail-with-body -sS -X POST \
  "$API_BASE/api/leaderboards/$LEADERBOARD_ID/prompt-optimizations" \
  -H "Authorization: Bearer $SESSION_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  --data-binary @prompt-optimization-request.json | jq

A new start returns 202 with the job representation. A matching start retry returns 200 with the existing job; the same ID with different normalized controls conflicts. After a job exists, /quote with that ID also returns the existing job, rather than a new estimate. Use a new UUID only for an intentionally new experiment.

Only one queued/running experiment is allowed per leaderboard. An offline worker or missing system optimizer configuration blocks starting; unsupported provider pricing, reference types or insufficient data also prevent a usable experiment.

Polling and statuses ​

bash
curl --fail-with-body -sS \
  "$API_BASE/api/leaderboards/$LEADERBOARD_ID/prompt-optimizations/$JOB_ID" \
  -H "Authorization: Bearer $SESSION_ACCESS_TOKEN" | jq

Poll at a modest interval, such as three seconds, and stop on a terminal status. If a session expires, refresh it through your normal authentication flow. Token expiration in the client does not cancel an already queued job.

statusMeaning
queuedWaiting for a worker.
runningWorker owns the job.
completedExperiment ended; inspect result.outcome. This does not imply improvement.
canceledCancellation processed.
failedExecution stopped on an error.
budget_exhaustedThe spending cap prevented completion.

The last four statuses are terminal. stage uses baseline, improvements, verify, then normally complete; use status, not stage alone, to decide whether to stop polling. cancel_requested:true is not terminal confirmation.

Relevant response fields:

FieldMeaning
round, completed_callsProgress counters. The UI formats the in-progress round for display. Calls include optimizer work.
spend_usd, reserved_usdSettled or conservatively retained spend, and current in-flight reservation.
spend_estimatedWhether unavailable billing information/interruptions required retained estimates.
target_spend_usd, optimizer_spend_usdCost breakdown.
config, target, optimizer, estimateFrozen controls, resolved model identities and estimate.
original, sampling, iterationsAvailable on job-detail responses; original bundle, sample report and proposal history.
errorExecution/recovery explanation when present.
applied_version_id, rolled_back_atAdoption/restoration state, separate from experiment execution.

Result contract ​

result may be absent/null on failure. Its outcome is one of:

  • verified: completed positive holdout gain passed the verification gate.
  • inconclusive: evidence is insufficient or verification was not completed/reliable.
  • no_improvement: no accepted candidate or no positive final gain.

After final evaluation, the result can include before_accuracy, after_accuracy (fractions from 0 to 1), before_correct, after_correct, total, fixed, regressed, net, paired_p_value, development/validation comparisons, class_results, and estimated before/after context tokens. These fields are not promised when final verification did not happen. class_results is an object keyed by expected output, with correct, total, and an outputs map of observed output counts; generation failures use the [generation error] key.

bundle contains the candidate prompt and reference content when one is available; explanation and note explain the result and its limits. A positive numeric gain alone is not authorization to apply: check outcome === "verified" and let the user review the bundle first. Server-side Apply enforces this requirement independently.

Cancel, apply and restore ​

Cancellation is explicit:

bash
curl --fail-with-body -sS -X POST \
  "$API_BASE/api/leaderboards/$LEADERBOARD_ID/prompt-optimizations/$JOB_ID/cancel" \
  -H "Authorization: Bearer $SESSION_ACCESS_TOKEN" \
  -H "Content-Type: application/json" -d '{}' | jq

After reviewing a verified result, apply it:

bash
curl --fail-with-body -sS -X POST \
  "$API_BASE/api/leaderboards/$LEADERBOARD_ID/prompt-optimizations/$JOB_ID/apply" \
  -H "Authorization: Bearer $SESSION_ACCESS_TOKEN" \
  -H "Content-Type: application/json" -d '{}' | jq

Apply checks current workflow inputs and active runs, updates the bundle atomically, and creates a new version. It does not rerun the ranking. Use the existing leaderboard run API or Rerun ranking in the UI next.

To deliberately restore the original, POST {} to the same job's /rollback route. Restoration creates another new version and refuses to overwrite newer input edits. A reranking by itself is allowed. Apply/rollback retries are idempotent once their operation has completed; they do not reapply or undo later unrelated changes.

Export ​

bash
curl --fail-with-body -sS \
  "$API_BASE/api/leaderboards/$LEADERBOARD_ID/prompt-optimizations/$JOB_ID/export" \
  -H "Authorization: Bearer $SESSION_ACCESS_TOKEN" \
  -o prompt-optimization-experiment.json

The JSON contains the public job details plus sample and responses. Responses are keyed by call identity and contain output, failure details and available cost, usage, latency and model metadata. The sample contains partition membership, inputs and expected answers. Protect this export like the underlying dataset.

Errors and recovery ​

  • 401: use/refresh a Supabase session, not a Dr.Gero API token.
  • 403: verify workspace membership, owner/admin write access and start entitlement.
  • 404: check leaderboard/job identity and whether the backend feature is deployed.
  • 400/409/413: inspect the returned message for invalid controls, prerequisites, duplicate active jobs, changed configuration, unsupported data or size limits.
  • 503: the worker, system optimizer credential or price catalog is unavailable.
  • Other server/database errors: inspect the message and current job/workflow state. Some dataset validation and transactional Apply/rollback rejections currently surface as server errors rather than a stable, specialized conflict code.

Do not depend on private SQL error codes. After an uncertain start, GET the job by its stable ID and retry the same start body if necessary. Do not invent a new ID as an automatic retry: that could request another paid experiment. After an uncertain Apply, inspect applied_version_id before retrying. See troubleshooting.