Agent Studio authoring benchmark — 11 September 2026¶
Status: Current | Last reviewed: 2026-09-11
Use this Vanguard pilot to compare authoring model, reasoning and transport choices for an order-service voice agent. It measures generated drafts, tool behavior, generation time and estimated model cost. It does not measure live caller experience.
The interactive report contains every attempt, draft ID, quality finding and review caveat. You can also inspect the CSV and JSON.
Metered comparison¶
Each row contains two completed runs. OpenAI rows use the explicit Responses benchmark transport; Claude rows use Claude Code through Bedrock. The managed Agents API is measured separately below. These are model-and-transport comparisons, not isolated model-speed measurements.
| Model / effort | Time per run | Estimated cost per run | Verified specs per run |
|---|---|---|---|
| GPT-5.6 Terra / medium | 2:22, 2:01 | $0.34–$0.37, $0.35–$0.38 | 0/3, 0/3 |
| GPT-5.6 Sol / medium | 3:58, 4:21 | $0.88–$0.96, $1.18–$1.28 | 0/3, 0/3 |
| GPT-6 Astra / low | 7:17, 6:26 | $3.03–$3.24, $2.73–$2.93 | 3/3, 3/3 |
| GPT-6 Astra / medium | 8:07, 9:25 | $3.14–$3.37, $3.44–$3.68 | 3/3, 3/3 |
| Claude Sonnet 5 / medium | 11:04, 7:37 | $1.24, $1.43 | 0/3, 0/3 |
| Claude Opus 5 / medium | 13:24, 15:30 | $4.01, $4.21 | 0/3, 2/3 |
All twelve completed drafts pass the 21 independent execution checks. Verified specs is a separate measure: the saved sample-tool source, parameters and input schema must still match the prepared specification, and its supplied tests must pass. A zero does not mean the tool failed to execute. It means the draft lost, altered or invalidated the format that Studio can verify without executing arbitrary model-written Python.
For Astra, low averages 21.7% less active time and 12.5–12.6% lower
estimated cost than medium, while preserving all three specifications in
both runs. The API value is low, not light. This small sample supports
further testing of low effort; it does not establish a general speedup or
equal reasoning budgets across model families.
Managed Agents API¶
These checks use the provider path selected by OpenAI Agents API in Vanguard's generation settings. Usage is captured before session deletion. All three completed checks return no session/turn token usage, so their costs are unknown. Responses prices and timings must not be substituted for them.
| Model / effort | Active time | Estimated cost | Checks | Verified specs | Draft |
|---|---|---|---|---|---|
| GPT-5.6 Sol / medium | 9:56 | Unknown | 21/21 | 3/3 | 4354 |
| GPT-5.6 Terra / medium | 10:23 | Unknown | 21/21 | 3/3 | 4374 |
| GPT-6 Astra / low | 7:13 | Unknown | 21/21 | 3/3 | 4375 |
Astra-low is the fastest of these three managed runs, with all execution and specification checks passing. Sol and Terra preserve their prepared tool specifications on this transport even though their Responses drafts do not. That difference is another reason to select a model-and-transport combination rather than transfer results between harnesses.
Two-stage authoring¶
The reviewer receives a copy of the first-stage draft and the same brief, with instructions to repair material defects and preserve satisfactory parts. The original first-stage draft stays available. Totals add both measured stages and exclude the manual gap between launching them.
| Route | Combined time | Estimated cost | Checks | Verified specs | Draft |
|---|---|---|---|---|---|
| Terra → Sol | 7:13 | $1.55–$1.69 | 21/21 | 0/3 | 4361 |
| Sonnet 5 → Opus 5 | 28:40 | $6.73 | 21/21 | 3/3 | 4370 |
Terra → Sol improves the blind prompt score from 15/16 to 16/16, but does not restore verifiable sample specifications. Direct Sol is faster and cheaper in both observed runs. Sonnet → Opus restores all three specifications, at much greater time and cost than either single-stage Astra-low run.
These results do not support always adding a stronger reviewer to this task. Test escalation triggered by explicit quality gates, and compare its complete cost with direct generation by a model that already passes those gates.
Quality findings and review limits¶
The cheapest draft is not necessarily the most useful draft. Terra's second run returns vague arrival/location text and leaves its sample fixtures in implementation source rather than documenting them in the prompt. Sonnet's first clean run allows an item description to be passed where its tool requires a specific SKU. These findings help explain why passing the execution cases alone is insufficient.
The blind reviewer is Sol at high effort, scoring eight criteria out of 16. It receives the artifact without its authoring-model label. Its raw scores are retained, including mistakes:
- It alleges a postal-verification leak in Sonnet's second draft. Executing the saved tool with both an incorrect postal code and an incorrect SKU does not reproduce the leak; the source contains the necessary postal guard.
- Several Astra reviews deduct a point because the builtin
end_sessionschema is absent from the artifact. Builtins are stored by reference, and the repository contract supports the authored arguments. That is missing review context, not a verified generated-tool defect. - Opus's first clean review includes the same builtin uncertainty alongside separate concerns about verification wording and unsupported merchant capabilities.
The report shows these caveats beside the original scores. Do not use the raw scores as a precise model ranking or as proof of a runtime defect.
Recommended next decision¶
Use Astra-low as the next authoring configuration to test for this brief: it preserves verifiable sample tools on both transports, and its managed run finishes faster than the managed Sol and Terra runs. Its Responses runs provide the metered quality baseline. Keep Sol as the faster, cheaper candidate to test with a strict tool-integrity gate. Terra needs more evaluation of prompt completeness before it can serve as an unattended default. The current Claude Code runs do not show a compelling time/cost advantage over those OpenAI options for this task.
The implementation also points to a useful optimization: keep prepared tool definitions on the server and let the model refer to them when saving a draft. Re-emitting expanded source, specifications and schemas consumes output tokens and introduces copy errors. This optimization is a follow-up hypothesis, not a measured saving in this pilot.
Before selecting a general default, add other real briefs, more repeats and live voice evaluations. Include builtin contracts in future review context.
Method and retained evidence¶
- Source: WIMO Bot, agent
4310, generation5c2f60b0-ff9f-4ec6-9de7-365bb79bf0c2. The same original request plus fixed author decisions and four documented order fixtures is used for every main run. This is a controlled adaptation of a real generation, not an untouched replay of the original prompt. - Runtime: every completed draft retains
gpt-realtime-2and voicemarin. Authoring model and runtime voice model are different settings. - Models: exact requested/resolved IDs are recorded. Claude Code
2.1.258uses the available global Sonnet 5 and Opus 5 Bedrock profiles; helper-model usage is included in its cost estimate. - Timing: active generation includes tools, validation, the final model reply and cleanup. The controlled two-worker queue is reported separately.
- Execution: 19 allow/deny cases plus two follow-up SKU-disclosure cases run in code-executor mock mode with no integration grants. The latter were added after the review allegation and applied to every unchanged artifact. They do not assert every returned field or exercise real merchant actions.
- Interpretation: equivalent refusal statuses (
already_shipped,item_not_available,not_completed) are recognized consistently. Earlier evaluator revisions and raw results remain available. - Setup exclusions: four failed attempts are retained but excluded from model comparisons: the initial Claude MCP metadata incompatibility, its interrupted Opus attempt, and two managed session-creation rejections caused by shared Responses tool options. Clean replacements are used.
- Reproducibility: artifacts retain job IDs, prompts, document hashes, model/effort, usage, draft snapshots, supplied tests, independent results and blind reviews. Measurements are checked for duplicate response IDs, token-sum consistency, cost arithmetic and tool wiring.
- Cleanup: the source draft and tenant defaults remain unchanged. All 21 attempt tokens are revoked, all successful targets remain drafts, and all three completed managed sessions return HTTP 404 after deletion. Failed placeholders retain their normal error/retry panel.
The historical Claude generation's exact authoring model is not recorded, and its draft was edited after generation. It is not used as an immutable quality or latest-Claude baseline.
OpenAI estimates use the dated standard pricing snapshot, including cached input and a range for unavailable cache-write counts. Sol's listed price is promotional. Claude estimates come from the CLI's cumulative list-price calculation, including helper models. Neither is an invoice. Local infrastructure, evaluation calls and live voice usage are excluded. Managed usage can remain unavailable; unknown cost is never treated as free.