Agent Studio authoring benchmark

• The built-in end_session tool’s schema and behavior are not shown, leaving its prescribed argument names and values unverifiable.

Reproduction caveat: The sole blind-review deduction is that the builtin end_session schema is absent from the draft. Builtins are stored by reference. The repository contract supports the exact authored arguments reason="goodbye" and session_escalated=false (builtin-contracts.json and builtin-tools.ts). This is missing review context, not an established generated-tool defect; retain the raw 15/16 score with this caveat.

• The actual schema and behavior of builtin:end_session are not included, so the prescribed reason and session_escalated arguments cannot be confirmed as valid.

Reproduction caveat: The sole blind-review deduction is that the builtin end_session schema is absent from the draft. Builtins are stored by reference. The repository contract supports the authored reason and session_escalated arguments. This is missing review context, not an established generated-tool defect; retain the raw 15/16 score with this caveat.

• The artifact does not provide the actual schema or implementation behavior for builtin:end_session, so alignment of reason='goodbye' and session_escalated=false cannot be confirmed without relying on unwritten platform assumptions.

Reproduction caveat: The sole blind-review deduction is that the builtin end_session schema is absent from the draft. Builtins are stored by reference. The repository contract supports the authored reason and session_escalated arguments. This is missing review context, not an established generated-tool defect; retain the raw 15/16 score with this caveat.

• The end-session reference contains the unresolved text “{{TOOL:end_session}}” and no actual built-in schema is shown, leaving its stated arguments and invocation format unverifiable.

Reproduction caveat: The reviewer treats {{TOOL:end_session}} as unresolved and cannot find the builtin schema in the draft. The tool token is supported Studio prompt syntax, and builtin:end_session is stored by reference. Its repository contract supports reason and session_escalated. This is missing platform context for the reviewer, not an established generated-tool defect; the raw 15/16 score is retained.

• The verification flow acts as an order-number existence oracle: known-order postal mismatches produce “verification_failed” and the caller is told the postal code “does not match what is on that order,” while unknown numbers produce “not_found.” This conflicts with “do not hint at whether the order exists.”
• The unsupported-request instruction says the merchant’s customer service “can” perform the listed actions, which includes issuing refunds or credits, sending labels or emails, and booking callbacks. Those real-world capabilities are not established by any tool or fixture.
• The artifact does not expose the builtin end_session schema, yet directs calls with reason and session_escalated and describes hand-back behavior; that portion of tool alignment is unverifiable.

Reproduction caveat: The reviewer could not verify the builtin end_session arguments from the draft alone. The platform contract supports reason and session_escalated; builtins are stored by reference. This resolves that schema uncertainty, but does not prove deployment-specific handoff behavior or dismiss the separate wording and verification findings.

• The return flow misleadingly permits a caller's item description to be sent as item_sku, but the wired tool accepts only exact SKU-RED-1 for the eligible fixture. A valid item described without its SKU will therefore be reported as ineligible.

• The start_return scenario for ORDER-1003 with a nonmatching item SKU does not require the correct postal code and is evaluated before the wrong-postal scenario. A caller supplying ORDER-1003, any wrong postal code, and a wrong SKU can therefore learn that SKU-RED-1 is returnable, violating the no-disclosure-on-verification-failure requirement.

Reproduction caveat: The blind reviewer claimed that a wrong-item return reveals SKU-RED-1 before postal verification. Executing the unchanged saved Python with ORDER-1003, postal 99999 and item SKU-NOT-ON-ORDER returned verification_failed with no SKU; the saved source includes postal_code=10003 in the wrong-item branch. Both follow-up privacy checks pass. Preserve the raw 13/16 model score, but do not treat this alleged runtime leak as established or use it uncritically to rank Sonnet.

• The lookup and action tools create an order-existence oracle: a known order number with the wrong postal code returns “verification_failed,” while an unknown number returns “not_found,” contradicting the instruction not to reveal whether the order exists.
• The actual schema and return behavior of the wired built-in end_session tool are not included, so its instructed arguments cannot be fully checked against the artifact.

Reproduction caveat: The platform end_session contract supports the authored reason and session_escalated arguments; the draft stores this builtin by reference. Its schema being absent from the artifact is missing reviewer context. This does not establish that a deployment-specific handoff works, and it does not resolve the separate verification-wording finding. The raw 14/16 review is retained.

• The cancellation flow does not explicitly require lookup_order before cancel_order, even though the agent is told not to call cancel_order for shipped orders and cannot otherwise reliably know shipment state.

• The lookup fixture provides no actual estimated arrival and gives most orders only the vague location “Carrier or destination sample location.”
• Sample order numbers, matching postal codes, and the return SKU are hidden in implementation source rather than documented as caller-usable fixtures.
• The cancellation flow does not clearly require a status lookup before applying “Do NOT use for shipped orders,” even though shipped state otherwise becomes known only through lookup or the prohibited cancellation call.
• “Before every call” unintentionally requires order verification before calling end_session, conflicting with ordinary goodbye handling.