Skip to content

Tool testing

Status: Current | Last reviewed: 2026-09-21

Tool mocks are reusable tool contracts, not instructions asking a judge to invent tool results. Library tools store append-only revisions in tool_mock_revisions. An agent-local tool stores output_schema and testing on its inline definition. Both use the same version-1 data contract and are pinned when a simulation starts.

GET /api/tools/:id/testing returns operation contracts, current hashes, latest revisions and stale indicators. POST accepts action: generate or action: save, operation_id, and optional generation instructions. Save also accepts contract_hash and definition. A changed contract returns 409. The permission is AIManager:ManageTools; tenant checks apply to definitions and revisions. POST /api/tools/testing checks or generates drafts for inline tools without creating a library row.

An operation needs an input and output JSON Schema. MCP operations use their discovered schemas; platform operations use their expanded runtime names. Description, operation identity and input/output schemas participate in the contract hash. A library tool's output schema is the fallback when discovery did not provide one. Schema-derived defaults are saved only when their examples validate; unsupported schemas need an authored mock.

Platform parameter normalization matches async-agent: sfdc_field_mapping becomes an object, async_options becomes a string, option values become enums, and both operation-level and field-level required flags are retained. A schema correction changes the contract hash; regenerate checked examples and save a new revision before starting new tests. Do not relabel an old fixture as current. Salesforce responses retain the adapter's native {"status":200,"data":...} envelope (201 for created records/events). Returning bare Salesforce JSON can pass an overly broad object schema while silently breaking workflow output paths such as tool.data.records[0].FirstName.

Example definition for a tool returning { "reference": string }:

{
  "schema_version": 1,
  "default_variant": "success",
  "initial_state": { "reference": "TEST-742" },
  "variants": {
    "success": {
      "result": { "reference": { "$state": "reference" } },
      "set_state": { "booked": true },
      "delay_ms": 0,
      "rules": []
    }
  },
  "examples": [
    {
      "variant": "success",
      "input": {},
      "expected": { "reference": "TEST-742" }
    }
  ]
}

Rules select the first matching case. Each when condition has source (input or state), a dot-separated path, op (eq, ne, in, exists) and value. A rule supplies result, set_state and delay_ms; otherwise the variant's default applies. {"$input":"path"} and {"$state":"path"} preserve JSON types. Missing references fail. All references in one call read its pre-call state; updates become visible to the next call. State is isolated per attempt and updated under a database row lock, including calls from nested workflows.

Two bounded expressions support coherent records without code execution:

  • {"$number":{"$input":"amount_dollars"}} converts a finite decimal string or number to a JSON number. Booleans, empty strings, hexadecimal, exponent strings and non-finite values are rejected.
  • {"$dateAdd":{"value":{"$input":"from_date"},"days":7,"hours":16}} returns an ISO UTC timestamp seven days and sixteen hours after the supplied ISO date's midnight. Integer offsets are bounded to ±3660 days and ±24 hours. The evaluator never reads the wall clock. Dates must remain in years 1–9999.

Every variant needs a checked example. Input examples and rendered results are schema-validated. Definitions are limited to 100 KB, 30 variants, 100 rules per variant, 100 examples and 30 seconds simulated delay. Definitions cannot execute code, templates, HTTP requests or model calls.

At runtime the real tool dispatcher validates inputs and applies the pinned mock before external execution. Missing-input repair goes through that same dispatcher. The agent receives the declared result without a synthetic success wrapper; evidence separately records tool, operation, revision, variant, arguments and result. Nested services authenticate the attempt capability and share its fixture state. Unknown external tools fail closed.

Invalid arguments to a valid pinned contract return an ordinary tool error and persist tool_input_errors; the real agent can recover, and the conversation is judged on that behavior. Invalid caller values supplied to bounded numeric or date expressions use the same path. Missing definitions/references, invalid fixture literals, output-schema violations and expired leases remain harness_errors, even if a nested service catches the exception. These must never become passing agent results.

The HealthPlus example's reusable fixture source is apps/ai-manager/lib/performance/examples/healthplus-fixtures.ts. Its tests exercise coherent lookup/write chains, current-date-relative appointments, exact settlement amounts, no-match/retry state, optional adverse-event fields, and parity with the Python runtime. The corresponding seed script updates capability-owned definitions and Salesforce operation revisions without changing the agent's prompts or prerequisite guards. See the qualification procedure.

Workflow control flow can remain real. Direct effect nodes that bypass tool dispatch must be replaced by tool nodes for sandbox tests. An explicit real-tools integration run retains the actual external handlers.

See Agent evaluation and Operating evaluation workers.