code-executor¶

Status: Current | Last reviewed: 2026-09-21

Sandboxed Python runtime. Executes user-authored Python tools in isolation so a bad tool can't crash agent-host.

Source¶

apps/code-executor

Runtime¶

Aspect	Value
Language	Python 3.12
Sandbox	nsjail chroot + resource limits
Port	8010 locally, 8000 in-container

What it does¶

Receives Python tool and workflow code execution requests.
Resolves or builds the dependency venv under /venvs.
Spawns the worker inside nsjail with CPU, memory, and wall-time caps.
Loads the tool source, calls the entry function with the request payload.
Returns stdout + structured result.

Capability implementation tests send network_access: "none" in the execution request. This creates a separate nsjail network namespace and refuses to run without nsjail. The normal configured mode retains the existing egress setup. The shared runtime protocol, service and worker image must be updated together. Tests receive fixture variables and patched dependency returns, with no integration grants. See Capability testing.

Dependencies¶

Service	Purpose
`ai-manager`	Live Python tool syntax validation and workflow code test/warm calls
`tsf_runtime` shared package	Helpers exposed to user code
Docker	Local devcontainer runtime for the production code-executor image

Vanguard¶

overmind restart code-executor

Overmind supervises the local launcher, and .devcontainer/scripts/start-code-executor.sh builds the production image and runs it through Docker with the same service paths as production:

CODE_EXECUTOR_USE_DOCKER=true
CODE_EXECUTOR_USE_NSJAIL=true
CODE_EXECUTOR_VENVS_DIR=/venvs
CODE_EXECUTOR_WORKSPACE_DIR=/workspace-tmp
CODE_EXECUTOR_ROOTFS_DIR=/rootfs-minimal
CODE_EXECUTOR_NSJAIL_BIN=/usr/local/bin/nsjail

The Docker command uses a named volume for /venvs, a tmpfs for /workspace-tmp, a read-only root filesystem, and --security-opt seccomp=unconfined so nsjail can create namespaces under the local Docker runtime. CODE_EXECUTOR_USE_DOCKER=false is not supported for shared dev because it would bypass the production execution path.

The container has to be privileged, and a healthy service proves nothing¶

On Docker 29 — which is now every dev host, Codespaces on Debian 13 as well as the city EC2s on Ubuntu 24.04 — nsjail cannot create its mount namespace unprivileged. It dies before the worker starts:

[E] initCloneNs():391 mount('/', '/', NULL, MS_REC|MS_PRIVATE, NULL): Permission denied
[F] runChild():486 Launching child process failed

which the API reports as SandboxCrashed: sandbox exited rc=255 with no envelope. CODE_EXECUTOR_DOCKER_PRIVILEGED=true is therefore the dev default (env.local.template); the launcher then runs the container as root/privileged with a searchable /workspace-tmp, while nsjail still executes user code inside its chroot. Set it back to false only on a host where an unprivileged nsjail smoke test passes.

The reason this is worth its own heading: every health signal stays green. /healthz answers 200, the container is up in overmind status, and /v1/validate works — syntax checking runs in the service process and never touches nsjail, so the ai-manager Python editor keeps saying "Syntax OK". Only execution fails, so the symptom presents as "my tool doesn't work" with a service that looks perfect. Smoke-test it directly rather than trusting the health check:

curl -sS -X POST http://127.0.0.1:8010/v1/execute -H 'content-type: application/json' -d '{
  "execution_id":"probe","tenant_id":"00000000-0000-0000-0000-000000000001",
  "source":"async def run(args, context):\n    return {\"ok\": True}\n","entrypoint":"run"}'

{"status":"ok",...} means the sandbox works; a SandboxCrashed envelope carries the nsjail log in error.traceback, which is what tells you it is the host and not the code. The Test Run panel in the Tools editor renders that traceback for the same reason.

If the rebuild dies before the container starts with Docker credential-helper errors while resolving public images, set CODE_EXECUTOR_DOCKER_CONFIG_DIR=/tmp/tsf-code-executor-docker-config. The launcher creates a minimal config.json there and exports DOCKER_CONFIG for the build/run process.

All three callers — ai-manager, async-agent (code_executor_client.py), and ai-orchestrator (inline_python_tools.py) — default to CODE_EXECUTOR_URL=http://localhost:8010 when the environment does not override it, and the in-cluster Service listens on 8000, not 8010. That fallback is a trap in a cluster: the only chart that sets CODE_EXECUTOR_URL by default is charts/ai-manager. async-agent and ai-orchestrator do not — each deployment must set CODE_EXECUTOR_URL: "http://code-executor:8000" in its env, or every sandboxed Python tool call from those services fails with code-executor unreachable: All connection attempts failed (TransportError, duration_ms: 0) while the code-executor pod itself is perfectly healthy. The saved-tool "Test run" button in the ai-manager Tools editor routes ai-manager → async-agent → code-executor, so it surfaces the async-agent gap.

Inline Python outbound DNS and HTTPS (AS-272)¶

Inline-tool Test run calls ai-manager's /api/tools/inline/test, which forwards run(args, context) directly to code-executor's /v1/execute. A successful browser request does not establish that the jailed Python runtime can resolve or trust the same server.

Two rootfs prerequisites apply:

DNS: sandbox.py mounts the container's live /etc/resolv.conf and /etc/hosts read-only. Without these mounts the empty resolver file in the chroot reproduces [Errno -3] Temporary failure in name resolution. The DNS fix is already in commit c850de83; confirm the deployed image contains it before investigating DNS policies. Branch membership alone is not deployment evidence.
HTTPS: the Dockerfile copies the public /etc/ssl/certs directory into the rootfs, dereferences certificate symlinks, and creates OpenSSL's default /usr/lib/ssl lookup paths. Missing trust material or lookup paths otherwise leave ssl.create_default_context().get_ca_certs() empty. The image build now checks that this default trust store is nonempty. Without this fix, urllib fails with CERTIFICATE_VERIFY_FAILED after DNS succeeds. Rebuild and roll out the executor image; never work around this with disabled certificate verification. Private key directories are excluded.

Use the image smoke test for a real jailed execution. The regular unit suite runs without nsjail and cannot validate the image's resolver or CA files. /healthz also cannot verify outbound HTTPS. For a QA incident, correlate the deployed executor image and QA logs separately from any successful local smoke test.

Official Dev/QA verification — 2026-09-21¶

Read-only Grafana/Prometheus kube_pod_container_info queries confirmed that both official environments run executor tag 17dc6d5c25748ea5bde9cdad9a974ad59e2302f9, with image digest sha256:34ada3d709ea82644740f66a63d2e9ba4f640db4aa81cd5665184f644856fbed. Dev uses the trusrv-docker-scratch registry repository; QA uses trusrv-docker-staging. The image tag maps to the September 10 DNS-fix merge; its executor source matches the refreshed origin/dev and origin/qa trees. That revision includes resolver mounts but does not stage the HTTPS CA store or OpenSSL lookup paths described above. This establishes that the packaging fix applies to both deployed revisions; it is not an in-cluster execution test.

QA Loki showed /v1/execute requests on September 21, including 11:35:57 UTC. The access log's HTTP 200 does not establish tool success: this endpoint returns execution errors in a JSON envelope with HTTP 200 and does not log that envelope. A 72-hour search across executor and caller logs found no matching DNS/TLS error text. Dev Loki returned no app labels or matching log streams in the queried windows, so its empty results provide no execution evidence. Reproduce through an authenticated agent Test run and inspect the response body to confirm the specific current failure; retain the test timestamp and target agent URL.

An authenticated backend reproduction subsequently confirmed the live QA TLS failure at 2026-09-21 12:04:52 UTC. Diagnostic draft 4140, named AS-272 DNS and HTTPS diagnostic (qa), contains the saved as272_network_diagnostic inline Python tool. Its saved source was executed via POST /api/tools/inline/test with empty arguments and a verified request to https://example.com (no customer data). The result was:

{
  "trusted_ca_count": 0,
  "dns_ok": true,
  "https_error": {
    "type": "URLError",
    "message": "<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1010)>"
  }
}

The tool catches the exception for diagnostics, so the outer execution envelope has status: "ok" even though its HTTPS request failed. The draft remains unpublished for retesting after deployment. QA access used the existing AWS profile and documented application service token, held only in memory.

The same authenticated backend test confirmed the live Dev failure at 2026-09-21 12:07:02 UTC, using unpublished diagnostic draft 4159, named AS-272 DNS and HTTPS diagnostic (dev). Its saved tool returned the same result: DNS succeeded, the default trust store contained zero CAs, and verified HTTPS failed with CERTIFICATE_VERIFY_FAILED. Both environments therefore have a confirmed live TLS failure, beyond the matching-image evidence. The drafts can be reused for validation after the fixed executor image is deployed.

Staging verification — 2026-09-21¶

The same saved-source backend test reproduced the TLS failure in staging at 2026-09-21 12:13:49 UTC. Unpublished diagnostic draft 4036, named AS-272 DNS and HTTPS diagnostic (staging), returned dns_ok: true, trusted_ca_count: 0, and the same CERTIFICATE_VERIFY_FAILED error as Dev and QA. All three environments now have live execution evidence for the missing sandbox trust store.

The local staging AWS profile was configured for us-east-1. Staging retains legacy trusrv-prod-orch-* resource names; the application target was explicitly studio.stg.trueserve.deloitte.com. Direct public and private ALB HTTPS checks from the workspace timed out. A temporary SSM tunnel to the healthy Studio backend registered by the staging ALB allowed normal application-token authentication and draft/test API calls. The tunnels were closed after testing; no deployment, network policy, or cluster configuration was changed.

Production¶

Helm chart at charts/code-executor. Pods run with securityContext lockdown — no host network, read-only root filesystem, dropped capabilities. The optional egress proxy image is configured via egressProxy.image.*; production overlays should pin this to an approved immutable tag or digest from the image mirror. Enable gvisor.enabled=true only in clusters with a matching RuntimeClass and node pool; Kind/dev overlays keep it disabled.

nsjail needs a relaxed seccomp profile¶

nsjail builds each sandbox by creating user/mount/pid namespaces (clone(CLONE_NEWUSER|CLONE_NEWNS|…)). The chart default podSecurityContext.seccompProfile.type: RuntimeDefault blocks those clone flags, so on a real cluster the first execution fails with clone(...) failed: Operation not permitted → sandbox exited rc=255 with no envelope (SandboxCrashed). Locally the Docker launcher sidesteps this with --security-opt seccomp=unconfined; the chart has no equivalent by default. Each deployment that runs nsjail must relax the profile — set code-executor.podSecurityContext.seccompProfile.type: Unconfined in its values file (deep-merges over the chart default, so runAsNonRoot/runAsUser/fsGroup persist; nsjail still drops to uid 99999 for the user's code). Applied in values-kms-{dev,qa,prod}.yaml. If a mount error appears instead (mount('/', '/', … MS_REC|MS_PRIVATE): Operation not permitted), the node additionally requires CAP_SYS_ADMIN / allowPrivilegeEscalation: true or a gVisor RuntimeClass.