Vanguard on EC2

Status: Current | Last reviewed: 2026-09-17

A long-lived EC2 instance running the same devcontainer. Used for shared demos and the city dev environments.

Topology

EC2 host (Debian)
  └── Docker (DinD inside the devcontainer? no — host Docker)
       └── ONE devcontainer container (running everything)
            ├── Overmind processes (apps/*)
            ├── docker-compose-managed infra containers
            ├── docker-compose-managed media containers
            └── nginx host for public HTTPS

This is not the city two-stage Ubuntu/ubuntu setup. It's a single Debian/admin host with one privileged devcontainer that runs everything inside it.

Trying a second VPN

The devcontainer shares the host network. Running a VPN client there can replace the host's routes and DNS, interrupting VPC access and running services. A distinct tunnel interface name alone does not isolate those changes.

Run experimental VPN connections in a separate Docker container using bridge networking, its own resolver file, NET_ADMIN, and /dev/net/tun. Do not use host networking or mount the host resolver file into that container. Run commands that need the second VPN inside the container; existing host applications retain their original network context. Compare host routes and resolver contents before and after connecting.

For browser-based SAML sign-in, keep the browser in the isolated environment and expose its remote desktop only on a loopback port through an authenticated SSH or private editor port forward. Enter credentials and MFA in that browser. Keep authentication logs private because VPN helpers can print session cookies. If the portal offers multiple gateways, pass --authgroup="<gateway label>" to OpenConnect through the SAML helper. Otherwise it can stop at a terminal selection prompt after successful browser sign-in. Keep the remote desktop running when the VPN process exits so the failure can be diagnosed. A retry may require fresh browser authentication; do not assume the original sign-in cookie is reusable. The selected gateway may also require its own SAML sign-in. If portal authentication does not carry over, use gp-saml-gui --gateway with a gateway hostname returned by the authenticated portal. This authenticates directly to that gateway and avoids repeating portal discovery on each attempt. An established ESP session and assigned tunnel address do not prove usable access. Check the VPN-provided DNS server and an intended destination from inside the container. Record any gateway HIP/device-posture requirement separately from transport failures; a HIP warning alone does not establish why traffic timed out.

Temporary WireGuard relay test

For the maintained Mac/Windows client scripts and city-host setup, use Laptop access to Dev and QA. The notes below record the original proof of concept. Dev and QA relay network paths now pass after the scoped QA SG/WAF allowlist updates recorded in that guide; native QA browser/audio acceptance remains.

The optional local studio-wireguard container relays selected Studio traffic through the EC2 host's existing connection. It uses a separate network namespace, container-local forwarding/NAT, and an authenticated WireGuard listener published only at 10.120.206.51:51827/udp. It does not use the experimental GlobalProtect container as its upstream. No AWS security-group change is part of this setup.

The machine-specific Mac bundle is .local-data/network-tests/wireguard/StudioTest-mac.zip (git-ignored; contains a private client key). Import StudioTest.conf in WireGuard while keeping the existing VPC connection to the EC2 endpoint. Confirm a recent handshake before running the included sudo bash studio-dns.sh enable on the Mac. The helper adds only the named endpoints to /etc/resolver; it preserves existing unrelated resolvers. Run sudo bash studio-dns.sh disable before deactivating the tunnel. Stop the server with docker stop studio-wireguard.

The profile routes a snapshot of the Studio, SSO, signalling and TURN IPs, plus the tunnel DNS address; it has no default route or global DNS override. Routes and the container's pinned DNS answers must be refreshed when target IPs change. Direct WebRTC ICE candidate addresses outside that list still use the original client route and may require additional explicitly scoped routes after diagnosis. Do not broaden the profile to all private networks just to cover unknown media addresses.

The 2026-09-17 test used a second isolated WireGuard client and passed DNS, Studio HTTPS (200), Janus WSS (101), STUN/UDP 3478, and certificate-verified TURN/TLS 5349. Those checks do not establish Mac-to-EC2 UDP reachability, TURN allocation, or two-way audio. A missing Mac handshake requires checking the VPN/firewall path to UDP 51827; SSH and VS Code TCP forwarding do not transport WireGuard's UDP.

On macOS, Homebrew wg-quick can establish a handshake while printing File exists for every route: an earlier utun interface may still own the exact destination routes. Compare the active WireGuard interface from wg show with route -n get 10.254.253.249 and a Studio IP. Correct only the test's destination routes to the active interface; leave the EC2 endpoint and core VPC routes alone. In the 2026-09-17 Mac test, moving the test routes from stale utun8 to active utun9 restored DNS over WireGuard (two A records, 31 ms). Interface numbers are session-specific. The downloadable DNS helper now refuses to enable scoped DNS unless a direct query to the tunnel DNS server succeeds.

After correcting those routes, the Mac user confirmed Studio sign-in and a voice call with audio in both directions. Gateway counters also recorded traffic to Studio, SSO and TURN destinations. The selected ICE candidate pair was not captured, so this establishes functional end-to-end success without proving that every media packet used the relay. EC2 host routes and resolver contents remained unchanged. The setup remains a temporary, manually managed test.

Reboot recovery

The instance survives stop/start because:

• restart=unless-stopped on the devcontainer.
• A systemd unit trueserve-devcontainer-wake.service re-runs wake-up.sh on boot. (Added 2026-05-18.)
• wake-up.sh brings up infra/media compose, nginx, and the asterisk health check is non-fatal so a slow ARI doesn't block the rest.

Two boot-recovery failure modes were fixed 2026-07-17 (both had silently broken recovery on the 07-16 and 07-17 reboots):

• The unit matched the container only by the devcontainer.local_folder label. A container created straight from the universal image (this box) carries no such label, so ExecStartPre timed out every boot. The unit now matches the well-known name trueserve-devcontainer first, label as fallback.
• wake-up.sh exited at env validation because this box's canonical dev Postgres password is the literal REPLACE_ME_POSTGRES_PASSWORD, which validate-env.sh rejected as an unfilled placeholder. Resolved in validate-env.sh itself: that exact value now produces a warning instead of a failure (it IS the value docker-compose.infra.yml hardcodes for initdb, so a matching .env.local is correct; it is localhost-only). Any other placeholder-looking password still fails. No SKIP_ENV_VALIDATE bypass is needed anywhere.

Continuous watchdog (not just boot)

Boot-time recovery alone cannot heal a stack that dies after boot (OOM, overmind crash, container stop). A host systemd timer, trueserve-stack-watchdog.timer (every 3 min), runs trueserve-stack-watchdog.sh which recovers each layer in order:

1. devcontainer not running → docker start it.
2. Overmind socket dead or ai-manager :3001 unresponsive (host-network) → re-run wake-up.sh --mode=container inside the container (detached, flock-guarded so runs never stack).
3. Stack healthy → still re-run the per-developer personal boot hook .local-data/wake-up-local.sh (idempotent) so personal long-running processes recover even when the stack never went down.

Install/update on the host:

cd /home/admin/trueserve-framework/scripts/ec2-dev
sudo install -m755 trueserve-stack-watchdog.sh /usr/local/bin/
sudo install -m644 trueserve-stack-watchdog.{service,timer} trueserve-devcontainer-wake.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now trueserve-stack-watchdog.timer

Observe: journalctl -u trueserve-stack-watchdog.service -n 20.

Bootstrapping

bash scripts/bootstrap-ec2-dev.sh

This script:

1. Provisions the EC2 instance with the right AMI.
2. Installs Docker.
3. Writes /etc/systemd/system/trueserve-devcontainer-wake.service.
4. Pulls the devcontainer image and starts it.

See scripts/ec2-dev/ for the cloud-init artifacts.

Devcontainer entrypoint

Open the repo through the root .devcontainer/devcontainer.json. The local Overmind/Codespaces stack now lives directly under .devcontainer/: Dockerfile, Compose files, Procfile, setup.sh, and wake-up.sh. The former .devcontainer/local/ tree is compatibility shims only, with no devcontainer.json, so VS Code cannot accidentally attach through a second or Kind-era configuration.

Port forwarding should come from the devcontainer's explicit forwardPorts list. Workspace settings disable VS Code's listener auto-forwarding so the Ports panel does not drift between Vanguard and city boxes based on whatever temporary service ports happen to be listening.

Memory tuning

Daily hangs on this host were tracked to no-swap + uncapped devcontainer memory. The fix (2026-05-20):

• 32G swap file.
• vm.swappiness=10, vm.vfs_cache_pressure=50.
• 48G memory cap on the devcontainer.
• CloudWatch agent + 4 alarms (CPU, mem, disk, swap).

Pre-baked into scripts/bootstrap-ec2-dev.sh.

Public ingress

Nginx fronts *.vanguard.dev.trueserve.deloitte.com:

Subdomain	Service
aim	ai-manager
surface	surface-app
status	status-page
tools	agent-host MCP (:8003)
genesys	genesys-connector-go
code	code-server
orca	Optional personal Orca headless runtime (wss, nginx to :6768)
events	async-agent workflows API (:8000, SSE unbuffered)

The ws-events. vhost and its ws-events-proxy (:3004) were RETIRED with the native WebSocket transport (2026-09-01): TSF_EVENT_TRANSPORT=ws now points browsers at same-origin /ws/* paths, which the ai-manager. and surface-app. vhosts route to conversation-api (:8080) — and /ws/executions on the ai-manager host to async-agent (:8000). See conversation-api § WebSocket event streaming.

Source of truth: configs/nginx/trueserve-vanguard.conf.template. wake-up.sh reconciles changes on boot.

Scanner / bot-spam drop

The public host is constantly probed by untargeted botnet scanners (.env, /wp-*, /cgi-bin, exposed *.sql/*.conf, SQLi query strings, …). None reach anything sensitive (our routes 404 or redirect to /login), but they flood logs and add load. The web-facing server blocks (ai-manager, surface-app, dashboard, apex) include snippets/scanner-drop.conf, which return 444 (drop the connection, no response) when the request matches the $is_scanner_path map defined at the top of the config — BEFORE the request reaches the app. They also limit_req zone=perip (20 r/s, burst 40) to throttle scan floods. The snippet is staged to /etc/nginx/snippets/scanner-drop.conf by wake-up.sh; the map + limit_req_zone live in the template's top-level config block. Patterns are conservative — none match a real TrueServe route (/api/..., /agents, /workflows, embed/collab/A2UI). Add a pattern → edit the map in the template; the change applies on the next wake-up.sh nginx reconcile.

Operator note: on this box the live nginx process tree can be confusing — the code-server auth-proxy is a separate containerized nginx (binds :8081) and /run/nginx.pid may point at it, so a bare nginx -s reload can signal the wrong master or spawn an orphan. Prefer service nginx reload (or let wake-up.sh reconcile) and nginx -t to validate before reloading.

Optional Orca headless runtime

On a Vanguard box with the personal Orca runtime installed, it runs as the unprivileged codespace user inside the host-network devcontainer. Its pinned AppImage, extracted runtime, config, and logs live under the gitignored .local-data/orca/ tree. The personal boot hook starts it in the orca-vanguard tmux session; the host watchdog re-runs that hook every three minutes.

# Inspect the process and its redacted pairing state
bash .local-data/orca/status.sh

# Print the one-time desktop/server pairing URL
bash .local-data/orca/pairing-url.sh

# Restart with a mobile-scoped pairing link, then print that link
bash .local-data/orca/restart.sh mobile
bash .local-data/orca/pairing-url.sh

# Return future restarts to normal remote-runtime pairing
bash .local-data/orca/restart.sh runtime

The runtime binds local port 6768. Nginx publishes only its WebSocket route as wss://orca.vanguard.dev.trueserve.deloitte.com/runtime through the existing Global Accelerator TCP/443 listener. The Orca vhost disables nginx access logs because pairing URLs and client handshakes can contain credentials. Treat every pairing URL like a password and revoke an unintended grant from Orca's Remote Orca Servers settings.

Headless mode does not auto-update. Pin and checksum each replacement AppImage, extract it without FUSE, and update .local-data/orca/run.sh only after the new release passes a foreground smoke test. Keep ELECTRON_RUN_AS_NODE unset when launching Orca; the devcontainer inherits that variable from VS Code and Electron otherwise starts in Node compatibility mode.

See also

• Vanguard on Codespaces
• City dev environments
• Update flow