Skip to content

City dev environments

Status: Current | Last reviewed: 2026-10-06

Eight EC2-hosted Vanguard environments, one per city. Each has a city branch based on Vanguard plus a dedicated EC2 instance.

The cities

The city branch is the bare city name (rome, not city/rome — the old docs were wrong). A normal update rebases the city-local overlay commits onto environment/vanguard. A newly cloned city may instead begin with a frozen commit of the exact local Vanguard worktree when uncommitted local code must be reproduced. EC2 instances are reached via AWS SSM (no inbound SSH).

City Instance ID Branch PUBLIC_FQDN
rome i-046f91c16be786ecd rome *.rome.trueserve.ai
kyiv i-0811a65c936ce21e2 kyiv *.kyiv.trueserve.ai
istanbul i-02b6050ce87d79f33 istanbul *.istanbul.trueserve.ai
richmond i-088ae335f2b4162ac richmond *.richmond.trueserve.ai
macau i-00ec0a3baa10fa305 macau *.macau.trueserve.ai
shimla i-0c5f27581b883048b shimla *.shimla.trueserve.ai
seattle i-04b216eeca842d87c seattle *.seattle.trueserve.ai
athens i-0d921e0f42e8e1331 athens *.athens.trueserve.ai

Each is a stand-alone TrueServe install for a tenant or demo line. Instance IDs are also tagged Name=trueserve-dev-<city> in EC2.

City-specific topology

Cities use the two-stage EC2 boot, distinct from Vanguard's single-container EC2:

Stage Host What runs
Host Ubuntu Docker, trueserve-wake.service
Container devcontainer Overmind processes, infra/media compose

The host's trueserve-wake.service only runs infra/media/nginx — application services boot inside the devcontainer via wake-up.sh --mode=container. The devcontainer user is vscode (not codespace).

Public ingress uses nginx.org stable 1.30.3, installed by install-nginx-stable.sh. city-update.sh --step nginx upgrades an existing city independently; the restart, rebuild, and all paths reconcile the pinned version and the verification phase fails on version drift.

Per-city .env.local is gitignored and must be reconciled separately from code. Use scripts/ec2-dev/city-update.sh <city> --step env before a restart or rebuild; --step all runs it automatically before DB migrations and devcontainer rebuild. The step takes a mode-0600 backup and key-only manifest before changing the file, preserves unrelated keys, and fetches required OTP/payment SecureStrings directly on the instance. The validator inside .devcontainer/scripts/validate-env.sh enforces the public Janus, TURN, surface-app, code-server, and MCP URL patterns for DEV_ENV=ec2.

Topology details

The application services run as Overmind processes inside a devcontainer (e.g. cranky_brahmagupta, image vsc-trueserve-framework-…), labelled devcontainer.local_folder=/home/ubuntu/trueserve-framework. There is one shared checkout: /home/ubuntu/trueserve-framework on the host is bind-mounted at /workspaces/trueserve-framework in the container. Its .git/config, branches, index and worktree are shared, but the two accounts have separate home directories and credentials. Run host Git as ubuntu and devcontainer Git as vscode; SSM itself starts as root.

Cities carry connect-kvs-connector (Java) as an Overmind process. The devcontainer image includes a headless JDK and Maven so the launcher can build the fat JAR and keep the service running; if it is dead, check the pane for Java/Maven or stale-JAR failures rather than treating it as expected.

Krisp-enabled cities keep the shared SDK key in the per-city SecureString parameter /trueserve-dev/<city>/krisp_viva_api_key. Provisioning and city-update.sh <city> --step env install it atomically as the gitignored secret/KrispApiKey.txt with mode 0600 before ai-orchestrator starts; never pass the key through SSM Run Command text or commit it to a city branch.

scripts/trueserve-dev show <city> reports provisioned resource metadata but redacts every stored SSO client_secret. Read application credentials from their encrypted SSM parameters when an authorized recovery workflow genuinely needs them; command output and handoff notes must not expose them.

Login stays on Loading after a checkout change

A successful /login response and /api/health database check do not prove that the browser can start the app. Check a fresh browser for failed scripts and whether it requests /api/auth/session. If every script downloads but the session request never starts, inspect frontend dependencies and generated output on the affected city.

First confirm that the checkout is the intended city branch. Inspect git branch --show-current, git log -1 and git reflog -10 --date=iso-strict inside the city devcontainer. A recent merge date on the legacy vanguard branch does not establish that it contains the city's latest UI. Compare the expected design commit with git merge-base --is-ancestor <commit> HEAD. If the checkout was switched accidentally, restore the intended branch before reconciling dependencies; making the wrong branch boot does not restore the intended application.

Preserve local changes before switching, and stop the affected frontend processes while replacing their dependencies and generated output. Remove only a recovery symlink you created if the target branch restores that path as a tracked directory. Reinstall dependencies and regenerate caches for both ai-manager and surface-app when both checkouts changed.

Compare the intended branch's lockfile with the installed packages:

cd /workspaces/trueserve-framework/apps/ai-manager
node -e 'console.log("locked:", require("./package-lock.json").packages["node_modules/next"].version, "installed:", require("next/package.json").version)'
npm ls --depth=0

Before restarting, verify that the launch script retained by the running Overmind session still exists. A checkout change can move launchers between .devcontainer/scripts and .devcontainer/local/scripts while Overmind continues using the old path. If .devcontainer/scripts is absent and the checked-out launcher exists under .devcontainer/local/scripts, a temporary .devcontainer/scripts -> local/scripts symlink restores that path without restarting unrelated services. Reconcile the running Procfile with the checkout at the next planned stack restart.

For dependency drift, stop only ai-manager and run npm ci --prefer-offline --no-audit --no-fund from its directory. Preserve the generated .next/dev directory under a timestamped backup name inside .next while the service is stopped, then restart ai-manager to compile against the installed dependencies. Dependency drift alone does not justify changing branches or resetting the database.

If the locked wrtc install fails with node-pre-gyp: not found, supply the missing helper without changing the application manifest or lockfile: npm exec --yes --package=node-pre-gyp@0.17.0 -- npm ci --prefer-offline --no-audit --no-fund.

Verify /api/health, the expected unauthenticated /api/auth/session response, and a fresh browser reaching the configured sign-in flow. An HTTP 200 containing the loading screen is insufficient.

Browser media endpoints

City browsers reach Janus through the nginx TLS vhost, not the raw Janus listener. The ai-manager runtime env must therefore advertise JANUS_WS_URL=wss://janus.<city>.trueserve.ai/ws and NEXT_PUBLIC_JANUS_WS_URL with the same value. Raw wss://<city>:8188 can be healthy on the box and still fail externally.

TURN should likewise use public DNS (turn.<city>.trueserve.ai) with TURN_TYPE=tcp, FORCE_TURN_ONLY=true, and ALLOW_STATIC_TURN_CREDENTIALS=true for the local dev Node server. The private EC2 ENI IP belongs in Janus NAT mapping (HOST_IP / LOCAL_IP), not in the browser-facing TURN server fields. If Janus works locally but browser calls fail, compare the live ai-manager process env (tr '\0' '\n' < /proc/<pid>/environ) against those values; a file edit alone is not enough until ai-manager is restarted.

The other browser-facing city URLs follow the same public-DNS rule:

Env var Pattern
NEXT_PUBLIC_SURFACE_APP_URL, SURFACE_APP_URL, SURFACE_APP_BASE_URL https://surface-app.<city>.trueserve.ai
CONVERSATION_API_PUBLIC_URL https://conversation-api.<city>.trueserve.ai
CES_MCP_BASE_URL, AGENT_HOST_MCP_PUBLIC_URL https://tools.<city>.trueserve.ai/mcp
CODE_SERVER_URL https://code.<city>.trueserve.ai
WHISKER_PUBLIC_WS_URL wss://code.<city>.trueserve.ai/whisker-ws

The CRM dashboard launcher pins its runtime PORT to CRM_DASHBOARD_PORT (3003 by default), matching the nginx city vhost. It must not inherit a generic devcontainer/Codespaces PORT, which may be 6800 and would leave dashboard.<city>.trueserve.ai returning 502.

Standing up a new city

  1. Create and push the bare city branch, including charts/trueserve-stack/values-branch-<city>.yaml. If the source worktree is dirty, freeze the exact worktree into the city branch without moving the developer's branch or index.
  2. Refresh the Vanguard state snapshot with scripts/trueserve-dev snapshot-vanguard.
  3. Run scripts/trueserve-dev create <city> --parent-zone trueserve-ai --volume-size 1000 --with-genesys --with-dialplan --with-sso. The command creates the EC2 instance, encrypted volume, Global Accelerator, ACM certificate, Route53 apex and wildcard records, TLS secret, SSM identity, read-only deploy key, Genesys routing, and Authentik clients.
  4. First boot clones the city branch and hydrates the Git LFS assets before setup. Restore the current Vanguard snapshot, then immediately reconcile the city env so source-box public URLs cannot leak into the new tenant.
  5. Run scripts/ec2-dev/city-update.sh <city> --step db. It quiesces writers, restore-tests a custom logical dump, takes a completed encrypted EBS snapshot while PostgreSQL is stopped, and only then migrates. Install/enable the host and devcontainer systemd units, and finish with --step verify plus external HTTPS and Global Accelerator health checks.

Warning: the Genesys/BYOC trunk has a three-way constraint — Global Accelerator TCP :5065, the per-city Kamailio cert, and the trunk ACL. Mismatch on any one yields 404 Not Found on transfer.

Genesys BYOC wiring

See SIP integrations for the protocol; the city-specific notes:

  • Each city's Kamailio cert is unique and registered with Genesys per-trunk.
  • Genesys-side trunk routing keys on the Global Accelerator anycast IP plus city-specific port.
  • Transfers fail silently (404) if the city's cert isn't trusted.

Read-only deploy keys

Provisioning creates a read-only deploy key (~/.ssh/trueserve_deploy, owned by ubuntu) for unattended host clone/fetch operations. That key cannot push. Developers can push from a city devcontainer using their own authorized GitHub account over HTTPS or a personal SSH identity. Write permission belongs to the authenticated identity; choosing SSH or HTTPS alone does not grant it.

City checkouts can contain unpublished commits and local edits. Use the preserve-work update path when integrating Vanguard into an active development checkout.

Host automation and the GitHub org migration: city-update.sh sets the host's origin fetch URL to git@github.com:Deloitte-US/trueserve-framework.git, where its deploy key authenticates. The earlier HTTPS warning came from a rollout whose host account had no HTTPS credentials: fetch failed, and a later reset could use stale refs. Keep unattended host fetches on SSH unless host HTTPS credentials are separately configured and verified. Check git ls-remote origin as ubuntu and stop on fetch failure. This host requirement does not prohibit authenticated HTTPS Git inside a developer's devcontainer.

Git access inside the city devcontainer

Host-side Git runs as ubuntu; an editor terminal inside the devcontainer runs as vscode. Matching UID 1000 gives both accounts access to the checkout, but it does not share their home directories or SSH configuration. A successful host fetch does not prove that the editor can authenticate to GitHub.

If the editor reports Permission denied (publickey), check /home/vscode/.ssh/config and the identity it selects. Recover read access by copying the existing host /home/ubuntu/.ssh/trueserve_deploy into the running container at /home/vscode/.ssh/trueserve_deploy, keeping owner vscode and mode 0600. Configure that account's Host github.com entry to select IdentityFile ~/.ssh/trueserve_deploy with IdentitiesOnly yes. Preserve existing identities and known-host entries; never put the private key in Git, command arguments, or logs. Confirm the fix by running git fetch inside the devcontainer as vscode.

This restores the deploy key's existing read access only. A push dry-run still fails with “The key you are authenticating with has been marked as read only.” when that key is selected. Publishing requires the developer's own authorized GitHub identity or an explicitly provisioned write-capable repository key; do not silently copy another environment's personal token.

HTTPS for interactive pushes

Authenticate the developer's own GitHub account inside the devcontainer, using the editor's GitHub integration or GitHub CLI. With GitHub CLI installed, gh auth login --hostname github.com --git-protocol https followed by gh auth setup-git --hostname github.com configures that account's Git credential helper. Check gh auth status --hostname github.com there.

The host and container share remote URLs through .git/config. To keep unattended host fetches working, leave the SSH fetch URL in place and configure an HTTPS push URL after personal authentication is ready:

# Run inside the city devcontainer as the developer.
git remote set-url --push origin https://github.com/Deloitte-US/trueserve-framework.git
git remote get-url origin          # SSH fetch URL for the host deploy key
git remote get-url --push origin   # HTTPS push URL for the developer's login
git push --dry-run --no-verify origin HEAD

Alternatively, use an explicit HTTPS URL for an individual push without changing repository configuration. The read-only deploy key can remain in place. A GitHub login or SSH agent available in an editor terminal may be absent from docker exec or SSM; test the actual terminal the developer uses. An unattended authentication failure does not establish that editor-session credentials are broken.

A key copied into an unmounted container home survives a restart but not container recreation. Repeat this recovery after a rebuild, or arrange SSH agent forwarding through the developer's normal editor connection. Do not reset the city branch while repairing authentication; its unpublished commits and local edits are independent of the credential failure.

Per-city pgadmin gotcha

If you manually docker compose up -d --force-recreate pgadmin, you must source .env.local first or compose substitutes empty strings into the OIDC config and pgadmin errors with "public client". Bit me twice during 2026-05-19 OIDC URL fix.

See also