Skip to content

Update flow

Status: Current | Last reviewed: 2026-09-30

How an environment/vanguard commit reaches all eight city EC2s.

The model

  • environment/vanguard is the upstream branch.
  • <city> (bare name — rome, richmond, … NOT city/rome) is each city's branch. Most are environment/vanguard plus city-local overlay commits; a city created from a dirty Vanguard workspace can start from a frozen commit of that exact workspace and then carry the same overlay/runtime conventions.
  • City checkouts can contain unpublished commits and in-progress edits. Inspect the live branch, index, worktree and untracked files before updating it. A reset-based rollout requires an explicit decision that the existing checkout is disposable.
  • Cities are reached via AWS SSM, not SSH. See instance IDs in city-dev-environments.md.

Host automation fetches with the city's read-only SSH deploy key. Interactive developers can use their own write-authorized GitHub login over HTTPS or a personal SSH identity inside the devcontainer. These are separate credential contexts even though they share one checkout. See city Git authentication for configuring developer pushes while retaining SSH host fetches.

Driver script

scripts/ec2-dev/city-update.sh encapsulates the per-city remote steps with the SSM safety rules baked in (base64 ferrying, stderr surfacing, sudo -u ubuntu). The branch rebase happens locally first.

Preserve city work when merging Vanguard

Use this path when the city has work you need to retain. The driver's code and all phases run reset --hard; they are unsuitable for this case.

  1. Inspect the live city checkout through SSM. Record its HEAD, branch, staged and unstaged changes, untracked files and any active Git operation.
  2. Save a backup branch and a private backup outside the checkout containing Git metadata, tracked and untracked source, the index, binary patches and local configuration. Verify the backup before changing source files.
  3. Fetch environment/vanguard, pin the committed revision, and prepare a normal merge in a separate worktree based on the city's actual HEAD. Resolve conflicts to retain both branches' behavior and run the affected lint and test checks. Keep both histories as parents of the merge commit.
  4. Confirm the live city's HEAD and edits still match the saved state. Fast-forward its branch to the tested merge while retaining local edits. If an edit overlaps incoming files, preserve and reapply it separately; never discard it to make the merge succeed.
  5. Verify both original heads are ancestors, local edits retain their contents and staging state, and no conflict remains. Keep recovery files until the owner has verified the result.

Code integration does not migrate the database or rebuild the running stack. Treat those as separate operational steps, with their own backups and checks.

Preserve .devcontainer/docker-public/ as private city runtime state when merging older city branches that tracked it. Back up the directory before fast-forwarding such a checkout and restore its contents afterward; the shared source now ignores Docker client identities, lock files and BuildKit activity. Do not copy another city's generated identity into the target.

# 1. Locally: rebase the city branch onto the new upstream, force-push.
git fetch origin
git checkout -B rome origin/rome
git rebase environment/vanguard          # replays the 2 city-local commits
git push --force-with-lease origin rome

# 2. Remotely (per city): migrate remote, pull code, reconcile env, migrate DB, rebuild, verify.
scripts/ec2-dev/city-update.sh rome --step all
# or a single phase: --step remote|code|env|db|db-backup|db-snapshot|db-migrate|nginx|restart|rebuild|verify

The nginx genesys-vhost commit conflicts with the upstream apex/docs/developer vhosts on every city (both add server blocks at the same spot). Resolution is identical for all cities: keep all four vhosts. Copy a known-good resolved trueserve-vanguard.conf.template (e.g. git show richmond:<path>) onto the conflicted file and --continue.

Remote steps (what city-update.sh does)

  1. remote — org migration for unattended host fetches. Set origin to git@github.com:Deloitte-US/trueserve-framework.git, using the host's read-only deploy key, and verify with ls-remote. The earlier HTTPS rollout failure occurred because the host account had no HTTPS credentials; it was not a restriction on authenticated developer HTTPS usage. An explicit HTTPS push URL can coexist with the SSH fetch URL. This step changes the shared checkout's fetch URL, so inspect both URLs when diagnosing editor Git access.
  2. code — installs Git LFS when an older city host does not have it, configures the repository's local LFS filters, runs git fetch origin && git reset --hard origin/<city>, hydrates the branch's LFS objects, and requires git lfs fsck to pass before asserting HEAD == origin/<city> (guards against both stale refs and proprietary runtime archives remaining as pointer text). Git operations run as sudo -u ubuntu against /home/ubuntu/trueserve-framework.
  3. env — first copies the existing gitignored .env.local to /home/ubuntu/trueserve-env-backups/<city>-<timestamp>.env.local with mode 0600, records its hash and a key-only manifest, then reconciles city values. It preserves unrelated keys and fails if any are removed unexpectedly. Payment-vault URLs and safe provider gates are explicit. Shared service ports are also reconciled: the Connect SCV translator uses 8101 for webpack and 3031 for Express, leaving 8094 exclusively to mkdocs-developer; a legacy SCV_TRANSLATOR_CLIENT_PORT=8094 makes whichever service starts second fail. The OTP pepper, two payment-vault tokens, and Krisp contracted key are fetched on the instance from encrypted /trueserve-dev/<city>/... SSM parameters, so secret values never enter Send Command text or output. The Krisp key is atomically installed as the gitignored mode-0600 secret/KrispApiKey.txt rather than added to .env.local. Missing, malformed, duplicate, or empty required values fail the rollout. It also reconciles the browser event-stream transport (ensure_env TSF_EVENT_TRANSPORT sse — per-city flips to ws survive re-runs) and deletes retired vocabulary: the ws-events-proxy keys (NEXT_PUBLIC_EVENT_TRANSPORT, NEXT_PUBLIC_WS_EVENTS_URL, WS_EVENTS_PORT) and the MINIO_* keys (no reader at HEAD — cities ride the shared agent-studio-local-* S3 buckets through the instance role; lib-env.sh defaults every STORAGE_* value).
  4. db — a guarded three-part migration. db-backup first stops the complete application supervisor and Kamailio, then cleanly restarts PostgreSQL to clear every stale client pool without requiring a database superuser. With application reconnects disabled, it writes a mode-0600 custom-format dump under /home/ubuntu/trueserve-db-backups/ and validates its SHA-256/archive listing. The dump is restored into a disposable, network-unpublished PG15+pgvector container whose PostgreSQL major version must match the source; its temporary local admin can recreate extensions without granting the city application role extra privileges. Core row counts must match before the restore container and its anonymous volume are removed. The source PostgreSQL container is then stopped. db-snapshot confirms PostgreSQL is stopped, verifies the root EBS volume is encrypted, creates and waits for an encrypted snapshot, and records its ID. If either pre-migration guard fails, the combined db step restarts PostgreSQL and the container services before returning the failure. Only after both guards pass does db-migrate restart PostgreSQL and run scripts/schema-audit/migrate-city-db.sh; core counts, required RLS, security triggers, columns, builtin templates, and removal of the legacy briefing table must all pass. Never run db-init-local.sh on a city — it DROP DATABASEs and wipes tenant data. Running --step db performs all three substeps in order; the split steps exist for recovery and inspection, not for skipping a prerequisite.
  5. rebuild — raises host inotify limits for the devcontainer (fs.inotify.max_user_watches, fs.inotify.max_user_instances), reconciles the host to nginx.org stable 1.30.3, rebuilds repo-owned compose images (code-server, asterisk, rtpengine), restarts the host trueserve-wake.service for infra/media/nginx, removes the old devcontainer, clears persisted Next.js dev caches (apps/*/.next for the web apps), and runs devcontainer up --workspace-folder /home/ubuntu/trueserve-framework against the root .devcontainer/devcontainer.json. The nginx installer supports both nginx.org's conf.d layout and a package-preserved Ubuntu nginx.conf; it enables sites-enabled/* exactly once so repeated upgrades remain valid. The devcontainer postStartCommand runs wake-up.sh, so Overmind starts from the same Procfile and launchers used locally. Use nginx when only the pinned host ingress package needs reconciliation.
  6. verify — HEAD matches, every Overmind process is running, the sandbox child containers are ready, and application health routes respond. Payment-vault must answer locally and at https://payment-vault.<city>.trueserve.ai/health; the verifier asserts fake capture is disabled, internal/frame credentials are configured, the frame URL and outcome reporting are active, and the credential pool is usable. It also checks ai-manager's auth route handlers directly: /api/auth/session must return 401 unauthenticated JSON and /api/auth/callback?code=probe&state= must return 307 to /login.

restart is still available as a lighter single phase when you only need to bounce the host service and re-run wake-up.sh --mode=container inside an existing devcontainer. It also raises the host inotify limits, stops the web Next.js dev processes, and clears their .next caches before wake-up. Use rebuild after devcontainer config, port-forwarding, Dockerfile, or setup changes. Fresh image rebuilds can run several minutes on a cold city box; the script's SSM wait can be tuned with CITY_UPDATE_SSM_POLLS and CITY_UPDATE_SSM_SLEEP_SECONDS, and cold sandbox child image waits with CITY_UPDATE_SANDBOX_WAIT_POLLS. The Next.js dev apps must keep allowedDevOrigins broad enough for both **.dev.trueserve.deloitte.com and the city public hosts **.trueserve.ai; otherwise Next 16 rejects HMR/client refresh with 403s behind nginx. Host inotify limits must be set from the EC2 host (SSM runs as root); setting them from an unprivileged devcontainer launcher can fail silently and leave Turbopack with too few watchers. Stale or partially initialized Next dev caches can register only some API routes, so the city driver clears them before web app restart/rebuild and verifies /api/auth/* explicitly.

Don't run setup.sh on the host

wake-up.sh --mode=container's stale-venv detection rebuilds Python venvs in-container. Running setup.sh on the host duplicates that work and can leave the host venv ahead of the container's.

The webex-connector-go build target is ./cmd/webex-connector — not ./... (the latter pulls unrelated test packages and breaks).

Venv rebuilds depend on complete requirements.txt

wake-up.sh prunes stale venvs and setup.sh rebuilds them with pip install -r requirements.txt or, for ai-orchestrator, uv sync --frozen from the locked project file. The devcontainer image installs uv; after changing the image, run the rebuild step rather than only restarting an existing container. If a service imports a package that's used but not declared in pyproject.toml, uv export drops it from requirements.txt and the rebuilt venv 500s at import (ModuleNotFoundError). This bit conversation-api (asyncpg, minio, ...) during the 2026-06-07 rollout. Fix is always: declare the dep in pyproject.toml, uv lock, re-export requirements.txt. Don't hand-patch the venv on the box — the next wake undoes it. Managed Python app venvs must also have .venv-ready and an executable interpreter; a venv without either is treated as incomplete and rebuilt so host-only uv interpreter paths cannot leak into the devcontainer and dev-only helpers such as watchfiles are present. A few Overmind launchers also self-heal partial venvs at start (conversation-api, multimodal-service), the conversation-api API/worker pair uses .venv.lock around their shared .venv, and the mkdocs launchers use separate .venv-mkdocs-* directories so their concurrent installs do not corrupt a shared repo-root venv.

SSM quoting

SSM mangles nested quotes/$ in the commands array. base64-encode every script:

B64=$(printf '%s' "$SCRIPT" | base64 -w0)
aws ssm send-command --instance-ids <iid> --document-name AWS-RunShellScript \
  --parameters commands="echo $B64 | base64 -d | bash"

SSM Status: Success does not equal script success — always read StandardErrorContent. For Postgres, dollar-quote ($$...$$) string literals to avoid the quote-mangle entirely.

.env.local is NOT in git — reconcile it separately

A git reset --hard updates code but never touches the box's .env.local (it's gitignored, per-box). So changes that live in env — new vars, longer secrets, per-city URLs — do not ride along with a code rollout and silently leave the box on stale values. This bit the 2026-06-07 rollout repeatedly:

  • CODE_SERVER_JWT_SECRET was a 28-byte legacy value; the editor-link route requires ≥32 bytes → "code-server is not configured". The secret must be identical across ai-manager, collab-gateway, and the code-server auth-proxy container (recreate the container after changing it: docker compose -f .devcontainer/docker-compose.code-server.yml up -d --force-recreate).
  • COLLAB_SERVICE_TOKEN was <32 bytes → autosave/publish silently failed (collab auth rejects short tokens).
  • CODE_SERVER_URL pointed at the vanguard host, not https://code.<this-city-fqdn>.

When a pushed change adds or lengthens an env var, add the safe default or check to .devcontainer/scripts/lib-env.sh / .devcontainer/scripts/validate-env.sh, then run city-update.sh <city> --step env before the restart or rebuild. setup.sh must also export any SSM-reconciled values it writes into .env.local; otherwise later setup blocks can see stale defaults while the file looks correct. After editing .env.local, verify the value reached the live process (tr '\0' '\n' < /proc/<pid>/environ), not just the file — and read the process that owns the port (ss -ltnp | grep :3001), since several next dev workers look alike. Secret-length checks (Buffer.byteLength(x) < 32) are a recurring failure mode: grep the route's guard, don't guess.

For customer authentication and hosted payment capture, city setup and the env phase read three 64-hex SecureStrings: customer_auth_otp_pepper, payment_vault_internal_token, and payment_vault_frame_token_secret below /trueserve-dev/<city>/. Create or rotate them through Parameter Store, never by embedding a value in city-update.sh, a shell transcript, or SSM Send Command parameters. The env backup contains secrets and is deliberately mode 0600; its companion .keys file contains names only.

ASYNC_AGENT_API_TOKEN is a per-city secret, not a committed local-dev placeholder. Mint a tenant-scoped tsf_ API token in ai-manager with the workflow/tool/config scopes the server routes need (workflows:manage, tools:manage, configuration:manage, and agents:manage for the common local workflow/test/config paths), set it in the box's .env.local, then restart the readers (ai-manager and ai-orchestrator). Do not pass this token through city-update.sh --step env; SSM command history is the wrong place for secrets.

Separately, .env.local is not the only non-git state: Python venvs are rebuilt on the box and only contain what requirements.txt / pyproject.toml + setup.sh install. An import that isn't declared (conversation-api asyncpg/minio) or an editable install that setup skipped (code-executor tsf_runtime) leaves a service 500ing after a venv rebuild. setup.sh now self-heals the code-executor tsf_runtime case unconditionally; the general rule is declare every import in pyproject and make setup's install idempotent rather than gated on a .venv-ready marker. Node node_modules is the same kind of non-git state: ai-manager's Overmind launcher now checks the npm install marker, lockfile freshness, and required package assets before starting Next.js, then runs npm install --prefer-offline when the checkout has moved ahead of the installed dependencies. Do not put code-executor's Docker config in global DOCKER_CONFIG; city-update --step env migrates legacy values into CODE_EXECUTOR_DOCKER_CONFIG_DIR so only the code-executor launcher exports it. A global DOCKER_CONFIG can make host docker compose read the sandbox config and fail before Postgres starts.

When a force-push rollout applies

The reset-based driver assumes the city only consumes origin/<city>. Confirm that assumption against the live checkout before a rebase and force-push rollout. Unpublished city commits or edits require the preservation workflow above; resetting such a checkout would remove that work.

See also