City access relay rollout

Status: WireGuard/OpenVPN available on all eight cities; Dev/QA Grafana queries verified | Last reviewed: 2026-09-18

September 18 update

Pushed the access-tool commits to origin/environment/vanguard. The runtime implementation validated at 2a262b72aa896ba3fa5841b3b34a7ae44e3b8fa4 is installed on Richmond, Rome, Kyiv, Istanbul, Macau, Shimla, Seattle and Athens, together with the subsequent report/guide update. The exact installed tools revision is recorded in /var/lib/trueserve-access/deployment.json and /var/lib/trueserve-openvpn/deployment.json. Both stable host commands are available: sudo trueserve-access and sudo trueserve-openvpn.

Updated the toolkit, Mac/Windows guides, Grafana instructions and managed AGENTS.md / CLAUDE.md blocks. OpenVPN now runs beside WireGuard on every city, using the per-city endpoints and subnets. Existing application branches/HEADs and all pre-existing container IDs/start times matched the preflight. Existing WireGuard state, OpenVPN server/device keys and provisioned Grafana credential files were preserved. Shimla's enrolled jyoti-windows identity and running OpenVPN container remained in place.

A separate temporary OpenVPN client connected to every city's private UDP endpoint. All eight passed full Dev and QA checks: tunnel DNS, Studio HTTPS, Janus WebSocket upgrade, UDP STUN, TLS TURN, ArgoCD, Dex, Grafana login and shared SSO. Default routes were preserved and relay endpoints stayed outside the tunnel. All temporary identities were revoked, their private exports removed, and local test containers deleted. The seven new cities have no developer peer until a city owner enrolls an actual laptop.

Authenticated QA Grafana/Loki queries passed from every city using the existing developer account and private credentials. The old Dev token initially returned HTTP 401 because it had been removed. Later on September 18, the operator-provided replacement was distributed privately to all eight cities. Dev and QA Loki queries now pass from all eight hosts and all six active devcontainers. Vanguard's local CLI was updated too. QA credentials, tunnel state and running services remained unchanged. No token was copied into Git. See the private token setup procedure for future rotations. Earlier Dev HTTP 503 and 401 observations are historical.

Thirty-five offline Python tests and syntax/diff checks passed. The user confirmed the Windows OpenVPN setup worked on Shimla. Each additional laptop still needs its own profile and browser/audio acceptance test; infrastructure probes do not establish an authenticated browser voice session.

For OpenVPN onboarding, run on the selected city host:

sudo trueserve-openvpn add-peer DEVICE-NAME

Deliver /var/lib/trueserve-openvpn/exports/DEVICE-NAME.zip privately and give the developer the Windows OpenVPN guide. Use one both profile for Dev and QA together. Device certificates expire 30 days after issuance.

September 17 baseline

The access runtime is installed and running on every current city host. Each relay supports a single laptop profile for Dev and QA together. Developers still need their own device bundle and the laptop setup steps: Mac or Windows.

Initial installed release

Runtime commit: 0f637c36c2feb4ff49921e3de3852a85f27d60e0, following toolkit commit 1ac3c6535dc6. Region: us-east-1; account: 465572510980. Installation and verification completed on September 17, approximately 16:02–16:07 UTC, through AWS Systems Manager.

Only the access runtime and four related guides were transferred. Releases live under /opt/trueserve-access/releases/<commit>; the stable host command is sudo trueserve-access. Access files and managed agent-instruction blocks were also copied into each city's existing checkout for developers and agents.

City	Private UDP endpoint	Tunnel subnet	Preserved application branch / HEAD	Relay / QA result
richmond	10.120.206.52:51828	10.254.240.0/24	richmond / c88adc3b5789	Running; QA pass
rome	10.120.206.10:51828	10.254.241.0/24	AS-382-Agent-Canvas / 8de01e6291b2	Running; QA pass
kyiv	10.120.206.42:51828	10.254.242.0/24	CMTS-29786-headless-render-local-hop / 4a9f65eeb3e0	Running; QA pass
istanbul	10.120.206.24:51828	10.254.243.0/24	istanbul / d12587f5b2fb	Running; QA pass
macau	10.120.206.58:51828	10.254.244.0/24	dev / dfc95520ac83	Running; QA pass
shimla	10.120.206.38:51828	10.254.245.0/24	bug/multi-agent-sessionid-fix / f23a7d2ba201	Running; QA pass
seattle	10.120.206.47:51828	10.254.246.0/24	seattle / e49415e97024	Running; QA pass
athens	10.120.206.59:51828	10.254.247.0/24	athens / 125af02d80aa	Running; QA pass

All application branches, HEAD revisions and pre-existing running Docker container IDs matched the preflight after installation. The relay uses restart: unless-stopped; host application services were not restarted. Each city has separate server keys. Temporary verification peers were removed; no shared developer peer or Grafana token was distributed.

Root AGENTS.md, root CLAUDE.md and .devcontainer/CLAUDE.md on all eight hosts now direct official Dev/QA investigation to the target APIs and Grafana, and to refreshed origin/dev / origin/qa source refs with deployed-revision checks. Existing instructions outside the managed block were preserved.

What passed on every city

An external disposable WireGuard client connected to each city's actual private UDP endpoint. The tests used the relay's DNS and routes, not DNS/IP overrides or a local-only relay fixture. All eight passed:

• WireGuard handshake, tunnel DNS and certificate-verified Dev/QA TLS transport.
• QA Studio HTTPS 200, Janus WebSocket upgrade 101, ArgoCD 200, Dex discovery 200, Grafana login redirect 307, and shared SSO redirect 302.
• Dev and QA STUN binding over UDP 3478 and verified TURN TLS on 5349.
• Dev Janus WebSocket 101 and ArgoCD 200.
• Cleanup of each temporary peer, with no test peer left registered.

A separate Richmond onboarding test generated a private per-device ZIP, verified mode 0600, both OS guides, both helpers and Dev/QA/both profiles, then revoked the peer and removed the generated export.

The city SG already admitted UDP 51828 through its existing high-port rule. This rollout made no additional cloud security-group, WAF or route changes. QA relies on the earlier NAT-source SG and WAF additions recorded in the operator guide. Their incorporation into KMS-managed configuration remains a ticket follow-up.

Dev service outage observed during rollout

Dev Studio's HTTPS connection established but its HTTP response timed out. Dev Dex and Grafana returned 503 with server: istio-envoy and no healthy upstream. Both Dev management IPs returned the same response, and direct probes from the previously working Vanguard path reproduced it. QA continued to pass through the same relays.

These observations distinguish an upstream Dev service outage from a failed WireGuard route. The installer used explicit --transport-only validation and recorded that limitation; the full application probe was still run through every new tunnel. No Dev application or cluster changes were made.

After Dev recovers, run on the chosen city host:

sudo trueserve-access probe --environments both

A successful transport check alone must not be used to close a Dev application incident. Native browser SSO and voice checks remain required below.

Start onboarding developers

On the developer's chosen city host, create a unique device peer:

sudo trueserve-access add-peer alex-windows
# Or another actual device name, for example: eric-mac

Deliver /var/lib/trueserve-access/exports/<device>.zip privately using the operator transfer procedure. Give the developer the matching laptop guide. Use both for simultaneous Dev/QA tabs; activate only one city tunnel per laptop. A second laptop gets a separate device key and bundle.

Revocation is immediate:

sudo trueserve-access revoke alex-windows

Remaining acceptance and evidence

The original Mac test demonstrated Dev sign-in and two-way audio, and the user confirmed the legacy Mac upgrade fixed its stale-interface problem. The new city bundles still need native Windows/Mac acceptance: core VPC access preserved, browser SSO, a real Dev and QA call with audio in both directions, and disconnect/reconnect/sleep recovery. STUN and TLS probes do not perform authenticated TURN allocation or establish a real voice call.

Offline validation passed 22 Python tests, Python fatal-error lint and shell syntax checks. Earlier PowerShell parser/NRPT mocks and disposable Docker DNS/TCP/UDP/restart/revocation tests also passed.

Per-host deployment manifests are root-only at /var/lib/trueserve-access/deployment.json. Private operator audit records contain SSM command IDs, preflight/postflight snapshots and complete probe results; these are operational records outside Git. No credentials or private profiles are included in this report.

Grafana credential follow-up

After the relay rollout, the user explicitly authorized using the supplied shared Dev/QA Grafana credentials on every city. Configuration code and the step-by-step credential guide were committed as 26d222bb and copied to all eight city checkouts.

The tokens are stored in .local-data/grafana/, excluded from Git, with folder mode 0700 and file mode 0600, owned by the city's developer UID. This host-backed folder remains available when a devcontainer is recreated. The CLI prefers a personal home-directory credential, then the city store. No application branch or service restart was required.

Credential transfer used a separate ephemeral RSA-OAEP key for each city; SSM commands carried ciphertext, and the ephemeral private keys were removed after installation. Plaintext credentials were not placed in Git or SSM command parameters. Private verification receipts are stored on each host at /var/lib/trueserve-access/grafana-verification-20260917.json.

QA datasource discovery and an authenticated Loki query returned success from all eight host developer accounts and all five currently running city devcontainers (Richmond, Rome, Macau, Shimla and Athens). Dev credentials were saved in the same contexts, but Dev Grafana still returned HTTP 503; this does not establish whether the Dev token is valid once service recovers.