Laptop access to Dev and QA through a city relay

Status: City rollout complete; native automatic-client acceptance pending | Last reviewed: 2026-09-21

Use a dedicated WireGuard relay on a city EC2 host to carry laptop traffic to TrueServe Dev and QA. Keep the laptop's existing core VPC VPN connected: it carries the encrypted WireGuard connection to the city's private IP. The city then forwards selected application traffic through its existing network.

The toolkit creates private per-device bundles with Mac and Windows helpers. Give developers the separate Mac laptop guide or Windows laptop guide. Both explain each step and assume the host is ready. They ship in every bundle as MAC-SETUP.md and WINDOWS-SETUP.md; LAPTOP-README.md is the shorter shared reference.

What has been verified

The September 17 city rollout report records the installed hosts, exact runtime revision and current acceptance results. During that rollout, Dev Studio timed out and Dev Dex/Grafana returned 503 no healthy upstream, including from the existing Vanguard path. The city tunnels passed transport checks; QA passed the full probe suite. Treat the earlier successful Dev checks below as historical evidence, not confirmation that the Dev application outage has recovered.

On September 17, the original Vanguard/Mac test loaded Dev Studio, completed sign-in and carried audio in both directions. It used the host's ordinary network egress, independently of the experimental GlobalProtect container.

Earlier on September 17, Dev and QA both passed HTTPS, Janus WSS, STUN/UDP, TURN TLS, ArgoCD, Dex and Grafana probes from Vanguard and from an authenticated, disposable WireGuard client using the toolkit's relay, scoped DNS and combined destination routes. Shared SSO HTTPS also passed. QA Studio returned 200; Janus upgraded with 101. This uses the ordinary host network and public application DNS.

QA initially timed out at its public ALB (52.38.231.248, 54.185.43.168). The September 17 fix added the city NAT egress 54.92.142.217/32 to both the QA ALB's TCP 443 security-group rule set and its existing WAF IP allowlist. See the exact resources and ticket follow-up below.

The original Mac proof of concept is an end-to-end result; the new Mac and Windows helpers require native laptop acceptance testing. A relay-side probe does not prove browser sign-in, microphone permission, ICE selection or audio.

The disposable Docker integration test passes scoped Dev/QA DNS, IPv6-answer filtering, TCP/UDP forwarding, restart/reconnection and live peer revocation. PowerShell parsing and mocked DNS ownership, rollback and policy-conflict checks pass. The live WireGuard probe additionally verifies the actual Dev and QA network paths, but does not replace authenticated browser/audio tests. The original StudioTest bundle was Dev-only. Its running relay was subsequently extended with six QA forwarding destinations and 21 total DNS names, and both environments were verified through a temporary authenticated peer on that exact relay/endpoint. The original Mac peer was left connected. Existing Mac users still need the local profile/DNS upgrade below; cloud allowlist changes alone do not update a laptop.

Upgrade the original Mac StudioTest connection

Keep the core VPN and the Homebrew-managed StudioTest tunnel active. Download upgrade-studio-test-mac.sh onto the Mac, then run it from the existing StudioTest-mac folder:

sudo bash ~/Downloads/upgrade-studio-test-mac.sh ./StudioTest.conf

This specific migration contains no private keys. It verifies the active interface and profile identity, backs up the old profile, adds QA routes to the live WireGuard peer and saved config, verifies QA HTTPS through the tunnel, and installs the scoped DNS names. It extends targets.txt so the original studio-dns.sh disable command still removes all its DNS entries. Version 3 tolerates a stale utun8 and live utun9 both retaining the 10.254.253.250 address: it selects the single WireGuard interface whose client/server keys, endpoint and recent handshake match. It moves this profile's routes from the stale interface and removes its duplicate address. An old CLI-visible WireGuard peer is disabled only when both keys and the endpoint match and its handshake is stale. Interfaces carrying the core endpoint or a different WireGuard identity are preserved. Two matching interfaces with recent handshakes are treated as ambiguous and left unchanged.

An address-only leftover interface may remain as an empty device until its owning application closes it; the script does not kill unknown VPN processes. It refuses unrelated conflicting routes or unmanaged resolver files. Mocked regression tests cover cleanup and these preservation boundaries; native Mac execution remains part of acceptance. Its address snapshot is dated September 17, 2026; new users should use the generated profiles and normal lifecycle helpers.

After the script succeeds, reload https://studio.qa.trueserve.deloitte.com/, sign in and verify audio in both directions. Native QA acceptance remains pending until that laptop test succeeds.

Connection model and scope

flowchart LR
    Laptop["Mac / Windows<br/>TrueServe WireGuard tunnel"]
    Core["Existing core VPC VPN<br/>UDP to city private IP"]
    Relay["City EC2<br/>isolated WireGuard + scoped DNS"]
    Dev["Dev applications, TURN<br/>and KMS tools"]
    QA["QA applications, TURN<br/>and KMS tools"]
    SSO["Shared SSO"]
    Laptop --> Core --> Relay
    Relay --> Dev
    Relay --> QA
    Relay --> SSO

Choose one relay that can reach both environments, and use the both profile. Dev and QA then work side by side in ordinary browser tabs. dev and qa profiles are alternatives for users who only need one environment. Disconnect before switching. Do not activate separate Dev and QA tunnels simultaneously: their shared SSO and possibly other destination routes overlap.

This is IP forwarding, so TCP and UDP to the selected destinations—including WebSockets, SSE and TURN media—can traverse the relay. It is not a universal browser proxy or a full-tunnel VPN. Only enumerated IPv4 destinations and any explicitly reviewed extra media routes are forwarded. WebRTC ICE candidates, third-party identity providers and integrations outside that list retain their normal laptop route. Dev-only/QA-only profiles are convenience filters, not server-side authorization boundaries; the relay permits its union of destinations.

The relay publishes one private UDP port and isolates its forwarding/NAT rules in a Docker network namespace. It does not change host routes, global DNS or host forwarding policy. Docker manages the ordinary published-port rules. Kernel WireGuard and Docker NET_ADMIN are required. No Kubernetes change, VPN password, shared Grafana token or GlobalProtect browser session is included in the toolkit.

City prerequisites

1. Use an existing city EC2, managed through SSM. Run relay commands on the Ubuntu host, not its vscode devcontainer. The host checkout is /home/ubuntu/trueserve-framework. Updating it remains the normal city release workflow; the access installer never resets Git.
2. Ensure the laptop can route to that host's private IP over its existing core VPC VPN. An SSM or VS Code TCP port forward cannot carry WireGuard UDP. City HTTPS through Global Accelerator does not imply UDP relay reachability.
3. Select an unused host UDP port (default 51828) and unused RFC1918 tunnel subnet (default 10.254.252.0/24). Check both the host and laptop routing tables, including corporate VPN ranges. The original test uses a different port/subnet (51827, 10.254.253.248/29) and can be retired after migration.
4. The host's security group must admit that UDP port from the actual core VPN client/source CIDRs. NACLs must permit it and return traffic. Inspect existing rules before proposing a narrowly scoped change; the scripts do not modify SGs, NACLs, TGWs or load balancers.
5. The city must resolve and reach all required downstream names, including private KMS management addresses and the shared SSO host. Its egress must be allowed by the destination SG/WAF/network policy. A working Dev path does not establish a working QA path.

The September 17 EC2 inventory places all eight current cities in us-east-1, VPC vpc-03baca9c704dd5749, subnet subnet-05f087a854a624070, alongside Vanguard. Its default route uses NAT gateway nat-037bd395f735109df, with public egress 54.92.142.217. This is separate from each city's private WireGuard endpoint:

City	Private endpoint IPv4
richmond	10.120.206.52
istanbul	10.120.206.24
macau	10.120.206.58
shimla	10.120.206.38
rome	10.120.206.10
kyiv	10.120.206.42
seattle	10.120.206.47
athens	10.120.206.59

Instance IDs are in city dev environments. Verify the selected instance's current private IP and egress during setup; these values are an inventory snapshot, not hardcoded installer defaults.

Current endpoint catalog: environments.json. It includes Studio, conversation API, surface-app, Janus, TURN, tools, code-server, ArgoCD, Dex and Grafana for each environment, and sso.trueserve.deloitte.com once. Platform hostnames intentionally retain QA's legacy truserve-qa-wus spelling.

Private application load-balancer path

If the public QA path fails but the city has approved private VPC connectivity, the operator can test the same application ALB at its private ENI addresses. Obtain those addresses from QA EC2 Network Interfaces using the ALB's name. Management gateway IPs and historical application NLB IPs are not substitutes. Never disable TLS verification to make an alternative address appear healthy.

Create a JSON mapping of exact application names to the confirmed ALB private IPv4 addresses. Include each applicable QA L7 name from the endpoint catalog (Studio, conversation API, surface-app, Janus, tools and code); leave TURN and management names on their own DNS records. Do not add guessed IPs.

sudo python3 scripts/studio-access/relay.py probe --environments both \
  --dns-overrides /root/qa-private-dns.json

If the probe passes, supply the same --dns-overrides file to init. Overrides change the socket destination while preserving the hostname, TLS SNI, certificate checks and HTTP Host. They are validated against the selected catalog, saved in private state, and used consistently for tunnel DNS and client routes. They do not alter corporate or public DNS. Private ENI addresses can change; update them through the coordinated refresh --dns-overrides FILE workflow. An empty JSON object removes overrides.

QA source allowlist fix and ticket follow-up

The verified Vanguard/city public egress is 54.92.142.217 and the QA application ALB DNS name is k8s-skasyste-publicl7-4bbb54704b-2119879782.us-west-2.elb.amazonaws.com. The initial 41 inbound source ranges did not include this address, and the second attached group (the shared Kubernetes backend SG) had no inbound rule allowing it either.

After the operator made the qa AWS CLI profile available, two targeted updates were applied in account 836010056554, region us-west-2:

Resource	Change
Frontend SG sg-0e81233c66c1ab9c3 (for-alb-rules)	Added TCP 443 from 54.92.142.217/32, rule sgr-0202a276efbb92963
WAF IP set waf-rules-westus_IPallowedlist_IPV4, ID cd2cb79b-d9cb-40d2-a582-ef97fa343786	Appended the same /32, retaining all 41 previous entries

The IP set is referenced by IPallowedlist in regional web ACL waf-rules-westus. Its existing allow rule and the other WAF rules were retained. The SG update changed QA from a TCP timeout to HTTP 403; the WAF update then produced Studio 200 and Janus 101. No Dev cloud resources, shared backend SG, DNS records or application settings were changed.

The previously submitted ticket should record both updates in the KMS-managed configuration so reconciliation does not remove them. Ticket closure has not been confirmed. These were explicit operational updates; the reusable relay installer still does not mutate cloud security controls.

For rollback, revoke only SG rule sgr-0202a276efbb92963 and remove only 54.92.142.217/32 from the WAF IP set using its current lock token. Preserve any intervening changes by other operators; do not restore an old full IP set blindly. Rollback removes the newly enabled city-to-QA application path.

Install on a city

Tactical release installation

The city rollout installs only the committed access files under /opt/trueserve-access/releases/<commit> and selects that release through /opt/trueserve-access/current. install-city.py builds and starts the dedicated relay, copies the access scripts and related guides into the city's bind-mounted checkout, and adds a bounded instruction block to root AGENTS.md, root CLAUDE.md and .devcontainer/CLAUDE.md. Other instructions are preserved. It does not fetch, switch or reset application branches.

The rollout's per-city endpoint and tunnel subnets are recorded in cities.json. Verify EC2 identity/IPs before using this inventory for a later rollout. Each city gets separate server keys and state; profiles belong to one city and one device.

After installation, run these commands on the host through SSM:

sudo trueserve-access status
sudo trueserve-access probe --environments both
sudo trueserve-access add-peer alex-windows
sudo trueserve-access add-peer eric-mac

The last two are examples: create a uniquely named peer only for the actual device being onboarded. Deliver its private ZIP from /var/lib/trueserve-access/exports/<device>.zip using the private transfer procedure below. Do not give every developer a shared city key. The installed relay can run with no enrolled peers while waiting for onboarding.

For city-side log access, follow Grafana credential setup. The relay itself does not supply Grafana authentication.

The active revision and preserved application HEAD are recorded in /var/lib/trueserve-access/deployment.json. Root-only installation/build logs are stored alongside it. The Docker restart policy keeps the relay running across host restarts; application services are managed separately.

Manual installation

Connect using the AWS CLI and Session Manager plugin, with the correct account and region for the selected city:

aws ssm start-session --region YOUR_REGION --target YOUR_INSTANCE_ID

On the host:

cd /home/ubuntu/trueserve-framework
sudo python3 scripts/studio-access/relay.py build
sudo python3 scripts/studio-access/relay.py probe --environments both

The probe runs on a Docker bridge, matching the relay's egress topology. It requires HTTPS responses, Janus upgrade 101, STUN binding success and certificate-valid TURN TLS. If an endpoint fails, investigate it before proceeding. Probe success does not authenticate to TURN or test a voice call.

After the selected environments pass:

# Replace the endpoint with this city's assigned private IPv4 address.
sudo python3 scripts/studio-access/relay.py init \
  --endpoint YOUR_CITY_PRIVATE_IP --environments both \
  --port 51828 --network 10.254.252.0/24
sudo python3 scripts/studio-access/relay.py add-peer eric-mac
sudo python3 scripts/studio-access/relay.py add-peer colleague-windows
sudo python3 scripts/studio-access/relay.py start
sudo python3 scripts/studio-access/relay.py status

init repeats the selected environment probes before creating state or keys; it refuses setup when a required path fails. It resolves the selected hostnames from the host's resolver and rejects recursive endpoint routes, overlapping host tunnel ranges, metadata/loopback destinations and broad extra routes. Each peer gets its own keypair and /32; all three profiles for that peer intentionally reuse that device identity. The container restarts with Docker unless explicitly stopped.

Private state is under /var/lib/trueserve-access (0700); keys, configs and ZIPs are 0600. Peer bundles are generated at /var/lib/trueserve-access/exports/<peer>.zip. Server-side key generation simplifies onboarding, so the operator also holds client private keys. Protect this state as credential material; do not commit it.

For repeatable remote administration from a checkout with Python 3 and the AWS CLI:

python3 scripts/studio-access/city.py \
  --region YOUR_REGION --instance-id YOUR_INSTANCE_ID -- build
python3 scripts/studio-access/city.py \
  --region YOUR_REGION --instance-id YOUR_INSTANCE_ID -- probe --environments both
python3 scripts/studio-access/city.py \
  --region YOUR_REGION --instance-id YOUR_INSTANCE_ID -- \
  init --endpoint YOUR_CITY_PRIVATE_IP --environments both
python3 scripts/studio-access/city.py \
  --region YOUR_REGION --instance-id YOUR_INSTANCE_ID -- add-peer eric-mac
python3 scripts/studio-access/city.py \
  --region YOUR_REGION --instance-id YOUR_INSTANCE_ID -- start

city.py executes the already-installed host script through SSM and reports the remote exit code. It does not upload source, change cloud resources, retrieve keys into SSM output or update application services.

Transfer a private bundle without SSH

In an interactive SSM host shell, start the single-download loopback server:

sudo python3 /home/ubuntu/trueserve-framework/scripts/studio-access/download.py \
  /var/lib/trueserve-access/exports/eric-mac.zip

In a separate laptop terminal, forward its TCP port:

aws ssm start-session --region YOUR_REGION --target YOUR_INSTANCE_ID \
  --document-name AWS-StartPortForwardingSession \
  --parameters '{"portNumber":["18086"],"localPortNumber":["18086"]}'

For Windows PowerShell, avoid native JSON quoting differences:

'{"portNumber":["18086"],"localPortNumber":["18086"]}' |
  Set-Content -Encoding ascii ssm-ports.json
aws ssm start-session --region YOUR_REGION --target YOUR_INSTANCE_ID `
  --document-name AWS-StartPortForwardingSession `
  --parameters file://ssm-ports.json

Open the random http://127.0.0.1:18086/<random>.zip URL printed by the host script on the laptop. The server listens only on host loopback, serves only that bundle, and exits after one download or ten minutes. Close the SSM port forward afterward. This TCP tunnel is for bundle delivery, not WireGuard. An approved private file-transfer channel is also suitable.

Extract the ZIP into a private folder and follow MAC-SETUP.md or WINDOWS-SETUP.md for the developer's laptop. On Windows, the active CLI config gets a restricted ACL, but the downloaded ZIP and extraction folder still need normal private-profile storage.

Everyday use and switching

Task	Mac CLI	Windows elevated PowerShell
Connect both	sudo bash mac.sh up both	.\windows.ps1 up both
Check	sudo bash mac.sh status	.\windows.ps1 status
Disconnect	sudo bash mac.sh down	.\windows.ps1 down
Dev only, after disconnect	sudo bash mac.sh up dev	.\windows.ps1 up dev
QA only, after disconnect	sudo bash mac.sh up qa	.\windows.ps1 up qa
GUI tunnel DNS on	sudo bash mac.sh dns-on both	.\windows.ps1 dns-on both
GUI tunnel DNS off	sudo bash mac.sh dns-off	.\windows.ps1 dns-off

DNS helpers verify the route to the relay endpoint stays outside the new tunnel, all selected routes use its interface, and direct tunnel DNS answers. Only then do they install exact-host scoped DNS. Mac uses /etc/resolver; Windows uses NRPT. Unknown names are not forwarded by the relay resolver, and AAAA answers are suppressed for the scoped names. Browser DoH and managed DNS policy require separate review if the browser ignores the OS resolver. Mac ownership state is retained under /var/db/trueserve-access so DNS cleanup still works after a reboot.

Turn scoped DNS off before deactivating a GUI tunnel. A reboot, sleep or VPN drop may leave DNS rules while connectivity is absent; use status and down/dns-off to recover. Helpers do not install a background reconnect agent. Windows CLI tunnel services may start at boot before the core VPN is ready; keep the core connection available when using them.

DNS changes and maintenance

The relay intentionally pins a versioned IPv4 snapshot, with DNS and client routes generated from the same records. It does not resolve new ALB addresses at runtime, which could make DNS return addresses absent from laptop routes. Address changes require a coordinated refresh:

1. Have clients disconnect and remove their scoped DNS.
2. On the host, run:

bash sudo python3 scripts/studio-access/relay.py stop sudo python3 scripts/studio-access/relay.py refresh sudo python3 scripts/studio-access/relay.py start sudo python3 scripts/studio-access/relay.py probe --environments both

1. Deliver the regenerated bundles to every device. Replace the entire old bundle; GUI users must replace their imported profile too. Reconnect and repeat browser and audio acceptance tests.

refresh refuses a running relay, requires the selected probes to pass, preserves existing peer keys, and reissues all bundles. Manual clients use snapshots. The optional automatic maintenance mode refreshes host DNS and laptop routes, starts at boot and checks HTTPS every 30 minutes. A new catalog or extra-media-route change needs the same coordinated treatment; the initialized catalog is stored in private state, so editing the repository catalog alone does not update an existing relay. Use a new state directory and explicit migration for catalog/subnet/port changes.

Add another device with add-peer while running; wg syncconf preserves other peers. Revoke a device immediately with:

sudo python3 scripts/studio-access/relay.py revoke eric-mac

Revocation removes the live peer and its local exports. Previously distributed bundles stop authenticating. remove stops/removes only the managed relay container and retains private state. Remove laptop DNS first; private state deletion is a separate operator decision.

Troubleshooting and acceptance

Application outages and relay faults need separate checks. The normal probe and init commands require successful application responses. For a confirmed upstream outage, probe --transport-only, init --transport-only, and install-city.py --transport-only explicitly require certificate-verified TLS and STUN reachability instead. This mode records limited validation and does not establish HTTP, WebSocket, sign-in or call health. Run the normal probe separately and record which environment remains degraded. Never describe a transport-only pass as full application readiness.

Observation	Next check
No handshake	Core VPN route to city IP; SG/NACL UDP rule; server status; matching peer keys
Handshake, direct DNS timeout	Route to tunnel DNS; stale Mac utun; peer address; resolver process
Direct DNS works, browser name fails	Scoped DNS installed; OS resolver cache; browser DoH; Windows effective NRPT policy
Mac route: File exists	Old TrueServe interface still owns routes; deactivate old GUI/CLI tunnel first
HTTPS times out only in QA	QA ALB reachability from city; downstream source allowlist; current DNS snapshot
HTTPS works, Janus fails	WSS /janus upgrade and certificate, then app-advertised signaling URL
Call connects, one-way/no audio	ICE selected pair, TURN allocation, browser mic permission, advertised candidates and allowed media destinations
ArgoCD Unregistered redirect_uri	Platform Dex client registration; this is not a WireGuard/port-forwarding issue

For the Mac stale-route case, status shows the interface holding the peer address and the interface used for tunnel DNS. The original failure had a healthy utun9 handshake but destination routes still on utun8. mac.sh repair-routes both utun8 is an explicit, narrowly scoped recovery after verifying utun8 belonged to the previous TrueServe tunnel. It refuses the interface carrying the core endpoint and routes on unexpected interfaces. Never delete all utun interfaces or flush corporate VPN routes.

Acceptance on each OS and each environment:

1. Record the core VPN route to the city before and after connection; it must remain outside WireGuard. Check a normal core VPC resource still works.
2. Confirm handshake and direct DNS, then OS DNS and browser Studio sign-in.
3. Confirm Janus WebSocket, place a real call, and verify sent/received audio. Inspect chrome://webrtc-internals or the equivalent browser diagnostics for the selected candidate pair; record relay/direct, IPs, ports and both byte counters. Avoid exporting credentials or conversation payloads.
4. Open ArgoCD and Grafana with their own normal SSO.
5. Test both with Dev and QA tabs, then disconnect and verify scoped DNS and the test routes are gone while core VPC access remains.
6. Test reconnect, GUI/CLI switching, sleep/reboot recovery and a fresh bundle.

If direct media addresses are outside the endpoint list, add only reviewed specific destinations at relay initialization with --extra-route IP/32 (up to /24). Do not add a default route or broad private network ranges. Alternatively have the application owner review TURN-only policy; the toolkit does not change app ICE settings.

ArgoCD CLI SSO

The CLI expects the hostname without https:// or a trailing slash. Browser SSO can work while CLI SSO fails. The current Dev external Dex setup has client ID argocd and did not accept http://localhost:8085/auth/callback. The KMS team needs to register an appropriate public CLI client and configure ArgoCD's oidc.config.cliClientID. A localhost tunnel cannot repair registration.

After the platform supports the CLI flow, run the CLI locally on the laptop where possible. For a headless city session, select a free remote port, disable automatic browser launch, and forward the callback to the laptop. Example only after the exact redirect is registered:

argocd login argocd.kms.dev.trueserve.deloitte.com \
  --sso --grpc-web --sso-launch-browser=false \
  --sso-port 18085 --callback http://localhost:8085

Forward laptop TCP 8085 to remote TCP 18085 through SSM or VS Code Ports, and open the printed authorization URL on the laptop. Port 8085 is commonly occupied by media-proxy in this dev environment. Neither a port change nor --insecure fixes an unregistered redirect.

References

For Windows devices that need another client, see the OpenVPN companion relay pilot and its laptop guide.

• WireGuard installation
• WireGuard Windows service and CLI support
• Microsoft NRPT rules
• AWS SSM port forwarding
• Endpoint inventory
• Grafana CLI and observability