Skip to content

Debugging official Dev and QA

Status: Current | Last reviewed: 2026-09-18

First identify the target deployment, not just the shell you are using. A city/Vanguard shell can investigate an official Dev or QA incident, but its local running services and checked-out branch are different deployments.

Connect to the target

On a laptop, use the city's WireGuard both profile and scoped DNS: Mac instructions or Windows instructions. Keep the core VPC VPN up. This carries the official API, browser and Grafana traffic through the city. If the city owner issued an OpenVPN bundle, use its both profile instead: Windows OpenVPN instructions. Activate one TrueServe tunnel at a time.

On the city host or its host-network devcontainer, the underlying city network already supplies that egress path. Verify it rather than creating a recursive access tunnel back to the same host:

python3 scripts/studio-access/probe.py --environments both

The tactical host installation also provides:

sudo /usr/local/bin/trueserve-access probe --environments both
Target Studio/API host Grafana Official source ref
Dev studio.dev.trueserve.deloitte.com grafana.kms.dev.trueserve.deloitte.com origin/dev
QA studio.qa.trueserve.deloitte.com grafana.truserve-qa-wus.qa.orchestrator.trueserve.deloitte.com origin/qa

Conversation APIs use conversation-api.dev.trueserve.deloitte.com or conversation-api.qa.trueserve.deloitte.com. Preserve normal TLS validation and the API's own authentication; the tunnel is network access, not an app login. Start with a small read-only request to the target, for example:

curl --connect-timeout 5 --max-time 15 -sS -o /dev/null -w '%{http_code}\n' \
  https://studio.qa.trueserve.deloitte.com/login

A login page response is only a reachability check. Use the relevant authenticated API and correlation IDs to investigate the actual incident.

Read official logs through Grafana

Follow Configure Grafana CLI access in a city for step-by-step token setup, the correct host/devcontainer user, verification and rotation. An authorized shared token can be provisioned across cities; it is still stored separately for Dev and QA under each CLI user's home.

Use Grafana's browser Explore view with the correct environment and log datasource. Select a bounded time range and filter to the affected namespace and service. Local overmind connect and docker logs show city processes; they are not evidence about official Dev/QA application failures.

For CLI access, obtain an authorized Grafana service-account token for the specific environment. Store it locally with mode 0600, for example:

  • Dev: ~/.local/state/trueserve/grafana-kms-dev.token
  • QA: ~/.local/state/trueserve/grafana-kms-qa.token

The relay deployment does not distribute tokens. Do not paste credentials in shell arguments, issue bodies or logs. Use the repository's GET-only helper:

python3 scripts/studio-access/grafana-read.py --environment dev /api/datasources

python3 scripts/studio-access/grafana-read.py --environment dev \
  /api/datasources/proxy/uid/loki/loki/api/v1/query_range \
  --param 'query={namespace="trueserve",app="ai-orchestrator"} |~ "(?i)error|exception"' \
  --param since=1h --param limit=100 --param direction=backward

Use --environment qa for QA, and discover its datasource UID first instead of assuming it matches Dev's loki. A different private token location can be passed with --token-file PATH. The helper uses fixed environment origins, refuses redirects and redacts common credential formats from output. Still review any incident excerpt before sharing it.

Group by the app label, not just a generic container called app. Read surrounding lines and correlate time, tenant/session/request IDs and deployed version. An “error” string can describe a handled challenge or normal hangup; it is not automatically a failed user request.

Inspect the code for the official environment

Refresh the remote refs without changing the current working branch:

git fetch origin dev qa
git log -1 --format='%h %cI %s' origin/dev
git log -1 --format='%h %cI %s' origin/qa

# Read or search QA code without switching the city checkout:
git show origin/qa:apps/ai-orchestrator/README.md
git grep -n 'SEARCH_TERM' origin/qa -- apps/ai-orchestrator

Replace origin/qa with origin/dev for Dev. If a file-based inspection or fix needs a checkout, create a separate worktree at the appropriate ref; do not reset or switch the live city's checkout.

Branch head is the intended environment source, but deployment can lag. When available, record the ArgoCD synced revision, deployed image tag or build SHA alongside the branch ref and inspect that exact revision. A city branch or environment/vanguard file cannot establish what official QA is executing.

Record and act on the evidence

Record the target environment, time window, hostname/API path, relevant log stream, response/correlation ID and source/deployed revision. Propose the fix against the correct environment workflow. Network access does not authorize cluster writes, local city restarts as an official-environment fix, or bypassing the existing ArgoCD/branch promotion rules.

See the relay operator guide for DNS/route troubleshooting and observability for log interpretation.