Configure Grafana CLI access in a city¶
Status: Current | Last reviewed: 2026-09-18
Dev and QA each have one shared official Grafana. Every city connects to those same instances. A city does not need its own Grafana server. Configure the credentials under the city OS account that runs your agent or CLI.
On September 17, the operator-approved shared Dev and QA credentials were provisioned on all eight cities. Existing city users can start at step 4; token creation and entry are for new setups or rotation. QA log queries passed from every host and all five active devcontainers in that initial check.
September 18 credential rotation: the previous Dev token had been removed. The operator supplied a replacement, which was distributed privately to all eight cities. Authenticated Dev and QA Loki queries now pass from every city host and all six active devcontainers. Vanguard's local CLI credential was also updated. QA credentials, tunnel state and running services were preserved. The earlier Dev HTTP 401 result is resolved; existing city users can use the reader directly without entering the replacement token themselves.
A working tunnel or browser login does not validate an API token. Use the checks below after any future rotation.
This setup is separate from the tunnel client and browser SSO. The relay supplies network access; a service-account token authenticates CLI log queries. Browser users can sign in normally without saving a CLI token.
1. Obtain the Dev and QA tokens¶
Use credentials supplied through your team's approved private channel. Keep the Dev and QA tokens labeled separately. A Dev token is not a QA token. An operator may distribute an explicitly approved shared credential to multiple cities; service-account permissions and revocation then apply to every city using that credential.
If new credentials are needed, open the appropriate Grafana in your laptop browser with the tunnel connected and complete SSO:
| Environment | Grafana |
|---|---|
| Dev | https://grafana.kms.dev.trueserve.deloitte.com/ |
| QA | https://grafana.truserve-qa-wus.qa.orchestrator.trueserve.deloitte.com/ |
An authorized administrator can use Administration → Users and access →
Service accounts → Add service account. Give it an identifiable name such
as richmond-dev-logs, assign Viewer or an approved narrower query role,
and confirm access to the intended Loki datasource. Open the account, choose
Add service account token, set an expiration according to team policy,
and generate/copy the token. Repeat in the other environment.
If the controls are unavailable, ask the Grafana/KMS administrator to issue the credential. Ordinary SSO access does not establish service-account administration permissions.
What this achieves: gives CLI requests an authorized identity. The setup helper below only saves an existing credential; it cannot create one in Grafana.
2. Use the correct city terminal¶
For VS Code users, open the integrated terminal inside the city's devcontainer:
cd /workspaces/trueserve-framework
whoami
python3 -c 'from pathlib import Path; print(Path.home())'
Run setup and subsequent Grafana commands as this same user, without sudo.
Tokens saved as the EC2 host's ubuntu user or SSM's root user are not
automatically available inside the devcontainer.
For operator-provisioned shared access, the city also has a private
.local-data/grafana/ folder in its bind-mounted checkout. It is excluded
from Git, owned by the city's developer UID, and survives devcontainer
recreation. The reader uses a personal home-directory token first, then this
city fallback. Thus the same provisioned files can serve the host and its
devcontainer without copying tokens into an image or restarting services.
If you intentionally query from the EC2 host, use its checkout
/home/ubuntu/trueserve-framework and the same host user for setup and queries.
What this achieves: puts the token in the home directory the CLI actually uses. Repeat setup when moving to another city or OS account, unless an authorized operator has already provisioned that account.
3. Save the credentials with a hidden prompt¶
python3 scripts/studio-access/grafana-token.py --environment dev
python3 scripts/studio-access/grafana-token.py --environment qa
Paste only the matching token at each prompt and press Enter. The terminal does not display its characters. The helper writes private files:
| Environment | Default file |
|---|---|
| Dev | ~/.local/state/trueserve/grafana-kms-dev.token |
| QA | ~/.local/state/trueserve/grafana-kms-qa.token |
What this achieves: saves credentials without including them in command
arguments or shell history. The dedicated folder has mode 0700; token files
have mode 0600. Re-running setup replaces only that environment's file.
If the operator-provisioned city folder exists, the setup helper updates that
folder instead and prints its destination. An existing personal token takes
precedence when reading; remove or rotate that personal copy too if applicable.
Never paste credentials into chat, agent prompts, source files, or SSM Run Command parameters. A shared city OS account shares credential access with processes running as that account. Keep approved source credential files outside Git.
4. Verify both environments¶
python3 scripts/studio-access/grafana-read.py --environment dev /api/datasources
python3 scripts/studio-access/grafana-read.py --environment qa /api/datasources
What this achieves: confirms that the network path, token and permissions
allow datasource discovery. Find the Loki datasource's uid in each response.
Then verify a small log query using that environment's UID:
# Replace LOKI_UID with the QA Loki datasource UID.
python3 scripts/studio-access/grafana-read.py --environment qa \
/api/datasources/proxy/uid/LOKI_UID/loki/api/v1/query_range \
--param 'query={namespace="trueserve"}' \
--param since=5m --param limit=10 --param direction=backward
An empty successful query may simply mean no matching recent logs. Datasource listing alone does not prove permission to query every datasource. See official environment debugging for application filters, correlation and the correct Git branch.
| Result | Next step |
|---|---|
| Cannot read token file | Check the current user/home and repeat setup there |
| Permissions rejected | Re-run the helper to create the private file |
| HTTP 401 | Check environment, expiration and revocation; obtain a valid token |
| HTTP 403 | Ask the administrator to review service-account/datasource permissions |
HTTP 503 / no healthy upstream |
Investigate the upstream Grafana service; changing a token will not repair it |
| DNS error / timeout | Run python3 scripts/studio-access/probe.py --environments qa or dev |
5. Rotate or remove credentials¶
Obtain a replacement token, re-run setup for that environment, and verify a datasource/log query. Then revoke the old token in Grafana. For a shared credential, coordinate rotation across all cities using it before revocation.
To remove local copies:
rm -f ~/.local/state/trueserve/grafana-kms-dev.token
rm -f ~/.local/state/trueserve/grafana-kms-qa.token
For provisioned shared access, the city owner must also remove or rotate
.local-data/grafana/grafana-kms-dev.token and
.local-data/grafana/grafana-kms-qa.token as appropriate. Removing a personal
override alone enables fallback to the shared city credential.
What this achieves: removes this user's stored credentials. It does not revoke the service-account tokens in Grafana; that is a separate administrator action. Recheck token availability after devcontainer recreation, since home directory persistence depends on the city's mount configuration.
Reference: Grafana service accounts.