Skip to content

Runbook: rotate master encryption key

Status: Current | Last reviewed: 2026-07-24

Rotate MASTER_ENCRYPTION_KEY. Re-encrypts every encrypted row under a new key.

When to run

  • Annual rotation per security policy.
  • After suspected compromise of the existing key.
  • When a key holder leaves the team and the policy says rotate.

Prerequisites

  • Current MASTER_ENCRYPTION_KEY value (the one in production).
  • A new 32-byte base64 key (generate with openssl rand -base64 32).
  • Direct DB access (the rotation script connects via DATABASE_URL).
  • Maintenance window — encrypted writes during rotation will use the old key and need a follow-up pass.

Steps

  1. Back up the database: bash pg_dump "$DATABASE_URL" > /tmp/backup-pre-rotation-$(date +%Y%m%d).sql

  2. Set both keys in the environment: bash export OLD_MASTER_ENCRYPTION_KEY="<current key>" export NEW_MASTER_ENCRYPTION_KEY="$(openssl rand -base64 32)"

  3. Run the rotation script: bash python scripts/migrations/rotate-master-encryption-key.py The script:

  4. Iterates integrations.encrypted_config, tools.encrypted_config, tool_secrets.value.
  5. Decrypts with OLD_MASTER_ENCRYPTION_KEY.
  6. Re-encrypts with NEW_MASTER_ENCRYPTION_KEY.
  7. Updates rows in place.
  8. Verifies a round-trip on each row.

It does not cover the secrets encrypted inline in tenant_settings JSON (Telemetry secret headers enc:{}, Knowledge api_token_enc — see secret-storage.md). Re-enter those from ai-manager Settings → Tenant Settings after the swap, or extend the script to walk tenant_settings.setting_value.

  1. Swap the key in production. For Vanguard: bash # Edit .env.local — set MASTER_ENCRYPTION_KEY to the new value bash .devcontainer/wake-up.sh For Sentinel/cluster: bash kubectl create secret generic trueserve-master-key \ --from-literal=MASTER_ENCRYPTION_KEY="$NEW_MASTER_ENCRYPTION_KEY" \ --dry-run=client -o yaml | kubectl apply -f - kubectl rollout restart deployment -n trueserve -l 'app in (ai-manager,ai-orchestrator,agent-host,async-agent,catalog-builder,config-sync-service)'

  2. Verify by reading any integration through ai-manager:

  3. GET /api/integrations/<id>/test — should return success.

  4. Existing voice agents should make calls without auth errors.

  5. Securely store the new key in your KMS or password manager. Do not keep the old key around — destroy it after a one-week confidence window.

If something goes wrong mid-rotation

The script tracks progress per row. If it crashes:

python scripts/migrations/rotate-master-encryption-key.py --resume

If a verification round-trip fails for a row, the script aborts and leaves prior rows already re-encrypted with the new key. To recover:

  • The DB now has a mix of old- and new-key rows.
  • Restart the rotation; it'll skip already-rotated rows by checking decryption against the new key first.

Why losing the key is unrecoverable

There is no escrow. Lose the key, lose every encrypted credential. Customers must re-enter every integration credential by hand. See secret-storage.md.

See also