Skip to content

Runbook: migrate object storage from the shared MinIO to S3

Last reviewed: 2026-08-30

Freeze-writes → sync → flip. No dual-write and no read-fallback code exists, so the rollback window is manual (see step 8). Keys are preserved byte-for-byte at every step — nothing re-keys or re-publishes (cloned A2UI catalog versions share bundle objects by absolute key), and the data lands at bucket ROOT with no STORAGE_KEY_PREFIX — the local envs shared one MinIO keyspace, and the S3 buckets mirror that topology.

Status: executed for the VANGUARD env on 2026-08-30 (bulk sync live, ~4 min; freeze → zero-delta re-sync → flip → smoke, freeze window ≈ 10 min). City EC2s are still on the shared MinIO; each flips with the same steps, but the shared data is already in S3 — a city cutover is usually just steps 1 (IAM), 6, and 7.

Measured inventory (2026-08-30, the live shared MinIO at minio-api.trueserve.deloitte.com):

Source Objects Size Target
conversations/conversations/ + whisker-traces/ 4,872 ~12.3 GiB agent-studio-local-recordings
conversations/{catalogs,previews}/ (stray — the old MINIO_BUCKET collision) 5 241 KiB both recordings and catalogs
trueserve-catalogs/ 299 13.3 MiB agent-studio-local-catalogs
trueserve/surface-configs/ 2 474 B agent-studio-local-assets
S3 trusrv-<env>-orch-recordings (KMS) 220 225 MB agent-studio-<env>-recordings (same-account server-side copy)

1. Provision

scripts/storage/create-buckets.sh local --region us-east-1
scripts/storage/create-buckets.sh dev   --region us-east-1
scripts/storage/create-buckets.sh qa    --region us-west-2
scripts/storage/create-buckets.sh prod  --region us-east-1

Attach the runtime IAM policy the script prints (per env, to the IRSA roles). Verify a create-bucket attempt from a service role is denied — that is the proof ensure_bucket is gone.

2. Deploy the refactored code against MinIO first

Point the new seam at the OLD data before any data moves:

STORAGE_ENDPOINT=https://minio-api.trueserve.deloitte.com
STORAGE_FORCE_PATH_STYLE=true
STORAGE_REGION=us-east-1
STORAGE_BUCKET_RECORDINGS=conversations
STORAGE_BUCKET_CATALOGS=trueserve-catalogs
STORAGE_BUCKET_ASSETS=trueserve
STORAGE_ACCESS_KEY/SECRET_KEY=<the MinIO keys>

This validates the whole refactor against existing data, and is simultaneously the live proof the GCS-shaped configuration (custom endpoint + path style) works.

3. Freeze

Drain inbound calls, then stop the writers: ai-orchestrator, conversation-api (and its archive worker), media-proxy, async-agent, catalog-builder, agent-host, ai-manager.

4. Sync (key-preserving)

One aws s3 sync cannot span two endpoints — run MinIO→disk then disk→S3:

export AWS_ACCESS_KEY_ID=<minio key> AWS_SECRET_ACCESS_KEY=<minio secret>
MINIO="--endpoint-url https://minio-api.trueserve.deloitte.com"
aws s3 sync $MINIO s3://conversations/      ./stage/recordings/ --exclude 'catalogs/*' --exclude 'previews/*'
aws s3 sync $MINIO s3://conversations/      ./stage/stray/ --exclude '*' --include 'catalogs/*' --include 'previews/*'
aws s3 sync $MINIO s3://trueserve-catalogs/ ./stage/catalogs/
aws s3 sync $MINIO s3://trueserve/          ./stage/assets/

# switch to the AWS credentials / profile, then:
aws s3 sync ./stage/recordings/ s3://agent-studio-local-recordings/ --region us-east-1
aws s3 sync ./stage/catalogs/   s3://agent-studio-local-catalogs/   --region us-east-1
aws s3 sync ./stage/assets/     s3://agent-studio-local-assets/     --region us-east-1
# the 5 stray objects go to BOTH buckets — removes any doubt about who resolves them.
# ⚠ ORDER MATTERS in the catalogs bucket: every stray key COLLIDES with a real
# trueserve-catalogs key (three byte-identical, two STALE older preview builds),
# so the authoritative catalogs sync must run AFTER the stray sync or the stale
# copy clobbers the real bundle (surfaced as a 42.5 KB byte mismatch in the
# vanguard cutover verification — re-syncing ./stage/catalogs/ last fixed it).
aws s3 sync ./stage/stray/ s3://agent-studio-local-recordings/ --region us-east-1
aws s3 sync ./stage/stray/ s3://agent-studio-local-catalogs/   --region us-east-1
aws s3 sync ./stage/catalogs/ s3://agent-studio-local-catalogs/ --region us-east-1  # authoritative copy wins

KMS rename (same account + region, server-side):

aws s3 sync s3://trusrv-dev-orch-recordings/ s3://agent-studio-dev-recordings/ --region us-east-1
# qa/prod likewise; trusrv-*-orch-catalogs and -scale-testing are empty — just repoint.

5. Verify before unfreezing

  • aws s3 ls s3://<bucket> --recursive --summarize — object count + total bytes match the inventory table per bucket.
  • Re-run each sync with --dryrun — must report zero transfers.
  • Spot-check the largest objects and one attachment by CHECKSUM, not ETag — the target buckets are SSE-KMS, so S3 ETags are opaque (never a content MD5). Compare md5sum of the staged file against a re-download of the S3 object.

6. Flip

Set the Agent Studio buckets — and SET the interim vars EMPTY, never delete their lines: STORAGE_ENDPOINT=, STORAGE_FORCE_PATH_STYLE=false, STORAGE_ACCESS_KEY=, STORAGE_SECRET_KEY=. set -a; . .env.local overrides but cannot UNSET a variable a parent already exported, and on vanguard the overmind daemon (respawned mid-cutover) still carried the interim MinIO endpoint + keys — deleting the lines produced a HYBRID env: agent-studio bucket names presigned against MinIO → 404 on every catalog bundle, a day later. The seam treats empty as unset (AWS S3, default credential chain).

Restart the process SUPERVISOR too (overmind daemon / pod), not just the services, and unfreeze. Verify against the actual WORKER process (/proc/<python pid>/environ), not the launcher shell — the wrapper shows the pre-with-env inherited env and reads as stale even when the worker is fine.

7. Smoke

Each check exercises a path a config-only change would leave broken:

  1. WebRTC call via the phone emulator → three objects at conversations/<id>/{caller,agent,mixed}.wav in the recordings bucket.
  2. Open that recording in the conversation detail view and seek mid-file (presign + ranged GET).
  3. Publish an A2UI catalog via catalog-builder, load it in surface-app (cloned versions must still resolve shared bundle keys).
  4. Upload a surface-config image and a Teams agent icon (assets role, read + write).
  5. Run an OCR node and a managed-transcription node with a staged object on a pod with no static keys (the libs/ocr/libs/transcription credential chain).
  6. Trigger the conversation archive worker against a tenant sink with its own credentials (streamed cross-account copy).
  7. CloudTrail shows zero CreateBucket attempts.

8. Rollback window, then decommission

Leave the MinIO buckets read-only for a deprecation window. Rollback = flip the config back (step 2's values) and restart; anything written to S3 after the flip would need a reverse sync, so decide fast. After the window, revoke the MinIO credentials and delete our three buckets from the shared instance.