Skip to content

Migrate a city EC2 from MinIO to the S3 storage seam

Status: Current | Last reviewed: 2026-09-01

Applies to the *.trueserve.ai city dev boxes that predate the 2026-08-30 storage cutover (object-storage.md). Worked example: shimla, 2026-09-01 (the first city updated past the cutover).

The short version: there is (almost) no data to move

The MinIO-era topology was one shared MinIO at minio-api.trueserve.deloitte.com — every local env (vanguard + cities) pointed its MINIO_ENDPOINT there. The per-box tsf-local-minio container was never the data path (shimla's held 136 KB). When vanguard executed the cutover it synced that shared MinIO key-for-key into the shared agent-studio-local-* buckets at bucket root — recordings, catalog bundles, assets, including the cities' objects. So a city migrating today finds its objects already in S3.

Consequently the per-city migration is code + env only:

  1. Code: update the city branch past commit 1d17c627 ([storage] replace MinIO-specific access with object-store seam) — the normal update flow.
  2. Env: run city-update.sh <city> --step env. Since 2026-09-01 it deletes the dead MINIO_* keys (no reader at HEAD) and leaves STORAGE_* unset so lib-env.sh defaults apply: STORAGE_PROVIDER=s3, region us-east-1, the four agent-studio-local-<role> buckets, no key prefix (the migrated data lives at bucket root), no static credentials — the SDK default chain resolves the instance role.
  3. IAM: nothing to do. Every city box runs the same instance profile as vanguard (TrueServe-MegaDev2-Profile → TrueServe-MegaDev2-Role), whose inline TrueServeDevSharedBucketAccess policy grants the shared buckets.
  4. Restart (--step rebuild or restart) so processes pick up the env.

Verify

  • docker exec <devcontainer> bash -lc 'source .devcontainer/scripts/with-env.sh --summary | grep STORAGE' shows the seam vars, no MINIO_*.
  • Open an existing A2UI catalog in ai-manager and load a component preview (bundle keys are absolute and unchanged, so pre-cutover bundles must resolve).
  • Make a test call and play back its recording.
  • CloudTrail must show no CreateBucket events from the box (the seam has no ensure_bucket; one appearing is a code regression, not a config gap).

Why bucket ROOT and not a per-city STORAGE_KEY_PREFIX

The shared-MinIO topology never namespaced envs, so the migrated keyspace is flat and the object keys stored in every env's DB rows are absolute. Keeping all local envs at root preserves those references AND makes cross-env DB syncs cheap: rows copied from one env resolve their bundles/recordings without any object copying (this is how shimla received vanguard's catalogs with zero S3 work). A STORAGE_KEY_PREFIX is only for a genuinely NEW env that must not share the keyspace — never introduce one on a migrating city, or every stored key breaks.

If a city really does hold local-only MinIO data

Check first: docker exec tsf-local-minio du -sh /data. If a box grew real local data (it shouldn't have), replay vanguard's two-leg pattern — download from the local MinIO with its MINIO_* creds, aws s3 sync up to the role buckets key-for-key (keys must stay byte-identical; cloned catalogs share bundles by absolute key). Vanguard's leg scripts are preserved in the cutover record; the mapping is conversations→recordings, trueserve-catalogs→ catalogs, trueserve→assets, with conversations/{catalogs,previews}/* strays copied to both recordings and catalogs.

Rollback

The seam still speaks MinIO: set STORAGE_ENDPOINT + STORAGE_FORCE_PATH_STYLE=true + the bucket names/creds explicitly in .env.local and restart. The local MinIO container and its volume are left untouched by this migration.

See also