Recover a built-in catalog clobbered by the empty scaffold¶

Last reviewed: 2026-08-31

Use this when a built-in A2UI catalog (financial-services, standard-primitives, common-extended, retail, utilities, healthcare, insurance) has lost its real components — its surfaces render blank or show only the starter ExampleCard — after a seed-builtins run published the empty scaffold over it. This is the 2026-08-27 incident described in catalog-builder.md → scaffold-clobber guard.

The publish-layer guard now prevents the clobber from recurring, but a catalog already pointing at a scaffold 2.0.0 (or whose real bundle was never uploaded to the target bucket) needs an explicit re-publish from the baked real source.

The "blank / empty-bucket" case below is now auto-healed. As of 2026-08-31 catalog-builder runs an idempotent built-in self-heal on startup (and via POST /seed-builtins) that re-publishes any built-in whose current bundle is missing from this env's bucket. So the second row of the table (good pointer, bundle never uploaded) reconciles on the next deploy or pod restart with no manual step. This runbook is still the recovery for the first row (a scaffold 2.0.0 is the current version): the self-heal sees a ready bundle that does exist and leaves it alone, so a deliberate allowScaffoldReset re-publish from real source is still required.

Cloned (non-TrueServe) tenants are also auto-healed as of 2026-09-01. A tenant provisioned before the shared→tenant rescope inherited the source's pre-rescope bundle_url verbatim (catalogs/shared/<slug>/…) and 404s even after the source tenant is healed — the source-only self-heal never touched it. reconcileClonedBuiltins (the self-heal's second pass) re-points such a clone onto the source's healed bundle on the next deploy or pod restart — no manual step. It only touches clones whose own bundle object is missing, so a tenant's own edits are never overwritten.

Two distinct failure states¶

Both are fixed by the same action (re-publish from real source), but they look different:

Symptom	DB state	Cause
Preview renders, but shows only `ExampleCard`	`current_version_id` → a `2.0.0` whose `source_manifest` is the scaffold; its bundle is in the bucket	The seed publish succeeded against a stale scaffold and deleted every real component (`financial-services`, `standard-primitives`).
Preview blank (manifest URL 404s)	`current_version_id` → a real-component version, but that bundle was never uploaded to this env's bucket	The seed publish failed (swallowed) — the good pointer survived but the bundle isn't in the bucket. The original empty-bucket bug.

A bare current_version_id repoint does not fix either: the scaffold 2.0.0 bundle is wrong, and the good older bundles were never uploaded to this env's bucket. You must re-publish so a real bundle is (re)built and uploaded.

Why the real source is safe¶

The real component source is baked into the catalog-builder image at /app/catalogs/shared/<slug> and was never touched. Only the PVC workspace copy (/workspaces/catalog-tenant-trueserve-<slug>) held the empty scaffold. Confirm before recovering:

# In a catalog-builder pod:
ls /app/catalogs/shared/financial-services/components          # real: AccountOverview.tsx, …
ls /workspaces/catalog-tenant-trueserve-financial-services/components   # stale: only ExampleCard.tsx

Recovery¶

Run against one environment at a time; verify the connected DB/bucket before starting. All steps are ArgoCD-safe (no chart change is required — the fix is a data/PVC repair).

1. Restore real source onto the PVC¶

If the workspace components/ holds only ExampleCard.tsx, replace the directory with the baked source. From inside a catalog-builder pod (the app container that mounts the PVC):

for slug in standard-primitives financial-services; do   # add any other clobbered slugs
  t="/workspaces/catalog-tenant-trueserve-$slug"
  rm -rf "$t" && cp -a "/app/catalogs/shared/$slug" "$t"
  chown -R 1000:1000 "$t" 2>/dev/null || true
  # Images built before the 2026-08-28 fix bake an inert `tailwind.config.ts`
  # into each built-in source. Tailwind v4 catalogs are configured entirely in
  # styles.css (@source/@theme), and `assertSafeTailwindInputs` HARD-REJECTS a
  # JS/TS config — so leaving it in place makes step 2's publish fail with
  # `HTTP 500: JavaScript Tailwind config files are not allowed`. Delete it.
  rm -f "$t/tailwind.config.ts"
done

On a later chart rollout the source seeder now self-heals this same condition (values.yaml) — but do the manual restore now rather than wait for a deploy. Note the seeder re-copies the baked source verbatim, so on a pre-fix image it re-adds tailwind.config.ts on every pod restart; the permanent fix is to drop the file from the built-in sources in the image (done 2026-08-28).

2. Re-publish each catalog from the restored source¶

Publish by slug against the TrueServe tenant, version:"auto" — the auto path bumps past the taken 2.0.0 and uploads a fresh bundle. Because the source is now the real components (not the scaffold), the publish-layer guard does not fire:

TOKEN="$CATALOG_BUILDER_SERVICE_TOKEN"    # from trueserve-catalog-builder-secrets
for slug in standard-primitives financial-services; do
  curl -fsS -X POST "http://catalog-builder:8090/publish?bySlug=true" \
    -H 'content-type: application/json' \
    -H "authorization: Bearer $TOKEN" \
    -d "{\"slug\":\"$slug\",\"scope\":\"tenant\",\"tenantId\":\"00000000-0000-0000-0000-000000000001\",\"version\":\"auto\"}"
done

Order matters: publish standard-primitives first — the other built-ins render their components against its primitives.

3. Verify¶

# current version now carries real components again (not just ExampleCard):
psql "$DATABASE_URL" -c "
  SELECT cat.slug, v.version, v.bump_kind,
         (SELECT string_agg(c->>'name', ',')
            FROM jsonb_array_elements(v.source_manifest->'components') c) AS components
    FROM a2ui_catalogs cat
    JOIN a2ui_catalog_versions v ON v.id = cat.current_version_id
   WHERE cat.slug IN ('standard-primitives','financial-services');"

Then open a surface that uses each catalog and confirm the real components render.

Don't forget qa / prod¶

The same stale scaffold likely sits on the qa and prod PVCs. Their built-ins were spared only because their seed publishes happened to fail — a clean re-seed there would clobber them. Before any catalog re-seed on qa/prod, either deploy the seeder-heal + publish guard (this change) or run step 1 there first. Confirm each env's PVC with the ls check above.