Phase 4 — Data Migration¶
Historical record
Retained as evidence of how the Azure platform was built. Not an active runbook for starting a new migration.
The decision log for moving Firestore and Firebase Storage onto Cosmos DB and Blob Storage. The operator sequence is the Migration-Runbook; this page records why each thing is the way it is, so a decision is never re-litigated from a log line. Plan-level context: Migration-Plan §5.
Baseline: Site-Main @ 088f458 (2026-08-18, v1.7.0) — 68 Firestore collections, one GCS bucket.
Target: 74 Cosmos containers (73 generated from the manifest + leases, declared separately in
infra/main.tf), 5 blob containers.
Decisions¶
Retired 2026-08-24 — read D1 and D2 in the past tense. The migration execution surface is gone:
migrate-data.ymland the five scripts were deleted in59e471b. The rehearsal estate and the three production-write grants are authorised for removal and deleted from the configuration, but the apply that carries it out has not run — as of 2026-08-25 the sandbox and the grants are still live in Azure. D1 therefore describes an account that is on its way out, and D2 a gate variable that no longer exists in code at all. Both are kept because they are the record of why the rehearsal was built the way it was — a keyless account so the rehearsal exercised the same auth path production takes, and RBAC rather than a workflow guard as the production lock. Q5 and P6 below carry the closing decisions.
D1. The rehearsal account is keyless, serverless and identical in shape to production¶
cosmos-site-sbx-cus (infra/scratch.tf) has keys off, the same
firewall shape, the same database name hcw, and the same 73 containers from the same generated
spec. A key-authenticated rehearsal against an open account would pass while proving nothing about
DefaultAzureCredential + native RBAC — which is the path production takes. The healer's 2026-08-20
failure ("cannot be authorized by AAD token in data plane", see D9) is exactly the class of defect a
key would have hidden.
It differs only where a sandbox should: its own resource group, no prevent_destroy, the CAF sbx
environment token in every name. It holds a full copy of production data while on; the variables
that create it also destroy it. Lifetime: recorded at Q5, closed 2026-08-24
— the estate is deleted outright rather than switched off, because the switches
are deleted with it.
D2. Production is locked by RBAC, not by a guard¶
While migration_writer_enabled is false (the default), the deploy identity holds no
database-scope Cosmos role and no blob-write role on production. The workflow's
"refuse write modes against production" step is the second lock, not the only one. The
production-import phase opens with that one variable, reviewed in HCP Terraform.
D3. Partition keys — the window is open now and closes on the first import¶
All 73 containers are empty as of 2026-08-20. A partition-key path is immutable; changing one after
data lands means destroy and re-import. Current choices: 67 on /id; content_versions on
/contentId, image_prompts_sets on /pageId, image_prompt_sets_prompts on /setName,
listen_and_learn_episodes on /setId (each assigns ids unique only within its parent — flattening
under /id would silently overwrite on upsert); admin_config on a constant /configScope so the
ContentForge save stays one TransactionalBatch. Owner sign-off on this list is a gate before
runbook step 9 writes anything — even to scratch, because the rehearsal should exercise the final shape.
D4. Collection dispositions¶
The manifest (scripts/lib/migration-manifest.mjs) classifies every entry — the 68 collections plus the subcollections it flattens — as one of five dispositions (counts as of 2026-08-20):
- migrate (55) — exported, transformed, imported, reconciled. Includes
admins,admin_config,config,_snapshotsand the four flattened subcollections. - regenerate (3) — derived data a ported job rebuilds:
homepage_feeds,tool_service_cache,rss_cache. Migrating it would carry stale derived state across; the jobs that rebuild it are Phase 3 work. - reseed (2) — seed data, re-run the seeder on Azure:
azure_landing_content,tool_service_catalog. - transient (5) — job and quota records with no value after cutover:
lab_jobs,tool_export_quota,tool_ai_plan_quota,submission_quota,lab_public_quota. - probe (15) — named so the preflight does not flag them as unmanifested, but not provisioned as
containers and not migrated until someone decides:
articles,metadata,users, the fivesocial_*collections (social_workspaces,social_libraries,social_library_items,social_schedule_slots,social_analytics), the two seeder-written ones added 2026-08-20 (azure_architectures,azure_frameworks), and five the first live preflight surfaced on 2026-08-21 —_rowy_(3 docs, Rowy GUI metadata),admin_audit_log(1, the pre-FINDING-07 singular),dashboard_stats(1, derived counters from themaintainDashboardStatstrigger),drafts(1),summaries(1). All ten of the original probes are empty in Firestore, so the real decisions are the five new ones — and none needs a container: our port already keeps the dashboard document assystem/dashboard_stats_v1and the ported trigger recomputes it. Runbook step 8 decides each from its measured count: content the site reads becomesmigrateand gets a container; anything else becomestransientor is dropped from the manifest.
preflight reports probe entries with their counts and exits 2 only on a collection the manifest
does not name at all — so a new upstream collection stops the phase rather than being silently
dropped, while a probe is a question, not a failure.
D5. admins — uid → oid remap is a later, human-reviewed step¶
Firebase uids and Entra object ids are unrelated. The rehearsal migrates admins faithfully
(keeping firebaseUid); the production import adds a --remap mapping file that a human reviews.
Not in this phase.
D6. Storage — manifest-driven, faithful, no URL rewriting¶
scripts/lib/storage-manifest.mjs maps each GCS top-level
prefix to one of the five Terraform blob containers. covers/, blogs/, certifications/,
speakerevents/ → same-named container, prefix stripped; database/{certifications,blogs,speakerevents}/
→ the family's container under database/; image-gallery/, character/, listen-and-learn/,
draft-images/, published-images/ → content, prefix preserved. Skipped: articles/ (90-day
scraped images the RSS job regenerates — note the Azure lifecycle rule for it is inert until the
scraper writes here) and uploads/ (per-user temp keyed by Firebase uid). Probe: thumbnails/ (empty),
content-submissions/ (3 objects, public-submission images) and designs/ (1 object) — the last two
surfaced by the first live inventory on 2026-08-21.
Flag for the owner: published-images/ is publicly readable in Firebase; content is not in
the API's public-media container list here. That is a disclosure decision for the API, not the copy.
The copy carries contentType/cacheControl and stamps gcsmd5/gcsgeneration/gcssource
metadata; it is idempotent by gcsmd5 match. imageUrl/storagePath values inside documents are
not rewritten — Firebase Storage stays warm until Go-Live and the re-pointing is its own
reviewed step.
D7. Public repository → summaries only¶
The repository is public. The original workflow uploaded scripts/reports/ — document ids and
240-character field samples — as an artifact, and the import dry-run printed samples to the log.
Both would have published production data. Now: every script writes a *.summary.json beside its
full report; only summaries are uploaded (1-day retention); MIGRATION_CI=1 makes --show-samples
an error; the export lives in $RUNNER_TEMP and dies with the runner; the upload step refuses any
non-summary JSON. Full reports for a rehearsal that needs keeping go to the private
migration-reports container on scratch storage, never to an artifact.
D8. Credentials — federated on both clouds¶
GCP through Workload Identity Federation (a dedicated read-only service account; no downloaded key
— the scripts refuse a service_account credential file in CI). Azure through the
environment:data-migration federated credential on the deploy identity. The Cosmos account has
keys disabled, so COSMOS_KEY cannot work and the client refuses it if set.
D9. The healer is broken and it is not this phase's problem¶
heal-computed-properties.yml fails with 403 "cannot be authorized by AAD token in data plane" on
PUT /dbs/hcw/colls/content. container.replace() (setting computedProperties) is a
control-plane operation; a Cosmos data-plane role can never satisfy it. It needs an ARM role
(Cosmos DB Operator on the account) or a move to az cosmosdb sql container update. Tracked in
TODO; cp_sortDate only matters once content/blogs hold data, which is the production-import phase.
D10. Why the probe exists¶
A 403 from Cosmos has two unrelated causes — the runner is not admitted by the firewall, or the
identity reached Cosmos and lacks a database-scope role — and they are indistinguishable from the
SDK error. Without the probe, a rehearsal exports everything and fails on the first upsert with an
error naming neither. migration-probe.mjs runs one SELECT VALUE COUNT(1) against system first and classifies
the failure. system, not content: the deploy identity holds container-scoped grants on content
and blogs for the healer, so those two answer on production even without the database-scope role
(run 32438525274 proved it). In runbook step 11 against production, cause: rbac on system is the
expected result — it is the proof that the production lock holds.
D11. The storage firewall window must open the default action, not just an IP rule¶
Azure documents it (Storage firewall limitations, item 5): IP network rules have no effect on requests that originate from the same Azure region as the storage account. GitHub-hosted runners are Azure VMs. Every scratch copy had drawn a runner outside centralus; the first production copy drew one inside it and 403'd on its first listing with the roles provably live. So migrate-data.yml (PR #134) sets the account's default action to Allow for the minutes of the copy alongside the IP rule, and the always() close step restores Deny first and fails if it is not back. Exposure during the window is network reachability only: every data-plane call still needs an Entra token with a blob role, and allowBlobPublicAccess is false on both accounts. The same latent flaw exists in deploy-functions.yml (TODO T-509). Cosmos never had this problem — the 0.0.0.0 datacenter sentinel exists for exactly this reason.
D12. Two collections are rewritten by machines every few minutes — the cutover delta run must follow their pause¶
The post-import verify (P4) found social_posts and lab_agents drifted within twenty minutes of the import while every other container still matched. Both have non-human writers in Site-Main: reconcilePublerCalendar, run by syncSocialCalendarScheduled every 5 minutes, writes lastSyncedAt (and publerStatus / syncStatus when Publer changes) on every social post; the VPS agent re-set()s its own lab_agents document on heartbeat. A write-freeze on the admin UI does not stop either. So the cutover sequence (Migration-Plan §6) must disable the Publer sync timer and stop the VPS agent before the delta import, then run rehearse target=production once more and verify — only then is failed: 0 on all 62 achievable. lab_agents is arguably transient (the re-pointed agent re-registers itself on Azure); decide at cutover. Until then the production copy is complete for everything a visitor or an editor sees.
Open questions (owner)¶
| # | Question | Decided at | Answer |
|---|---|---|---|
| Q1 | The fifteen probe entries (D4): which migrate, which are residue? Preflight 2026-08-21: the original ten are empty; the five new ones hold 7 documents between them and none has a reader |
runbook step 8 | Decided 2026-08-21: none migrates. Entries stay as probe so the gate passes for a reason |
| Q2 | Storage probes: thumbnails/ (empty — drop), content-submissions/ (3 objects), designs/ (1): copy into content or drop? |
runbook step 10 | Decided 2026-08-21: thumbnails/ → skip; content-submissions/ and designs/ → migrate into content, prefix preserved |
| Q2b | covers/ is 3.10 GiB of the 3.17 GiB bucket — 1,011 AI/uploaded covers for 1,142 content documents. Copy all, or only covers still referenced by a document? |
before step 10 copy | Decided 2026-08-21: copy all. Referenced-only pruning is a later cleanup with the document set in hand |
| Q3 | published-images/ public on Azure? |
before Go-Live | Signed 2026-08-21: not public. The prefix holds 0 objects; content stays out of PUBLIC_MEDIA_CONTAINERS. Revisit only if the ported publisher starts writing there |
| Q4 | Partition-key list (D3) signed? | before step 9 | Signed 2026-08-21. Spec and the live scratch account agree: 67 on /id; admin_config /configScope, content_versions /contentId, image_prompts_sets /pageId, image_prompt_sets_prompts /setName, listen_and_learn_episodes /setId. Window closes at the first production import |
| Q5 | Scratch copy lifetime after sign-off? | step 12 | Decided 2026-08-21: keep through the production dress rehearsal. Flip cosmos_scratch_enabled / storage_scratch_enabled off after the production import is verified. Closed 2026-08-24: it was never flipped off and the readiness review found it still live holding a full copy of production. The owner authorised removal; the estate is deleted rather than switched off, and both switches are deleted with it — P6 |
Evidence log¶
| Date | Step | Artifact / link | Result |
|---|---|---|---|
| 2026-08-20 | 1 | PR #128 — tooling, scratch.tf, workflow, docs | CI green |
| 2026-08-20 | 2 | SA hcw-migration-reader; provider github-actions/providers/github-actions-hcw (repo id 1268997852, main only); repo variables GCP_* |
done |
| 2026-08-20 | 3 | Environment data-migration, reviewer saulpatinojr. Site-Main read token: not yet (needs the GitHub UI — App or PAT) |
partial |
| 2026-08-20 | 4 | TFC cosmos_scratch_enabled / storage_scratch_enabled = true; applied: 86 add, 1 change, 0 destroy; set-github-variables.ps1 seeded the scratch variables and moved COSMOS_ENDPOINT to a variable |
done |
| 2026-08-21 | 5 | Run 32435842524 mode=preflight — WIF proven; 8,064 documents, 8,004 to migrate; exit 2 on five unmanifested collections (added as probe, PR #130). Summary artifact verified: counts only |
gate loop |
| 2026-08-21 | 5 | Run 32436854557 mode=preflight after #130 — exit 0, no unmanifested collections, 8,064 / 8,004 / 60 |
pass |
| 2026-08-21 | 7 | Run 32437095217 mode=export-dry-run — 8,023 documents across 62 collections (8,004 + 19 subcollection docs); warnings: id-field-conflict 60, id-collision 0 |
pass |
| 2026-08-21 | 9 | Run 32437751076 mode=rehearse target=scratch — Azure login via the environment:data-migration credential; probe reached content on Entra auth in 1.3 s; dry-run 8,023 across 62; import 8,023/8,023, 0 failed; reconciliation 62 containers, 0 missing, 0 extra, 0 field mismatches. First pass, no retries. Summaries verified counts-only |
pass |
| 2026-08-21 | 10 | Run 32438131444 mode=storage-inventory — 1,438 objects, 3.17 GiB (covers/ 1,011 objects / 3.10 GiB); exit 2 on three unmanifested prefixes: database/{blogs,speakerevents}/ (13 objects, added as migrate), content-submissions/ (3) and designs/ (1) (added as probe). thumbnails/, draft-images/, published-images/ are all empty |
gate loop |
| 2026-08-21 | 11 | Run 32438525274 mode=verify target=production — 60 of 62 containers refused executeQuery (no database-scope role); content and blogs readable through the healer's container grants and empty (0 of 1,142 / 0 of 242). Production is empty and locked. Found a probe flaw: it checked content, which the healer grant makes readable, so it said OK instead of rbac — probe moved to system |
pass (run shows failed by design) |
| 2026-08-21 | 10 | Run 32439102543 mode=storage-inventory after #131 — exit 0, every prefix manifested, 1,438 objects / 3.17 GiB |
pass |
| 2026-08-21 | 10 | Run 32439235702 mode=storage-rehearse target=scratch #1 — firewall window opened/closed by the run; 1,438 copied, 0 unchanged; verify: every prefix matches on count, bytes, stored MD5 and a 5-per-prefix byte compare; 3.17 GiB in 2 min 35 s |
pass |
| 2026-08-21 | 10 | Run 32439595190 mode=storage-rehearse target=scratch #2 — 0 copied, 1,438 unchanged (idempotency by gcsmd5); verify identical |
pass |
| 2026-08-21 | 6 | Inventory gate run in the local two-clone form (no token needed) against Site-Main 088f458: inventory-collections.mjs --diff → "every collection this repo touches is known to the manifest"; 85 ids known; the 17 manifest-only names are the probes, the flattened config/* children and the transient quotas. The workflow mode stays available once a read token exists |
pass |
| 2026-08-21 | 12 | Signed. Q1, Q2, Q2b, Q5 decided; Q3 and Q4 signed (below). The scratch rehearsal phase is closed; the production-import phase is opened by migration_writer_enabled = true |
signed |
| 2026-08-21 | P1 | migration_writer_enabled = true applied by the owner: 3 added, 0 changed, 0 destroyed (database-scope Cosmos role, Blob Data Contributor, Storage Account Contributor on stsiteprodcus01). PRODUCTION_IMPORT_ENABLED = true set. Write-freeze on Site-Main admin from 03:44Z |
done |
| 2026-08-21 | P2 | Run 32444447404 mode=rehearse target=production — probe on system answered in 976 ms; export 8,023 (identical to the scratch run — nothing changed in Firestore in between); imported 8,023/8,023, 0 failed; reconciliation 62 containers, 0 missing, 0 extra, 0 field mismatches; 1 min 38 s |
pass |
| 2026-08-21 | P3 | Run 32444649912 mode=storage-rehearse target=production #1 — failed on the first listing with the storage network 403. Roles were live; the IP window step had succeeded. Cause: Azure ignores storage IP rules for same-region traffic and the runner was in centralus (D11). Nothing written |
fail → D11 |
| 2026-08-21 | P3 | Run 32444817361 mode=storage-rehearse target=production #1 re-run — out-of-region runner; 1,438 copied, 0 unchanged; every prefix verified on count, bytes, MD5 and sampled content; 2 min 58 s |
pass |
| 2026-08-21 | P3 | Run 32445335767 mode=storage-rehearse target=production #2 — first run on the D11 window (PR #134): 0 copied, 1,438 unchanged, every prefix re-verified; close step restored Deny, no IP rule left behind (checked live) |
pass |
| 2026-08-21 | P4 | Run 32445527271 mode=verify target=production — fresh export 8,023; 60 of 62 containers reconcile; social_posts (15/15) and lab_agents (1/1) show field mismatches with counts, ids and everything else matching. Cause confirmed in Site-Main source: reconcilePublerCalendar (the 5-minute syncSocialCalendarScheduled) stamps lastSyncedAt on every social post each run, and labs/vps-agent/index.js:62 re-set()s the agent document on heartbeat. Live machine writers, not a defect — D12 |
pass with D12 |
| 2026-08-21 | P5 | PRODUCTION_IMPORT_ENABLED deleted (API 404 confirmed). migration_writer_enabled left true for the cutover delta run — owner to flip after cutover |
closed → superseded by P6 |
| 2026-08-24 | P6 | The delta import is retired and the execution surface is gone. migrate-data.yml and the five migration scripts were deleted in 59e471b; the same commit staged these pages to the Wiki. The owner confirmed that day that the rehearsal is finished, and authorised both removals: the three migration_writer_enabled grants are revoked, and rg-db-site-sbx-cus — cosmos-site-sbx-cus with 73 containers and a measured 77,763 documents, stsitesbxcus01 with 6 containers, 4 role assignments — is torn down irreversibly, since a Cosmos account cannot be restored from its own continuous backup. All three variables are deleted rather than set false: the checked-in false was demonstrably not the effective value, so only removing the declarations settles it. Consequence: anything written on Firebase after the 2026-08-21 import does not come across, and the cutover moves DNS without a second pass (Cutover-Runbook step 4). Authorisation recorded in TODO.md |
retired |