Deployment Runbook — Azure platform (infra/)

Scope: provisioning and operating the Azure platform from infra/ in the main repository. Migration and DNS cutover are covered by the Migration Runbook; this page owns the Terraform lifecycle: validate → plan → apply → verify → operate → (eventually) ALZ absorption.

Authority: this runbook does not authorize anything. Production applies, destructive changes, DNS cutover, and decommissioning each require the explicit approvals listed in the root README and Migration Runbook roles table.

System of record

Concern	Where
Terraform source	infra/ on main in HCW-HybridCloudWorks
State and variables	HCP Terraform Cloud — org hcw, project Site, workspace hcw-azure
Required inputs (names, formats, consumers — never values)	Required-Inputs in this Wiki (was REVIEW.md Part 4 at the root until 2026-08-29)
Terraform's own identity	id-plat-terraform-prod-cus-01, federated to app.terraform.io — created once by scripts/bootstrap-terraform-oidc.ps1, outside Terraform state (section 0)
Deployment identity	User-assigned managed identity + GitHub OIDC federated credentials (infra/oidc.tf) — no static credentials exist
Working rules for the directory	infra/README.md

0. Bootstrap — once per subscription, before anything else

Everything below this section assumes HCP Terraform can already authenticate to Azure. On a subscription that has never been applied to, it cannot, and no amount of Terraform fixes that: Terraform cannot create the credential Terraform authenticates with. This section breaks that chicken-and-egg. It runs exactly once in the life of a subscription.

The two handshakes

Confusing these is the most common way to get stuck, because both are called "the OIDC setup" and only one of them is in the repository.

	HCP Terraform → Azure	GitHub Actions → Azure
Who authenticates	Terraform runs in HashiCorp's cloud	The deploy workflows
Created by	scripts/bootstrap-terraform-oidc.ps1 (manual, once)	infra/oidc.tf (Terraform)
Identity	id-plat-terraform-prod-cus-01 in rg-mgmt-boot-prod-cus	id-site-github-deploy-prod-cus-01 in rg-web-site-prod-cus
Issuer	https://app.terraform.io	https://token.actions.githubusercontent.com
Exists when	After you run the script	After the first successful apply
Consumed as	TFC_AZURE_RUN_CLIENT_ID in the workspace	CLIENT_ID repository variable

If you are hunting for CLIENT_ID to give azure/login, it comes from the Terraform outputs after the left-hand column works. It does not exist before the first apply.

Why a managed identity, not an app registration

Same reason as infra/oidc.tf, which documents it at length: app registrations need Application Administrator in Entra, and Azure Owner does not grant that — Entra and Azure RBAC are separate permission planes. A user-assigned managed identity is an ordinary Azure resource, and Entra supports federating one to an arbitrary external issuer, so an Azure Owner can create the whole chain with no directory role at all.

Why it is not in Terraform state

The bootstrap identity is deliberately excluded from infra/. If Terraform managed the identity it authenticates with, a destroy, a taint, or a bad plan would lock the workspace out of the subscription with no path back in except re-running this script. It lives in its own resource group for the same reason: nothing in infra/ can reach it.

Run it

# Dry run first — prints every change without making one.
./scripts/bootstrap-terraform-oidc.ps1 -WhatIf

No arguments, by design. Every value the three deployment scripts need is a GUID, and GUIDs passed as flags go wrong in ways that surface later as something else: they land in shell history, one transposed character reads as a permissions problem, and the operator has to know which of four similar subscription IDs belongs in which slot from a terminal that cannot show them the list. So the scripts discover what Azure already knows, offer a numbered list where there is a real choice — with the subscription matching the naming convention preselected — and prompt only for what cannot be found. Everything resolved is printed for one confirmation before anything is written.

Here that means the tenant comes from your az sign-in, the identity's home is matched from sub-plat-mgmt-*, and the deployment targets default to the three subscriptions the configuration actually targets (app, mgmt, conn — Identity is deliberately excluded, since that landing zone holds nothing).

Parameters still exist for every value, so CI can supply them; they are simply never required. -DeviceCode remains for a session with no browser.

The script is idempotent, so re-running it is how you repair a broken handshake, not just how you create one. It preflights before it proposes anything: CLI present, signed in, correct tenant, subscription visible, role-assignment rights held, Microsoft.ManagedIdentity registered.

Sign-in is handled for you. If you are not signed in, or your session is in a different directory — the normal state for anyone who works across tenants — the script runs az login --tenant itself and re-reads the account afterwards, because switching directories also changes which subscriptions are visible. Signing in happens even under -WhatIf: it reads your directory rather than changing it, and nothing can be inspected without a session.

Add -DeviceCode when this session has no browser of its own (SSH, a container, Cloud Shell, a locked-down VM), or when the browser that opens keeps silently reusing the wrong account. You get a short code and a URL to complete in any browser, on any machine. The script also falls back to it automatically if the interactive sign-in fails, since that failure is usually environmental — no display, no loopback — rather than a credential problem.

If the preflight says you hold no roles on the subscription: that is expected on a tenant you created yourself. Global Administrator is an Entra role and carries zero Azure RBAC. Re-run with -ElevateAccess, which takes the documented one-time root-scope elevation, grants you Owner on the target subscription, and removes the root-scope grant again.

It creates: rg-mgmt-boot-prod-cus, the id-plat-terraform-prod-cus-01 managed identity, two federated credentials, and two subscription role assignments (Contributor to create resources, Role Based Access Control Administrator to create the role assignments infra/ declares — Contributor alone cannot, and RBAC Administrator cannot grant Owner, so the identity cannot escalate itself).

Two federated credentials, not one. HCP Terraform stamps the run phase into the token subject, and Entra matches subjects as exact case-sensitive strings with no wildcards, so run_phase:plan and run_phase:apply are two different subjects. With only the plan credential every run plans cleanly and every apply fails at authentication — which reads like a permissions problem and is not one.

Then set the workspace variables

In HCP Terraform → hcw-azure → Variables, as environment variables (the script prints these with the values filled in):

Name	Value
TFC_AZURE_PROVIDER_AUTH	true
TFC_AZURE_RUN_CLIENT_ID	client ID of id-plat-terraform-prod-cus-01
ARM_TENANT_ID	tenant GUID
ARM_SUBSCRIPTION_ID	subscription GUID

These four names come from HashiCorp and Microsoft and are exempt from the 2-word variable rule as contractual names. Terraform variables for the same workspace are listed in Required-Inputs §4.2.

Both seeding halves are scripted — prefer the scripts over the UI forms, and both take no arguments for the reasons given in section 0:

• scripts/set-tfc-variables.ps1 writes all twelve HCP Terraform workspace values (the four environment variables above plus the eight Terraform variables) in one idempotent run, and reads back the workspace's real project name — the value the federated-credential subject must contain. It finds the Terraform identity's client id by reading the identity the bootstrap created, so that value is never copied by hand out of a scrolled-away console; the subscriptions come from az; the Cloudflare zone is chosen from the zones the token can actually see, after prompting for the token itself. Only the app-registration audience and the Cloudflare token are typed, and only when they cannot be discovered.
• scripts/set-github-variables.ps1 seeds the GitHub repository variables and secrets. Run it once before the first apply — that seeds only TENANT_ID and SUBSCRIPTION_ID, the two values that are inputs to Terraform rather than products of it — and once after, when it reads CLIENT_ID, APP_HOSTNAME, FUNCTIONS_URL, RESOURCE_GROUP, FUNCTIONS_STORAGE_ACCOUNT and the COSMOS_ENDPOINT secret straight from the workspace's state outputs over the HCP Terraform API. Outputs rather than hardcoded copies on purpose: a copy drifts silently when the code changes, an applied output cannot. The corollary is directional — re-run the script after any apply that changes an output (a renamed group, a new hostname), or the GitHub-side copies go stale. The first post-apply run is also what arms the self-skipping heal-computed-properties schedule.

Verify

cd infra && terraform login && terraform plan

A plan that authenticates and shows resources to create is success — you are not applying yet. AADSTS70021 ("No matching federated identity record found") means the subject did not match: re-run the script passing -TfcProject and -TfcWorkspace copied exactly off the workspace Settings page, capitalisation and spaces included. A workspace created without choosing a project is in Default Project, with the space.

Bootstrap is done when a plan authenticates. Continue from section 1.

1. Preflight (every change)

1. Branch from main; never push to main directly.
2. Local validation, no credentials needed:

   cd infra
   terraform fmt -recursive -check
   terraform init -backend=false -input=false
   terraform validate
   tflint --init && tflint
   

3. CI must be green: IaC Validation (fmt/validate/tflint/Trivy), Repository Policy, CI, CodeQL.
4. PR uses the infrastructure section of the template: plan linked, no unexpected destroy/create pairs, no address renames without moved blocks, tags on every new resource, Required-Inputs updated for any new required input.
5. If the change alters an accepted ADR, write the superseding ADR first (register).

2. Plan

Plans run in HCP Terraform Cloud, where the state and the workspace variables live — not on laptops, not on GitHub-hosted runners holding tokens.

1. Open a run in the hcw-azure workspace (VCS-triggered or CLI-triggered from the merged commit).
2. The infrastructure operator reviews the plan in TFC, checking:
3. zero destroy/create pairs on stateful resources (Cosmos, storage accounts, Key Vault carry prevent_destroy — a plan that wants to replace them fails; treat any attempt as a defect, not an obstacle). They were lifted exactly once, for the centralus rebuild on 2026-08-19, and restored the same day. If a plan proposes replacing a stateful resource and no one has deliberately lifted a guard, stop;
4. every change traceable to the merged diff;
5. cost-relevant changes against the USD 150/month ceiling (Cost analysis).
6. Anything surprising: discard the run, fix in a new PR.

3. Apply

Before confirming, if the run creates a budget: check budget_start_date. Azure rejects a monthly budget whose start date falls outside the current month, and the constraint is checked on create, so an existing budget is unaffected while a new one fails. Set the workspace value to the first of the month the apply actually lands in. Otherwise the failure arrives at the end of the graph, after everything else has run, for a reason unrelated to anything under review.

1. Apply is confirmed in HCP Terraform by a human who is not the change author where role separation permits.
2. The GitHub delivery workflow (deploy-infra.yml) stays hard-disabled until production applies are authorized. When that authorization lands, enable it as designed: workflow_dispatch-only, production-infra GitHub Environment with required reviewers, TFC still holds the apply confirmation. Enabling is a two-step, reviewed change documented in the workflow header.
3. Record in the run description: PR number, approver, and (for anything touching data-bearing resources) the rollback decision point.

4. Post-apply verification

Do these in order. Step 1 is first because it is the only one where a problem means production is already degraded rather than merely unchanged.

1. Assert the Function App is not degraded. Any apply that writes to azurerm_function_app_flex_consumption.hcw — including one that only changes a site setting — has azurerm re-inject a keyless AzureWebJobsStorage, which the azapi pair strips two graph hops later. A failure between those two ARM calls leaves the site in the state three recorded incidents came from, and no alert rule detects it (Alerting and support). Expect AzureWebJobsStorage absent and RUNTIME_CONFIG_WRITER equal to azapi-strip; the commands are on that page. Re-apply to convergence if either is wrong — do not edit the setting by hand.
2. terraform plan again → empty plan (no immediate drift). Expect the permanent 3-add / 1-change / 3-destroy signature from the two azapi resources and the FTP policy; infra/main.tf documents it beside them.
3. Prove alert delivery, if the run created or changed an alert rule. Two tests, answering different questions, and neither substitutes for the other:
4. az monitor action-group test-notifications create against ag-plat-prod-cus-01 in rg-mgmt-plat-prod-cus — pass --subscription explicitly or it resolves in the wrong one. This exercises the action group's own receivers.
5. Then make an application-subscription rule actually fire once: temporarily lower function_response_time or cosmos_throttled to a threshold certain to trip, wait one evaluation window, restore. Only this exercises a rule in one subscription invoking an action group in another, which is the hop nothing has yet proven.

A rule that exists and pages nobody is worse than a visibly empty alert inventory, because it looks fixed. 4. Smoke: from the repository root, node scripts/smoke-deployed.mjs (see script header for flags) — anonymous surface filtered, admin guards refusing, health endpoint answering. Run it from an operator machine, not from Actions. The Validate Deployed Surface workflow (.github/workflows/validate-deployed.yml) still runs the DNS, TLS and frontend-surface job usefully, but its smoke job cannot pass: both jobs execute on a GitHub-hosted runner whoever dispatches them, and through Cloudflare that runner is answered by Bot Fight Mode with a 403, while direct to the origin it is answered by the origin lock with a 403. deploy-functions.yml depends on exactly that behaviour — it fails the deploy if the same URL returns 200 from a runner. Making the smoke job pass from CI needs the same Cloudflare change that blocked the standard availability test (T-519 itself closed 2026-09-01 by routing around it with the ADR 0024 Worker; runner paths stay blocked); the alternatives are weakening the origin lock or asserting against a host that answers without touching the API. Tiers 2–3 need credentials and stay operator-run regardless. 5. Azure portal / CLI spot checks for the changed resources. 6. Application Insights: no new exception cluster in the 30 minutes after apply; both budgets' configuration intact after any change that touches them. 7. Update Required-Inputs status (SET → VERIFIED) for any input exercised for the first time.

5. Rollback

A failed apply is not a rollback. Terraform converges forward: on error it stops scheduling new nodes, lets the in-flight ones finish, and leaves state wherever it got to. Nothing is undone, and a destroy that has already run is gone. So the response to a red apply is §4 step 1 followed by re-running to convergence — investigate from a known state, not from a half-applied one. Most partial applies are harmless and self-heal on the next run; the one that does not announce itself is the Function App case in §4 step 1.

Deliberate rollback is roll-forward to the previous definition:

1. Revert the merge commit in Git (git revert), PR it, merge.
2. Plan and apply the revert through the same gates (§2–§3).
3. prevent_destroy resources cannot be rolled back by replacement. If a bad change landed inside one (e.g. an indexing policy), the revert updates it in place. If the resource itself must go, that is a human decision recorded in TODO.md — remove the guard in a dedicated PR that says so in its title.
4. State surgery (terraform state mv/rm, imports) is a last resort: snapshot the state first (TFC keeps versions), record the commands run in the PR that motivated them.

6. Day-2 operations

Concern	Mechanism	Where
Cost	Two subscription-scoped budgets — USD 150 on the application subscription, USD 25 on Platform Management for Log Analytics — each at 50/75/90/100% actual plus a forecast alert	azurerm_consumption_budget_subscription (two) in main.tf, Cost analysis
Alerting	Five metric and log rules routed through ag-plat-prod-cus-01 — declared, not yet applied; the live estate has none. What each one means, what to check first, and what nothing watches	Alerting and support, infra/observability.tf
Drift	Periodic TFC plan (enable a scheduled speculative plan); investigate non-empty plans — portal edits are defects	TFC workspace settings
Computed properties	heal-computed-properties.yml re-applies cp_sortDate on relevant pushes and every 6 h	.github/workflows/
Secrets	Values live only in Key Vault. Seeded from Admin → Platform → API Keys, which writes from inside the integration subnet through a set-only role — no firewall window. scripts/cutover/06-seed-secret.ps1 and an admin_ip_rules window remain the break-glass path for when the app itself is down. References in Required-Inputs §4.6	infra/variables.tf (admin_ip_rules), Key Vault
Storage access	Account keys are disabled on both production accounts by fix/go-live-remediation — declared, not yet applied. From that apply on, az storage data-plane commands and the portal Storage Browser default to key auth and answer 403: use --auth-mode login plus a data-plane role, and note the content account has no operator network path at all	Alerting and support
Purge protection	purge_protection_enabled stays false by an owner decision of 2026-08-24 — it is a one-way switch that removes the teardown-and-recreate path a single-environment estate depends on. Soft delete at 90 days is the compensating control	TODO.md Accepted risks, ADR 0021
Dependency and action updates	Dependabot (npm + github-actions) with CI as the gate	.github/dependabot.yml

7. ALZ absorption

The subscription is a standalone platform subscription today and is expected to move under an Azure Landing Zone management-group hierarchy. Nothing in infra/ assumes tenant-root placement, so the move is administrative — but policy inheritance is not, and it is where the friction will be.

Sequence, when the ALZ exists:

1. Inventory in audit mode. Ask the ALZ operators for the policy set of the target management group; run it in audit against this subscription. Expected friction points: public-network-access defaults on Cosmos and Storage (the static-first architecture deliberately serves public media through the Function App identity), Key Vault firewall shape, allowed regions, mandated diagnostic-settings destinations.
2. Remediate or exempt, in-repo. Every remediation is a normal PR through this runbook; every exemption is recorded as an ADR.
3. Move the subscription into the management group (ALZ operators).
4. Verify survivors. Budget, RBAC role assignments, and the OIDC deployment identities are subscription-scoped and should survive; run §4 verification plus a full plan to confirm zero drift.
5. Re-point diagnostics to the central Log Analytics workspace if the ALZ mandates one — additive diagnostic settings, not replacement of the local workspace.

Tags are already the ALZ contract (workload, environment, owner, costCenter, managedBy, criticality, dataClassification); do not fork the schema per-resource.

---

Companion pages: IaC Repository Standard · Migration Runbook · Architecture · Well-Architected assessment