Skip to content

Cold rebuild runbook

A single ordered, top-to-bottom disaster recovery guide to rebuild the entire HybridCloudWorks platform from bare accounts.

Disaster recovery objectives (PLAT-6 / ADR 0011)

  • RTO (Recovery Time Objective): 8 hours
  • RPO (Recovery Point Objective): 24 hours (nightly Cosmos DB export)
  • Prerequisites in escrow: Owner Bitwarden vault holding cloud provider root credentials, domain registrar ownership, and generated shared secrets.

Overview and execution phases

flowchart TD
    P0["Phase 0: Account Auth & Scopes"] --> P1["Phase 1: Bootstrap Custom Roles"]
    P1 --> P2["Phase 2: Entra ID App Registrations & OIDC"]
    P2 --> P3["Phase 3: HCP Terraform Workspace Setup"]
    P3 --> P4["Phase 4: Terraform Infrastructure Apply"]
    P4 --> P5["Phase 5: GitHub Repo Vars & Secrets Seeding"]
    P5 --> P6["Phase 6: Key Vault Secret Escrow Seeding"]
    P6 --> P7["Phase 7: Cloudflare Edge & DNS Configuration"]
    P7 --> P8["Phase 8: Code Deployments (Functions & Frontend)"]
    P8 --> P9["Phase 9: Cosmos DB State Restoration"]
    P9 --> P10["Phase 10: Hybrid Lab VPS & Arc Onboarding"]
    P10 --> P11["Phase 11: End-to-End Smoke Verification"]

Phase 0: Identity, accounts and CLI sign-in

0.1 Prerequisites checklist

Ensure administrative access to the following upstream platforms: - [x] Azure Tenant (00000000-0000-0000-0000-000000000001) with Global Administrator or Owner privileges across the four subscriptions: - Application Landing Zone: sub-app-site-prod-cus (00000000-0000-0000-0000-000000000002) - Platform Management: sub-plat-mgmt-prod-cus (00000000-0000-0000-0000-000000000003) - Platform Connectivity: sub-plat-conn-prod-cus (00000000-0000-0000-0000-000000000004) - Platform Identity: sub-plat-ident-prod-cus (00000000-0000-0000-0000-000000000005) - [x] Cloudflare: Account managing zone hybridcloudworks.com. - [x] GitHub: Organization owner of saulpatinojr/HCW-HybridCloudWorks. - [x] HCP Terraform: Organization hcw access with admin rights. - [x] Hostinger: VPS access for the Hybrid Lab host. - [x] Bitwarden: Owner password manager holding escrowed secrets.

0.2 Azure CLI authentication

Sign in scoped to the tenant:

az login --use-device-code --tenant "<ENTRA_TENANT_ID>"

Set subscription context to the Application Landing Zone:

az account set --subscription "<SUBSCRIPTION_APP_ID>"

Verify the session:

az account show -o json | ConvertFrom-Json | Select-Object name, id, tenantId

Phase 1: Bootstrap Azure custom roles

HCP Terraform's run identity holds Contributor and Role Based Access Control Administrator, which intentionally lack permission to create role definitions (Microsoft.Authorization/roleDefinitions/write).

Run the bootstrap script to create every custom role. It takes no ids: with the Application subscription set as the default in step 0.2, it reads that subscription from az account show and finds the Management subscription in az account list by its name, sub-plat-mgmt-prod-cus. The JSON files under infra/roles/ carry the placeholder 00000000-0000-0000-0000-00000000000N in their scopes on purpose (this repository is public, #1540); the script fills in the real subscription ids at create time, so never create a role straight from one of those files.

pwsh scripts/bootstrap-roles.ps1

POSIX alternative: 

scripts/bootstrap-roles.sh

Success looks like one "Created successfully" line per role and then a read-back table listing every HCW ... role with real subscription ids in its scopes. If the script stops saying a subscription id "is the repository placeholder" or that it cannot see exactly one sub-plat-mgmt-prod-cus, sign in again with an account that holds both subscriptions, or pass the ids yourself: in PowerShell with -SubscriptionId and -ManagementSubscriptionId; in bash as two positional arguments to scripts/bootstrap-roles.sh, the workload (Application) subscription id first and the Management subscription id second. Read the workload id from step 0.2's az account show output, and the Management id from the row named sub-plat-mgmt-prod-cus in az account list -o table, rather than typing either id from memory.

Created roles: 1. HCW Key Vault Secret Writer (infra/roles/keyvault-secret-writer.json) 2. HCW Function Config Refresh (infra/roles/function-config-refresh.json) 3. HCW Function Settings Reader (infra/roles/function-settings-reader.json) 4. HCW Static Web App Deployer (infra/roles/static-web-app-deployer.json) 5. HCW Alert Rule Reader (infra/roles/alert-rule-reader.json) 6. HCW Storage Firewall Writer (infra/roles/storage-firewall-writer.json)

Phase 2: Entra ID app registrations and federated credentials

Run the Entra ID setup script or manually create the required service principals and federated identity credentials:

2.1 HCP Terraform run identity

  • App Name: id-plat-terraform-prod-cus-01
  • Federated Credential:
  • Provider: app.terraform.io
  • Subject: organization:hcw:project:Site:workspace:hcw-azure:run_phase:*
  • Audience: api://AzureADTokenExchange
  • RBAC Grants:
  • Contributor on sub-app-site-prod-cus and sub-plat-mgmt-prod-cus
  • Role Based Access Control Administrator on sub-app-site-prod-cus and sub-plat-mgmt-prod-cus

2.2 GitHub Actions identities (OIDC)

Create application registrations and federated credentials matching infra/oidc.tf: - id-site-github-deploy-prod-cus-01 (repo:saulpatinojr/HCW-HybridCloudWorks:environment:production) - id-site-github-reader-prod-cus-01 (repo:saulpatinojr/HCW-HybridCloudWorks:environment:production) - id-site-github-alert-reader-prod-cus-01 (repo:saulpatinojr/HCW-HybridCloudWorks:environment:production) - id-site-github-manifest-prod-cus-01 (repo:saulpatinojr/HCW-HybridCloudWorks:environment:production) - id-site-github-copilot-prod-cus-01 (repo:saulpatinojr/HCW-HybridCloudWorks:environment:copilot)

2.3 HCWSite API and Lab Agent registration

  • App Name: HCWSite API (api://<HCWSite-Client-ID>)
  • Define App Role LabAgent with value LabAgent.
  • Register Service Principal sp-labs-agent-lab-hybrid-prod-cus-01 (managed in Phase 10).

Phase 3: HCP Terraform workspace configuration

In HCP Terraform organization hcw, configure workspace hcw-azure:

3.1 Environment variables

Set category Environment (sensitive where indicated): - TFC_AZURE_PROVIDER_AUTH = true - TFC_AZURE_RUN_CLIENT_ID = <Client ID of id-plat-terraform-prod-cus-01> - ARM_TENANT_ID = <ENTRA_TENANT_ID> (sensitive) - ARM_SUBSCRIPTION_ID = <SUBSCRIPTION_APP_ID> (sensitive)

3.2 Required Terraform variables

Set category Terraform: - subscription_app = <SUBSCRIPTION_APP_ID> (sensitive) - subscription_mgmt = <SUBSCRIPTION_MGMT_ID> (sensitive) - subscription_conn = <SUBSCRIPTION_CONN_ID> (sensitive) - entra_tenant_id = <ENTRA_TENANT_ID> (sensitive) - entra_api_audience = api://<HCWSite-Client-ID> - cloudflare_api_token = <CLOUDFLARE_API_TOKEN> (sensitive; Zone:Read, DNS:Edit, Transform Rules:Edit, Account Rulesets:Read and, for the Web Analytics site, Account → Account Settings → Write) - cloudflare_zone_id = <CLOUDFLARE_ZONE_ID> - cloudflare_account_id = <CLOUDFLARE_ACCOUNT_ID> (optional, #1302, the same value as the repository variable of that name; while unset the Web Analytics site is not created and its outputs read null) - cloudflare_origin_secret = <CF_ORIGIN_SECRET> (sensitive; from Bitwarden) - budget_alert_email = admin@hybridcloudworks.com

Phase 4: Terraform infrastructure apply

  1. Trigger a plan in workspace hcw-azure.
  2. Inspect the plan output:
  3. Resource Groups: rg-web-site-prod-cus, rg-data-site-prod-cus, rg-ai-site-prod-cus, rg-mgmt-plat-prod-cus, rg-lab-hybrid-prod-cus.
  4. Key Vault: kv-site-prod-cus-01 (purge protection enabled).
  5. Storage accounts: stsiteprodcus01, stfnsprodcus01.
  6. Cosmos DB: cosmos-site-prod-cus (Serverless, periodic backup, datacenter sentinel).
  7. Function App: func-site-prod-cus-01 with @Microsoft.KeyVault(...) app settings.
  8. Static Web App: swa-site-prod-cus-01.
  9. Observability: log-plat-prod-cus-01, Application Insights appi-site-prod-cus-01, alerting fabric.
  10. Cloudflare Ruleset: cloudflare_ruleset.origin_secret stamping x-hcw-origin-secret.
  11. Cloudflare Web Analytics site: cloudflare_web_analytics_site.hcw[0] for hybridcloudworks.com with the lite option, only when cloudflare_account_id is set (#1302).
  12. Confirm and apply the run.
  13. Record all Terraform outputs.

Phase 5: GitHub repository variables and secrets seeding

From a workstation with gh authenticated to the repository, seed variables from Terraform outputs:

pwsh scripts/set-github-variables.ps1

The script does not seed CF_WEB_ANALYTICS_TOKEN (#1302). Set it by hand from the web_analytics_site_token output recorded in Phase 4; the value is public (it ships in every page), so it is read at the prompt in the clear:

$t = Read-Host 'web_analytics_site_token from the hcw-azure outputs'; gh variable set CF_WEB_ANALYTICS_TOKEN --body $t

Nothing clears or refreshes this variable later: if the site is destroyed (cloudflare_account_id unset again) or replaced (the account id changed, which forces a new site with a new token), every page keeps shipping a beacon with a dead or wrong token until the value is re-copied from the new output, or cleared, and the frontend deployed again; after a clear the next deploy ships no tag:

gh variable delete CF_WEB_ANALYTICS_TOKEN -R saulpatinojr/HCW-HybridCloudWorks

Set the required repository secrets: - TFC_TOKEN: HCP Terraform user or team API token. - MANIFEST_APP_PRIVATE_KEY: Private PEM key of the repository GitHub App.

Ensure GitHub Environment production is configured: - Required Reviewers: saulpatinojr and hcw-architect (either can approve). - Deployment branch policy: Selected branches → main only.

Phase 6: Key Vault secret escrow seeding

Using the authoritative Where to Re-obtain paths documented in Required inputs §4.6, seed all 32 Key Vault secrets into kv-site-prod-cus-01:

# Generated high-entropy secrets (retrieve from Bitwarden escrow, or generate fresh with `openssl rand -hex 32` and immediately escrow in Bitwarden):
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "CF-ORIGIN-SECRET" --value "<FROM_BITWARDEN>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "CLIENT-IP-SALT" --value "<FROM_BITWARDEN>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "PREVIEW-SIGNING-SECRET" --value "<FROM_BITWARDEN>"

# Upstream API Keys & Tokens:
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "GEMINI-API-KEY" --value "<KEY>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "ANTHROPIC-API-KEY" --value "<KEY>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "OPENAI-API-KEY" --value "<KEY>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "NVIDIA-API-KEY" --value "<KEY>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "ELEVENLABS-API-KEY" --value "<KEY>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "AZURE-SPEECH-KEY" --value "<KEY>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "FIRECRAWL-API-KEY" --value "<KEY>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "REPLICATE-API-KEY" --value "<KEY>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "PUBLER-API-KEY" --value "<KEY>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "PUBLER-WORKSPACE-ID" --value "<ID>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "RESEND-API-KEY" --value "<KEY>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "LINKIE-API-KEY" --value "<KEY>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "TELEGRAM-BOT-TOKEN" --value "<KEY>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "TELEGRAM-CHAT-ID" --value "<ID>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "PODBEAN-CLIENT-ID" --value "<ID>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "PODBEAN-CLIENT-SECRET" --value "<SECRET>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "STREAMYARD-API-KEY" --value "<KEY>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "YOUTUBE-API-KEY" --value "<KEY>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "PLAUD-EMBEDDED-CLIENT-ID" --value "<ID>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "PLAUD-EMBEDDED-API-KEY" --value "<KEY>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "AWS-ACCESS-KEY-ID" --value "<ID>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "AWS-SECRET-ACCESS-KEY" --value "<KEY>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "GCP-BILLING-API-KEY" --value "<KEY>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "QLTY-API-TOKEN" --value "<TOKEN>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "TURNSTILE-SECRET-KEY" --value "<KEY>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "CLOUDFLARE-ANALYTICS-TOKEN" --value "<TOKEN>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "CODER-URL" --value "https://coder.lab.hybridcloudworks.com"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "CODER-STATUS-TOKEN" --value "<TOKEN>"
az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "CODER-BROKER-TOKEN" --value "<TOKEN>"

Phase 7: Cloudflare Edge and DNS configuration

  1. DNS Apex and WWW Bindings:
  2. In Cloudflare DNS, configure CNAME records pointing hybridcloudworks.com and www to the Static Web App default hostname (*.azurestaticapps.net).
  3. Validate custom domain binding in Azure Static Web App (swa-site-prod-cus-01).
  4. API Subdomain:
  5. Verify api-azure.hybridcloudworks.com CNAME points to func-site-prod-cus-01.azurewebsites.net (Proxied = true).
  6. Turnstile Widgets:
  7. Create Turnstile widget for hybridcloudworks.com in Cloudflare dashboard (Managed mode).
  8. Set site key in GitHub variable VITE_TURNSTILE_SITE_KEY.
  9. Seed secret key in Key Vault TURNSTILE-SECRET-KEY.
  10. Availability Probe & Security:
  11. Enable Bot Fight Mode.
  12. Deploy Edge Availability Probe Worker per Availability probe runbook.
  13. Web Analytics (#1302):
  14. No dashboard step to create the site: the Phase 4 apply creates cloudflare_web_analytics_site.hcw[0] once cloudflare_account_id is set (Phase 3.2), and the beacon is built into the pages from CF_WEB_ANALYTICS_TOKEN (Phase 5).
  15. One dashboard step to confirm it: the site is created with Cloudflare's lite option, which asks Cloudflare not to measure visitors in the EU. That is documented for the snippet Cloudflare's proxy injects and unverified for the hand-placed one this site carries, so open the site's Manage page in the Web Analytics dashboard at https://dash.cloudflare.com/?to=/:account/web-analytics, check that the EU option shows as applied, and record the date in the cloudflare_account_id row of Required inputs §4.1. Until that date is recorded, the privacy policy says only that we have asked Cloudflare to exclude EU visitors.
  16. Mint the read-only token for CLOUDFLARE-ANALYTICS-TOKEN (Phase 6) at https://dash.cloudflare.com/profile/api-tokens: Custom token, Account → Account Analytics → Read only, the HybridCloudWorks account only, an end date at most a year out, named azure-functions-cloudflare-analytics-account.
  17. Arm the read by adding AMBASSADOR_AUDIENCE_MONTHLY to enabled_timers (Required inputs §4.1).

Phase 8: Application code deployments

  1. Deploy Azure Functions: Trigger GitHub Actions workflow deploy-functions.yml on branch main. 
    gh workflow run deploy-functions.yml --ref main
  2. Deploy Azure Frontend: Trigger GitHub Actions workflow deploy-azure-frontend.yml on branch main. 
    gh workflow run deploy-azure-frontend.yml --ref main
  3. Restart Function App: Trigger reference refresh to bind all Key Vault secrets: 
    az functionapp restart --name "func-site-prod-cus-01" --resource-group "rg-web-site-prod-cus"

Phase 9: Cosmos DB state restoration

Restore production database collections from the nightly out-of-account export storage account:

  1. Identify the latest complete backup export in stsiteprodcus01 (container cosmos-export): 
    az storage blob list `
  --account-name "stsiteprodcus01" `
  --container-name "cosmos-export" `
  --auth-mode login `
  --query "[?ends_with(name, '.json.gz')].name" -o tsv
  2. Run dry-run restore validation: 
    node scripts/restore-cosmos-export.mjs --dry-run
  3. Execute restore to production Cosmos DB (cosmos-site-prod-cus): 
    node scripts/restore-cosmos-export.mjs
  4. Verify document counts against Cosmos restore runbook.

Phase 10: Hybrid Lab host (VPS) rebuild

Follow Labs host runbook to reconstruct the Hostinger VPS:

  1. Provision VPS: Hostinger hPanel → Ubuntu 26.04 LTS.
  2. Arc Onboarding: 
    pwsh scripts/lab/Register-LabArc.ps1 -Connect
  3. Lab Agent Registration: 
    pwsh scripts/lab/Register-LabAgent.ps1
  4. Run Ansible Configuration: 
    cd lab-host/ansible && ansible-playbook -i inventory.ini site.yml
  5. HashiCorp Vault Initialisation: Initialize Vault, store recovery shares in Bitwarden, and configure Azure Key Vault auto-unseal.
  6. Caddy & TLS DNS-01: Seed Caddy's Cloudflare API token into /etc/caddy/env and restart caddy to issue wildcard *.lab.hybridcloudworks.com.

Phase 11: End-to-end verification and smoke testing

Run automated verification checks:

  1. Alerting Fabric State Verification: 

    node scripts/verify-alert-state.mjs
    Must exit 0 with all alert rules confirmed enabled.

  2. Secret Resolution Check: 

    node scripts/check-unresolved-secrets.mjs
    Must report 0 unresolved secrets.

  3. Public API Health Check: 

    curl -fsS -i "https://api-azure.hybridcloudworks.com/api/public/addons/migration/status"
    Must return 200 OK.

  4. Frontend Reachability: 

    curl -fsS -I "https://hybridcloudworks.com"
    Must return 200 OK with valid security headers (Content-Security-Policy, Strict-Transport-Security).