Origin secret rotation — `CF-ORIGIN-SECRET`¶

Zero-downtime rotation procedure for the shared origin authentication secret.

Coordinated 3-way lock

Rotating CF-ORIGIN-SECRET requires synchronized changes across three distinct planes:

Azure Key Vault: Secret CF-ORIGIN-SECRET in kv-site-prod-cus-01.
HCP Terraform: Sensitive workspace variable cloudflare_origin_secret in workspace hcw-azure.
Cloudflare Transform Rules: Zone ruleset cloudflare_ruleset.origin_secret stamping HTTP header x-hcw-origin-secret.

A mismatch between the header sent by Cloudflare and the secret resolved by Azure Functions causes every anonymous request and public submission to fail with HTTP 403 Forbidden (unverified origin). Follow this runbook in exact sequence.

1. Architectural context¶

To protect the Azure Functions backend (func-site-prod-cus-01) from direct internet scraping and bypass attacks, the backend enforces an origin lock:

Network-level restriction: Azure App Service access restrictions limit traffic to Cloudflare IP ranges (functions_origin_lock_enabled = true).
Edge transform rule: Cloudflare Edge intercepts requests to api-azure.hybridcloudworks.com and rewrites request headers using an HTTP request late transform rule (infra/frontend.tf), stamping x-hcw-origin-secret with the value of var.cloudflare_origin_secret.
Application-level verification: The Function App resolves process.env.CF_ORIGIN_SECRET from Key Vault via @Microsoft.KeyVault(SecretUri=...secrets/CF-ORIGIN-SECRET). Every incoming request evaluated by functions/src/lib/auth/client-identity.js checks:

const viaCloudflare = (request) => {
  if (!originSecret) return false;
  return request?.headers?.get?.('x-hcw-origin-secret') === originSecret;
};

Because Azure App Service caches resolved Key Vault secret references for up to 24 hours, updating Key Vault alone does not update runtime memory immediately without an explicit restart or ARM configuration touch.

2. Pre-flight verification¶

Before beginning rotation, ensure the environment is stable:

Verify current edge health:

curl -fsS -i "https://api-azure.hybridcloudworks.com/api/public/addons/migration/status"

Ensure it returns HTTP 200 with JSON payload { configured: ... }.

Verify no in-flight Terraform runs: Ensure workspace hcw-azure on HCP Terraform is in state idle (no planning or applying runs).
Verify no in-flight GitHub Actions deploys: Check Actions to ensure neither deploy-functions.yml nor deploy-azure-frontend.yml is executing.

3. Step-by-step rotation procedure¶

Step 1: Generate new high-entropy secret¶

Generate a cryptographically secure 256-bit hexadecimal secret:

openssl rand -hex 32

Save this value to the owner's password manager (Bitwarden) as the staged new value for CF-ORIGIN-SECRET. Never commit, log, or transmit this value over insecure channels.

Step 2: Update Azure Key Vault¶

Write the new secret value into Key Vault kv-site-prod-cus-01:

Option A: Azure CLI¶

az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "CF-ORIGIN-SECRET" --value "<NEW_SECRET_VALUE>"

Option B: Admin UI¶

Navigate to https://hybridcloudworks.com/admin/integrations?tab=keys, locate Cloudflare origin secret, paste the new secret, and save.

Step 3: Refresh Function App runtime reference¶

App Service must evict the cached secret and resolve the latest version from Key Vault. Restart the Function App to make the new secret active in runtime memory immediately:

az functionapp restart --name "func-site-prod-cus-01" --resource-group "rg-web-site-prod-cus"

Planned service interruption window

Because the Azure Functions runtime immediately resolves the new secret upon restart while Cloudflare continues stamping the previous secret until the Terraform apply completes, there is an unavoidable brief service interruption window (typically 30–90 seconds) during which anonymous public requests will receive HTTP 403 Forbidden (unverified origin). Execute Steps 4 and 5 immediately to minimize the window.

Step 4: Update HCP Terraform workspace variable¶

Log in to HCP Terraform and navigate to organization hcw, workspace hcw-azure.
Go to Variables → Workspace variables.
Locate cloudflare_origin_secret (Category: Terraform, Sensitive: true).
Click Edit, enter <NEW_SECRET_VALUE>, and click Save variable.

Step 5: Plan and apply Terraform configuration¶

In workspace hcw-azure, click New run → Start run (Reason: Rotate cloudflare_origin_secret).
Review the speculative plan:
Expected change: Exactly 1 change on cloudflare_ruleset.origin_secret.
Verify no unintended resource replacements.
Click Confirm & apply.
Cloudflare's global edge network will propagate the updated header rewrite rule within seconds.

4. Post-rotation verification¶

Perform end-to-end verification across edge and origin:

Verify public endpoint via Cloudflare:

curl -fsS -i "https://api-azure.hybridcloudworks.com/api/public/addons/migration/status"

Must return 200 OK.

Verify direct origin rejection (bypassing Cloudflare): Verify that traffic attempting to reach the Function App directly without passing through Cloudflare is rejected by App Service access restrictions:
```
curl -i "https://func-site-prod-cus-01.azurewebsites.net/api/public/addons/migration/status"
```
Must return 403 Forbidden (Ip Forbidden). Note that Cloudflare's transform rule overwrites x-hcw-origin-secret on all proxied traffic through api-azure.hybridcloudworks.com.
Check Application Insights live metrics: Query recent traces in Log Analytics log-plat-prod-cus-01:
```
AppTraces
| where TimeGenerated > ago(15m)
| where Message has "unverified origin"
```
The result must be 0 records post-rotation.
Update secret escrow record: Confirm the old secret value is marked as rotated and archived in the owner's Bitwarden vault.

5. Rollback procedure¶

If Cloudflare rule application fails or the Function App cannot resolve the new secret:

Retrieve the <PREVIOUS_SECRET_VALUE> from Bitwarden history.

Re-set Key Vault:

az keyvault secret set --vault-name "kv-site-prod-cus-01" --name "CF-ORIGIN-SECRET" --value "<PREVIOUS_SECRET_VALUE>"

Restart Function App:

az functionapp restart --name "func-site-prod-cus-01" --resource-group "rg-web-site-prod-cus"

Revert cloudflare_origin_secret in HCP Terraform workspace hcw-azure to <PREVIOUS_SECRET_VALUE> and apply.
Re-run verification curl in §4.

Origin secret rotation — CF-ORIGIN-SECRET¶