Cosmos restore — the drill that proves RPO 24 h / RTO 8 h¶
Status: first drill passed 2026-10-13, RTO 1 h 39 min against 8 h and RPO 22 h against 24 h (the Drills table at the end). Next due by 2027-01-21. The exporter (
cosmosExportSchedulerand thecosmos-export-containerjob,functions/src/lib/backup/) runs whileFEATURE_FLAG_COSMOS_EXPORTistrue, which Terraform writes from thecosmos_export_enabledworkspace variable; its first run completed 2026-09-09 03:00 UTC and the daily missing-run alert is live. The first drill was declined on 2026-09-09, approved on 2026-10-06 (estate review, finding PLAT-1) and run on 2026-10-13. It found three bugs in the restore script, fixed in #1102: every restore with a delta in it stopped on a full-only container, the count check misread empty containers and reruns, and nothing kept the script off the live account. It also found the step 5 gaps this page now covers. This page records what was restored and how long it took, not what was configured.
The design is ADR 0028:
every Sunday the Function App exports every included container in full, every
other day it exports each container's change feed since the previous run, and
the copies land as gzip NDJSON in the cosmos-export blob container on
stsiteprodcus01 — a different Azure resource, in a different resource group,
geo-replicated to the paired region. A restore is "the latest complete full
plus every complete delta after it", upserted into a fresh account.
The objectives (owner decision 2026-08-30, #231): RPO 24 hours — the newest complete run is at most a day old — and RTO 8 hours — from the decision to restore to the site's public reads answering from the restored account. The drill below measures both. Run it once a quarter.
What comes back, and what does not¶
Say this before the drill, so nothing in it is a surprise:
- 64 of the 78 containers: the 47 authored and 10 configuration containers
from every run, the 7 operational-record containers (
admin_audit_logs,audits,ai_usage,telegram_bot_activity,workflow_alerts,workflow_digests,content_stats_markers) as of the last full only. The list is code:functions/src/lib/backup/export-classification.js. - Not restored, by design: the creator Second Brain's search index
brain_search(rebuilt from its notes by reindexing),_snapshots,homepage_feeds,rss_cache,tool_service_cacheandtool_price_history(regenerated by a publish or a scheduled job; the price history restarts from empty, which the owner accepted over exporting it),azure_landing_contentandtool_service_catalog(re-seeded by a script), and the six TTL-bounded job and quota containers. A restored site republishes its snapshots and refills its caches on the first scheduled runs. - A retired container in an older set is skipped, not restored. The
script restores only into containers that
infra/cosmos-containers.jsonprovisions today. A full export written before a container was retired still lists it (the Code Conductor brain'scode_brain_records, removed on #1479); the script prints its name under "Skipped containers" and goes on, and refuses outright if--containersasked for it by name. - Up to a week of resurrected deletes. The change feed does not carry deletes; only the weekly full reconciles them. A document deleted on a Tuesday and restored from Sunday's full plus Monday-to-Saturday's deltas is back. Accepted in ADR 0028; the all-versions-and-deletes change feed is the revisit path if it ever matters.
- Document bodies are exact; system fields are not.
_rid,_self,_etag,_attachmentsand_tsare stripped on restore and re-minted by the target account. Nothing in the application reads them across restarts. - Continuation state (
cosmos-export-statecontainer) is not restored. It lives in a dedicated containercosmos-export-state(isolated from immutablecosmos-exportblobs under ADR 0028 / PLAT-8) and belongs to the source account's change feed; it means nothing to another account. - Export container immutability (WORM). The backup container
cosmos-exportenforces a 35-day time-based immutability policy (azurerm_storage_container_immutability_policy.cosmos_export). Run blobs cannot be overwritten or deleted until the retention period lapses. Verify the policy:
Is the export running? Read this first¶
The quick answer: the Nightly export tile on the Health page¶
Since #1409
the site reads its own export. The cosmos-export blob container refuses every
workstation by design, and the log workspace has dropped the 03:00 UTC window
before (#1530),
but the Function App has both the identity and the network path. So the ops
snapshot (the block of facts behind the admin Health page) carries a
backups block, read by
functions/src/lib/backup/backup-status.js
from the run records and manifests of the last seven UTC dates. It holds names
and counts only, never a document.
Two places show it, with one verdict between them:
- Overview → Operational Signals → Nightly export
(
https://hybridcloudworks.com/admin/health?tab=overview): the newest completed run (delta/2026-11-03style), when it finished, how many containers it holds of the number planned, how many of the authored containers (class A inexport-classification.js) are in it with the names of any that are not, andbrain_recordson its own with its partition key/ownerIdand record count. - Checks → Platform → Nightly export
(
https://hybridcloudworks.com/admin/health?tab=checks): the same sentence as a probe, recorded every five minutes by the health pulse, so the Report tab and the stored history carry it too.
How to read the verdict:
| Verdict | Meaning | What to do |
|---|---|---|
| Healthy | A run completed within the last 26 hours and every authored container is in it. The sentence ends with brain_records: N records, partition key /ownerId. |
Nothing. |
| Critical, "finished N h ago; a run is due every 24 h" | The newest manifest is older than 26 hours: last night's run did not finish. | Look at Failed Platform Jobs on the same card, then the traces query below for the container that failed. |
| Critical, "N authored container(s) missing: …" | The run completed, but a container the classification calls authored is not in it. A container added to the classification after that run's record was claimed shows up here until the next night. | If it is still missing the next morning, the scheduler's plan and the classification disagree: file it. |
| Critical, "No completed export manifest in the last 7 days" | Nothing completed in a week. The sentence names how far the newest run got (markers written of the number planned) when there is one. | Same as the alert: check the switch with (PowerShell) az functionapp config appsettings list -g rg-web-site-prod-cus -n func-site-prod-cus-01 -o json | ConvertFrom-Json | Where-Object name -eq FEATURE_FLAG_COSMOS_EXPORT | Select-Object name, value (it should print true), then the queries below. |
| Unknown, "could not be read" | The worker could not read the blob container (role, firewall, outage). This is not "no export". | Check the Function App identity's Storage Blob Data role and the storage firewall's resource rule. |
The block is reread at most every ten minutes, so a run that finishes while
you watch shows within ten minutes of the next pulse. The partition key shown
is the one the code holds for the container
(cosmos-client.js PARTITION_KEY_PATHS);
the manifest itself does not carry it.
The long answer: the logs¶
Containers added after the most recent weekly full are included from the selected daily manifests. Their first delta reads the change feed from its beginning, so it provides the initial authored records. The restore selects the union of those manifests and applies only the layers that carry each container. A missing blob listed by a manifest still stops the restore.
Success is silent and only absence pages (ADR 0028 §5). To see the runs
directly, query Application Insights from PowerShell (the application-insights
CLI extension installs itself on first use; the resource is
appi-site-prod-cus-01, instance suffix included):
az monitor app-insights query --app appi-site-prod-cus-01 -g rg-web-site-prod-cus --offset 14d --analytics-query 'customEvents | where name == "cosmosExportCompleted" | project timestamp, mode = tostring(customDimensions.mode), runId = tostring(customDimensions.runId), containers = toint(customDimensions.containers), docs = tolong(customDimensions.docs), bytes = tolong(customDimensions.bytes), durationMs = tolong(customDimensions.durationMs) | order by timestamp desc' -o table
A healthy fortnight shows two full rows and twelve delta rows, one per
UTC date. Counts follow the classification at the time of each run (currently
64 full and 57 delta containers); earlier runs can have fewer. A gap in runId is a run
that never wrote its manifest — a container job failed or timed out — and the
missing-run alert will have fired on it. An empty table when the exporter has
been armed for more than two days is the same finding as the alert. The
per-container detail is in traces:
az monitor app-insights query --app appi-site-prod-cus-01 -g rg-web-site-prod-cus --offset 2d --analytics-query 'traces | where message startswith "[cosmosExport]" | project timestamp, message | order by timestamp desc | take 100' -o table
Every line names a container, a count and a duration. None names a document.
The drill¶
Everything below is PowerShell, one line each, from the repository root on
main with the exporter PR for #231 merged. Confirm the script is on the
checkout before believing any error that names it:
True is the only acceptable answer. Step 5 runs the Function App locally,
which needs Azure Functions Core Tools; install it once, then open a new
PowerShell window so func is on the path:
Then sign in, select the App landing zone subscription, install the script's dependencies, and start the clock — the RTO is measured from this line:
1. Open the export to the workstation¶
stsiteprodcus01 is Deny by default at the network and reads are
role-gated at the data plane, so a workstation needs both a firewall rule and
a data-plane role, scoped to the one container. Both are removed in step 6,
and Terraform's next apply on hcw-azure would remove the rule anyway (it is
not declared).
az storage account network-rule add -g rg-stor-site-prod-cus --account-name stsiteprodcus01 --ip-address $ip -o none
$storageId = (az storage account show -n stsiteprodcus01 -g rg-stor-site-prod-cus -o json | ConvertFrom-Json).id
az role assignment create --assignee-object-id $me --assignee-principal-type User --role "Storage Blob Data Reader" --scope "$storageId/blobServices/default/containers/cosmos-export" -o none
Success is four silent exits. The role takes up to five minutes to propagate;
if step 2 answers AuthorizationPermissionMismatch, wait and rerun step 2 —
that error is propagation, not a wrong command.
2. Dry run: what would be restored, and how old is it¶
Success looks like:
DRY RUN — counting full/2026-09-13 + 3 delta(s) (2026-09-14 … 2026-09-16) — 60 container(s)
container full deltas created replaced ms
admin_audit_logs 412 0 310
…
Would restore 70112 document(s) across 60 container(s) in 41207 ms (0.7 min) — distinct and overwritten counts come from the upsert responses and are computed only on a real run
Read two things off it. RPO: the newest run id on the first line is the
recovery point; it must be yesterday's date or today's. Completeness: a
Skipped incomplete runs line names runs that never wrote a manifest; they are
not applied, and the newest applied run is the real recovery point. Writes
nothing, needs no Cosmos account, and is the step to run on its own any time
the question is "what would a restore give me right now".
no complete full export means no Sunday run has completed since the exporter
was armed: there is nothing to restore yet, and the drill stops here.
3. Create the target account¶
A fresh serverless account in its own resource group, tagged with the day it
must be gone by — the 2026-08-24 sandbox stayed for weeks because its expiry
lived somewhere other than the resource (infra/scratch.tf keeps that record).
Account keys are never read; the drill authenticates the same way production
does, so a restore that passes proves the RBAC path too.
az group create -n rg-db-site-sbx-cus -l centralus --tags purpose=cosmos-restore-drill expires=$expires -o none
az cosmosdb create -n cosmos-site-sbx-cus -g rg-db-site-sbx-cus --locations regionName=centralus --capabilities EnableServerless --default-consistency-level Session -o none
The account takes five to ten minutes. Then the database and every container
from the provisioned spec, with the partition key each one has in production
— the five containers not keyed on /id restore wrongly into /id
containers, silently:
$spec = Get-Content infra/cosmos-containers.json -Raw | ConvertFrom-Json; foreach ($c in $spec.containers) { az cosmosdb sql container create -a cosmos-site-sbx-cus -g rg-db-site-sbx-cus -d hcw -n $c.name --partition-key-path $c.partition_key_path -o none }
az cosmosdb sql container list -a cosmos-site-sbx-cus -g rg-db-site-sbx-cus -d hcw -o json | ConvertFrom-Json | Measure-Object | Select-Object Count
Success is a Count equal to the number of containers in the spec, which
$spec.containers.Count prints (75 on 2026-10-12). Creating them one at a time
takes about a minute each, and Azure prints each one's state (Enqueued,
Dequeued, Succeeded) as it goes: the repeating pattern is the loop moving
through the list, not a loop stuck on one container. Grant the signed-in user the built-in
data-plane contributor role on the new account (the role id is the same
constant in every tenant):
az cosmosdb sql role assignment create -a cosmos-site-sbx-cus -g rg-db-site-sbx-cus --role-definition-id 00000000-0000-0000-0000-000000000002 --principal-id $me --scope "/" -o none
4. Restore and verify¶
node scripts/restore-cosmos-export.mjs --storage-account stsiteprodcus01 --target-endpoint https://cosmos-site-sbx-cus.documents.azure.com:443/ --verify
Success looks like the dry run with created and replaced filled in. The
upsert answers 201 for a document the target had not seen and 200 for one an
earlier layer, or an earlier attempt, already wrote. The target column on
every row equals what the container held before the run plus what the run
created: into a fresh account that is created, and an empty container
reads 0. The check counts each container before and after writing it, so
nothing else may write to the target until the restore and its check finish:
another writer in between shows as a MISMATCH the data does not have. The
drill's scratch account has no other writer. An operational container such as admin_audit_logs shows 0
under deltas, because it is in the weekly full and in no delta, by design.
The last line reads
Restored 11971 document(s): 11937 created, 34 replaced (already in the target, or superseded by a later layer), across 62 container(s) in 104979 ms (1.7 min),
with exit code 0. A row ending MISMATCH is a container whose count in the
target is not that; it names the count it expected, the script exits 1, and
the drill has not passed until the cause is known. The cost of this step is about
$3 in write request units (cost analysis, #385).
5. Spot-check the site's public reads against the restored account¶
Counts prove the copy landed; a read proves the site can serve from it. Run
the Function App locally against the target account and ask it the two
anonymous questions a visitor asks first. The local host authenticates with
the same az login identity, and needs Azure Functions Core Tools (func,
installed above) and a storage emulator. Without one it starts, fails every
timer listener and exits on its own, which the first drill found. Azurite is
Microsoft's emulator; start it in its own PowerShell window and leave it
running:
Back in the drill's window, write the settings file. It is gitignored, and it holds the four values the public reads need and nothing else: no key and no connection string. Then confirm it exists before starting the host:
@{ IsEncrypted = $false; Values = @{ FUNCTIONS_WORKER_RUNTIME = 'node'; AzureWebJobsStorage = 'UseDevelopmentStorage=true'; COSMOS_ENDPOINT = 'https://cosmos-site-sbx-cus.documents.azure.com:443/'; COSMOS_DATABASE = 'hcw' } } | ConvertTo-Json | Set-Content functions/local.settings.json
It takes about a minute. Red lines are expected: the Cosmos change-feed
functions fail indexing because COSMOS_CONNECTION is deliberately absent,
so nothing processes the restored data. Wait for the list of HTTP routes on
http://localhost:7071/api/.... Then, in a third PowerShell window:
node -e "fetch('http://localhost:7071/api/public/content').then(r=>r.json()).then(j=>console.log('content items: ' + j.items.length))"
node -e "fetch('http://localhost:7071/api/public/podcasts').then(r=>r.json()).then(j=>console.log('podcast items: ' + j.items.length + '; titles: ' + j.items.map(i=>i.title).slice(0,3).join(' / ')))"
Success is a non-zero count on both lines, with podcast titles (2026-10-13: 24 content
items, 6 podcasts). Node, not Invoke-RestMethod: the content documents
carry keys that differ only in case (Title and title), which
PowerShell's JSON reader cannot hold. It hands back the raw text instead, so a
PowerShell count prints 0 against a correct answer. Stop the local host with
Ctrl+C, then Azurite in its window. This is the point the RTO is measured to:
Under 08:00:00 meets the objective. Record the elapsed time, the run ids and
the document count in the Drills table below before tearing down.
6. Tear down¶
The target holds a full copy of production data; it does not outlive the drill. The resource-group delete is asynchronous and irreversible.
az storage account network-rule remove -g rg-stor-site-prod-cus --account-name stsiteprodcus01 --ip-address $ip -o none
az role assignment delete --assignee $me --role "Storage Blob Data Reader" --scope "$storageId/blobServices/default/containers/cosmos-export" -o none
The local settings file points at the account just deleted; remove it:
Azurite's data from step 5 is local and disposable:
false (after a few minutes) is the end of the drill.
A real restore, not a drill¶
The same steps with two differences. The target is a permanent account whose
name, resource group and firewall the owner decides at the time (the scratch
names above are for something that gets deleted); and after step 4 the Function
App's COSMOS_ENDPOINT app setting (Terraform, infra/functionapp.tf) and the
identity's data-plane role assignment (infra/cosmos.tf) move to the new
account, and only once step 4's check has finished: the Function App is a
writer, and the check expects nothing else to write while it counts. Then
republish snapshots and let the ingest timers refill the caches.
The restore script refuses a target whose account name has -prod- in it,
because a restore writes backup copies over whatever the target holds; a real
restore into a new production-named account adds --allow-production-target.
The live account is never the target.
Deleted-in-the-last-week documents must be re-deleted by hand from the admin
portal; the audit rows say which.
A regional loss¶
Nothing on this page rehearses one, and the objectives above do not cover one: continuous backup restores into the same region, and the drill restores into a scratch account in the same region. This section says what exists and the order it would be used in, so the first regional incident starts from a list rather than a blank page. It is untested, and the honest estimate is that it does not fit inside the 8-hour RTO.
What survives a regional loss:
- The exports.
stsiteprodcus01is RA-GRS, so thecosmos-exportcontainer is readable from the paired region at once (the secondary endpoint) and writable after an account failover. - The code and the configuration: this repository, HCP Terraform state, and
every Key Vault reference in
infra/. - The owner's copies of the secret values (the password manager; Key Vault itself is single-region and its backups restore only into the same geography).
What does not: the Cosmos account and its continuous backup, the Function App, the Key Vault, Log Analytics, and the Static Web App's deployment.
The order:
- Decide. A regional outage that Azure expects to resolve in hours is waited out; the site's public pages stay up from the edge (ADR 0011). A restore elsewhere is for a loss Azure is not going to reverse.
- Storage. Initiate the account failover for
stsiteprodcus01so the exports and media become primary in the paired region. Read the export manifest from the secondary endpoint first; the failover takes time and the dry run (§2 above) needs only reads. - Compute and data, in a new region.
infra/names every resource withregion_abbreviation(defaultcus), so a second region is a second set of names, not a rename: set the region variables in the workspace, apply, and treat the result as a fresh estate. The Cosmos account it creates is the restore target: steps 3 and 4 of the drill, into that account, from the exports. - Secrets. Re-seed every Key Vault value from the owner's copies through the API Keys page or the seeding window; the required-inputs standard lists each name. The origin secret must change in its three places together.
- Edge. Repoint the Cloudflare records for the API and the site at the new hostnames; the Static Web App is redeployed by the frontend workflow.
- Verify with §5 above (public reads against the restored account) and
scripts/smoke-deployed.mjs.
Record what it took in the Drills table with the note regional, even when it
was real rather than rehearsed.
Drills¶
One row per drill, real restores and regional ones included. Add the row before tearing down (step 5 says when), in this shape:
| Column | What goes in it |
|---|---|
| Date | The day the drill ran, UTC, as YYYY-MM-DD. Nothing else in this cell: the check below reads it |
| RTO measured | $elapsed from step 5, hh:mm:ss. The objective is under 08:00:00 |
| RPO measured | Hours between the newest applied run in step 2 and $drillStart. The objective is under 24 h |
| Who | Who ran it |
| Applied runs | The full and the deltas step 4 applied, as step 2 names them |
| Documents | The count step 4 reports |
| Notes | regional for a regional restore; anything that went wrong |
A row whose Date is — is a placeholder, not a drill. Every Monday,
check-drill-age.yml
runs scripts/check-drill-age.mjs over this table and fails, opening an
issue or commenting on the open one, when the newest dated row is more than
100 days old or there is none: a quarter plus ten days' grace. A Date cell it
cannot read as a real, past calendar date fails the check too, so a typo
cannot pass as a drill, and so does a dated row with any cell but Notes left
empty or —: a date alone is not evidence of a drill. Keep the columns
exactly as headed below; the check reads them by name. Run it locally the
same way:
Exit 0 and Last drill: <date>, <n> day(s) ago is a current drill; exit 1
names how old the last one is, or says there is none; exit 2 means the table
could not be read and quotes the line.
| Date | RTO measured | RPO measured | Who | Applied runs | Documents | Notes |
|---|---|---|---|---|---|---|
| 2026-10-13 | 01:39:04 | 22 h | saulpatinojr | full/2026-10-11 + 1 delta (2026-10-12) | 11971 | First drill. Found and fixed three restore-script bugs (#1102), and step 5 needed Azurite and a Node count; the RTO includes those fixes. All 62 containers matched the backups document for document |