Skip to content

CHANGELOG

Completed features, fixes, enhancements, security fixes, and released changes.

Classification (Code Review SOP, CODE_REVIEW_PROMPT.md v1.0, Phase 10): this file records completed work only. All outstanding work, including owner-gated work, is a GitHub issue (owner decision 2026-09-05; TODO.md keeps the accepted risks and an index to the issues); required-input references and formats belong in Required-Inputs. REVIEW.md was retired on 2026-08-29 and must not be recreated.

Format follows Keep a Changelog. This project has not cut a tagged release; entries are grouped under [Unreleased] and reference the pull request that landed them.

[Unreleased]

Added

  • The lab image builds PyYAML's wheel from its source archive, so the move off Python 3.14 does not wait for PyYAML (#1505). ansible-core imports PyYAML, and the image installs hash-checked wheels only. PyYAML 6.0.3 publishes no wheel for CPython 3.15, and since PyPI stopped accepting new files on a release older than 14 days only a new PyYAML release can add one (yaml/pyyaml#951). The owner's decision of 2026-11-02 is not to wait. The fetch stage of lab-image/Dockerfile now downloads PyYAML's source archive by its PyPI SHA256 (PYYAML_SDIST_SHA256 in a new, dated block of lab-image/versions.env), installs Cython, setuptools and wheel as hash-checked wheels (PYYAML_BUILD_DEPS, checked against the archive's own build requirements), builds the wheel in a step that runs with no network at all, links libyaml in statically (the runner has no libyaml shared library), and writes the built wheel's own hash into the runner's requirements file in place of the PyPI wheel's line. The runner's offline, hash-checked install does not change; nothing new reaches it. The build stops unless the wheel is a compiled cp3xx wheel whose extension links no libyaml dynamically and imports with the C loader, and smoke.sh checks the shipped image for the pinned release with yaml.__with_libyaml__ true. Checked locally on python:3.15.2-slim-trixie with only the base lines moved: the PyYAML step built a cp315 wheel the same way, and the build then stopped at the next step on markupsafe==3.0.3, a pin with no cp315 wheel that #1468 moves; PyYAML is no longer what blocks it. A pyyaml-wheel stage holds the built wheel alone, for the workflow in the next entry to publish; the pyyaml line in ANSIBLE_CORE_DEPS stays for the host, which installs PyPI's wheel until the entry after next takes effect. The block says it is deleted the day PyYAML publishes a wheel for the base's Python, and scripts/lab-image-pyyaml-wheel.test.mjs holds its shape.
  • A workflow builds PyYAML's wheel out of the lab image, attests it and attaches it to a release, so the lab host can install it by hash (#1505). ansible-core imports PyYAML, and both the lab image and the lab host install Python packages only as wheels with a known SHA256; PyYAML publishes no wheel for the CPython line the image is moving to, and only a new PyYAML release can add one. The image builds the wheel itself from PyYAML's hashed source archive (the Dockerfile side of #1505); the host cannot build from source (LAB-13), so publish-pyyaml-wheel.yml takes the one file out of the image's pyyaml-wheel stage, attests it with actions/attest-build-provenance, and attaches the wheel and its attestation bundle to a release tagged pyyaml-<version>-cp<abi> (pyyaml-6.0.3-cp314 today; -cp315 once #1468 moves the base), read from PYYAML_VERSION and BASE_PYTHON_VERSION in lab-image/versions.env. Three jobs on the publish-lab-image.yml pattern: plan (every trigger, read-only; on a pull request it is the whole run and says what a push would publish), build (push or manual run) and publish (push to main or a manual run from main), which holds the Sigstore and attestations grants and writes the release with the pin pull requests' GitHub App token scoped to contents: write for an hour, not with a contents: write grant, because workflow-write-permissions.test.mjs keeps that list empty. The first wheel published for a tag is the wheel for that tag: a re-run downloads it, checks its SHA256 against the release notes and its provenance with gh attestation verify, and reports the URL and SHA256 as a no-op; a tag whose notes name a different source archive, an asset that does not verify, or a different file appearing during the run fails the run, and --clobber is never used. The run summary names the asset URL, the SHA256 and the attestation. scripts/publish-pyyaml-wheel-workflow.test.mjs holds the grants, the pins (the same as publish-lab-image.yml's), the timeouts and the no-replace rule, and runs the pin-reading step under bash against a fixture. The runbook is docs/runbooks/pyyaml-wheel.md, with the PowerShell lines to verify an asset by hand and the list of what to delete the day PyYAML publishes a wheel for the base's Python.

  • The lab host takes that published wheel by hash and attestation (#1505). lab-image/versions.env gains PYYAML_WHEEL_SHA256 at the end of the PyYAML block, the SHA256 of the wheel the workflow attached to this repository's release for the base's Python line (the workflow's step summary prints the line to paste). While it is empty nothing changes: lab-host/bootstrap.sh installs PyPI's wheel as before and says so in its log. Once it is set, bootstrap.sh downloads the wheel and its attestation bundle from the release, refuses a file whose SHA256 differs, checks the provenance offline with cosign against the workflow's identity on main (the check the addons role makes on the add-on images), hands the file to uv through --find-links, and rewrites the pyyaml line of the requirements file to that SHA256, so --require-hashes still decides what is installed; the pins are still read as data, never sourced. A built wheel is not byte-identical from one build to the next, so the value is always the hash of the exact asset on the release, never one recomputed locally. Five new cases in scripts/lab-host-bootstrap-ansible-core.test.mjs run the block with stand-ins for uv, curl and cosign: the happy path, the empty pin, a wrong SHA256, a failed attestation, and a PYYAML_VERSION that disagrees with the pyyaml line.

  • The daily DNS audit now recognises Cloudflare renewing its own certificate (#1421). On 2026-10-26 the audit opened a security issue for 12 changes in hybridcloudworks.com: TXT records at _acme-challenge.hybridcloudworks.com created in pairs and deleted minutes later, by an actor carrying type: system and nothing else, with no request method and no user agent; the same pattern came back on 10-27 and 10-30. That is Cloudflare validating the zone's universal certificate, which renews from 30 days before expiry (Certificate validity periods): the zone is a full setup, and for one Cloudflare "handles DCV automatically on your behalf using a TXT record" (Changing DCV method; the name is the _acme-challenge.<domain> of its TXT method), and its audit-log reference defines a system actor as changes made automatically by Cloudflare systems. A fifth allowlist entry, cloudflare-dcv in scripts/cloudflare-dns-audit.mjs, allows exactly that shape and nothing wider: a TXT named _acme-challenge.<apex>, created or deleted by an actor whose only field is type: system. A token cannot take that shape, because a token's request is authenticated and lands as account or user with context: api_token; a challenge for www or a lab name, an update, or a system actor on any other record still reads UNEXPECTED, and the tests say so for each. The report names such an actor Cloudflare (system, no token) instead of system unknown. The runbook's allowlist table has the row.

  • The Health page shows the content agent, the day's platform jobs, and says when a read could not be made (#1491). Owner request of 2026-11-01 after the Koda work landed. The ops snapshot carries an agent block (Koda's key present, active, last used and rotated, never its hint or hash; the switches that are off; the last day's refusals by reason, from the content-free agent-content-denial audit rows) and operationalSignals.jobs24h (the last day's jobs by status). Overview gains a Content agent (Koda) card beside Operational Signals, which now shows the jobs strip; Checks gains a Content agent probe under Platform, recorded by the pulse and held to the same verdict on both sides by statusParity.test.js. Two honesty fixes ride along: a failed read of the jobs container used to look like "no failed jobs" and now says it could not look, and the Publishing Operations card shows the scheduler's own status word and recorded error, and explains that all-zero counts mean nothing in Approved had come due rather than a broken card (only after a run that completed; a run that broke shows its error instead). Every denial row the guards write now also carries its kind as action and its time as timestamp, the two paths the audit container indexes, so the Health read is an index seek and not a scan of ninety days of audit rows.

  • A Cost page under Platform: what the platform costs this month, and whether it will go over (owner request 2026-11-01, ADR 0039). /admin/cost, for editors, has five tabs. Overview shows the month at a glance (Azure against its budgets with the end-of-month estimate, AI spend, subscriptions per month, the alerts count, the last twelve months) and when each source was last read, with a Refresh. Azure shows a card per subscription or resource group with a budget: spend so far, Azure's own forecast, the daily line and the top services, read from Cost Management with the Function App's managed identity (functions/src/lib/cost/azure.js). AI shows the usage ledger's estimate for the month by provider and by task, ElevenLabs' credits left, the image budget and the Foundry budget. Services is a ledger the owner keeps of the subscriptions paid outside Azure (price, cycle, renewal date, auto-renew), with one-click suggestions for vendors the site already uses. Alerts lists every cost alert with its rule and holds the settings. The figures come from one snapshot (admin_config/cost_snapshot) rebuilt every six hours by the armed checkAgentHealth timer and on Refresh (at most once in five minutes); the ledger (admin_config/cost_ledger) is read live. Warning and critical alerts also go to the dashboard's Decision Center (a new cost source), to Telegram once each, and a monthly closing digest goes to Telegram on the first; renewals appear on the Calendar as a new renewal kind. Koda can read the page under its Platform switch and change nothing. Seven routes under cms/cost/ (functions/src/functions/cost-http.js); Terraform adds the COST_SCOPES app setting. The read needs the built-in Cost Management Reader role on the application and Platform Management subscriptions, which the owner grants once by hand (runbook), because the Terraform identity creates no role assignment at subscription scope.
  • Every draft answer carries an edit link and a preview link (owner request 2026-11-01, #1477). The Drafts routes, on the admin API and on Koda's agent API alike, now put editUrl (the Drafts page with that draft open, /admin/drafts?draft=<id>) and previewUrl (the public preview page with a signed token that expires after 72 hours, or null while the preview secret is not seeded) on every draft, in the list and alone (functions/src/lib/cms/drafts-handlers.js). The preview route now shows drafts on the desk and in review, not only content past review; the Drafts page opens the draft a link names, and shows the preview link beside the open draft.

  • Koda reaches the admin sections the owner named, creating and reviewing but never deleting (#1477, ADR 0038). The owner's instruction of 2026-11-01: Koda will be creating content and should have the Pipeline, Enhanced, Spotlight and Second Brain sections and the Platform health and integration reads, with no deletion. The same key now also passes the admin role guard for an allow-list of about a hundred routes (functions/src/lib/agent-content/admin-access.js), each under one of five new switches on the Koda tab. A DELETE never passes; publishing, scheduling, unpublishing, slugs, settings, secrets, admins and every section the owner did not name stay closed; the routes that could reach publishing through their body are limited (a status change to published or archived, a publish-content job, a content item born published, a book or chapter archived are refused), and the routes that unpublish as a side effect (request inspection, reset review state, the episode review) are not open; in the Second Brain the key acts as the owner so its notes land in the owner's workspace. Every refusal is an audit row with the reason. The tab and GET …/agent-content/v1/status list the routes per section. The agent API's own DELETE drafts/{id} route and its Delete drafts switch are removed: the key never deletes. The runbook's new part 7 is the matrix.

  • An hourly workflow removes the firewall windows a dead run left open, and puts the host storage account back to Deny (#1463). A "window" is a temporary allow rule for a workflow's own runner: ci-deploy-scm-<run id>, ci-smoke-<run id> and ci-manifest-<run id> on the Function App or its SCM site, and one IP rule on the host storage account with its default action flipped to Allow. Each is closed by an always() step, which does not run when the runner is lost or the job is force-terminated. The hourly monitor has named a leftover since #1050; now remove-ci-access-windows.yml runs scripts/remove-ci-access-windows.mjs half an hour after it: a ci-deploy-scm-, ci-smoke- or ci-manifest- rule whose run has ended is removed, one whose run is still in progress is left alone, and any other ci-* rule is left for a person. The storage account's IP rules carry no names, and Terraform declares an operator's own rule and an Allow rollback that look just like a leftover, so the deploy's open step now writes a ciwindow tag on the account (the run id and the runner's address) before it adds the rule, and its close step clears it last; the remover restores only a window whose tag names a run that has ended, doing what that run's close step would have (Deny, that one address, the tag), reads the tag again before each write and stops if a new deploy has taken the window, and reports an open firewall with no tag without touching it. It signs in as the manifest-window identity, not the deploy identity, because the deploy identity trusts only the production environment's subject and that environment's required reviewers would hold an hourly job every hour. The Function App half needs nothing new; the storage half needs a new custom role, HCW Storage Firewall Writer (infra/roles/storage-firewall-writer.json: the account's read and write actions, no listKeys), which the owner creates once, and its assignment in infra/oidc.tf, declared in scripts/assert-expected-plan.mjs. The lookup and the assignment are gated on a new workspace variable, storage_firewall_remover_enabled (default false), because a lookup of a role that does not exist did not fail the plan on 2026-11-04, it hung HCP Terraform's one plan slot with every other run queued behind it; the owner flips it after creating the role. Unit tests hold each decision, including a rule whose run is still running, and a workflow test runs the step under bash -e to prove the exit code reaches the job summary.

  • The Health page says whether last night's Cosmos export ran and what it covered (#1409). The nightly export writes its run manifest into the private cosmos-export blob container, which refuses every workstation, and the log workspace has dropped the 03:00 UTC window before (#1530), so the owner could not see whether a run completed or carried brain_records without the portal. The functions worker, which has the identity and the network path, now reads the run records and manifests of the last seven UTC dates (functions/src/lib/backup/backup-status.js, names and counts only, cached ten minutes, no new timer) and the ops snapshot carries a backups block: the newest completed run, its mode and times, its containers with record counts, how many authored containers are in it and which are missing, and brain_records on its own with its partition key /ownerId. Overview shows it under Operational Signals; Checks gains a Nightly export probe under Platform (critical when the newest completed run is older than 26 hours or an authored container is missing; "not known", never healthy, when the manifests could not be read), recorded by the pulse and held to one verdict with the page by statusParity.test.js. The Cosmos restore runbook says how to read the tile.

  • A browser journey drives the Second Brain outline-to-draft handoff, and the journeys now run in CI (#1408). frontend/e2e/second-brain.spec.js gained a desktop and mobile journey that signs in through the stubbed identity, selects a made-up idea, asks for an outline and checks the request names that idea, shows two citations that link to their own sources and one missing-evidence item, fills the editor through Review as a new note, saves the note with the sources as its links, and creates one draft: a failed attempt and its retry carry the same request id, and the button is disabled once a draft exists. It found that a citation link clicked inside the page opened nothing (the page followed its ?item= link once per visit), which is fixed in SecondBrainPage.jsx with a unit test. The spec had never been in the e2e (playwright) job's list, so ci.yml now runs it, and the runbook says which of the ticket's lines the journeys cover and which still need the owner's eyes on the real answers.

Session continuity

  • Two review notes from #1459 are answered (#1455). .claude/CLAUDE.md now names the two separate controls that stop a session approving its own pull request: GitHub's rule that an author cannot approve, and CodeRabbit's allow_author_approval: false, which makes it ignore @coderabbitai approve from the author. It also records what cleared #1460: a fresh CodeRabbit review did not replace its old request for changes and CodeRabbit refuses resolve from the author, so the owner dismisses such a review, with a link to where CodeRabbit withdrew it, and a session does not. MEMORY.md's header now says that #1346 left its later list because #1461 closed it.

  • .claude/CLAUDE.md now says what the merge rules really require, and holds the standing rules that were only in session notes (#1455). The bullet that said the Default ruleset (the repository's merge rules) requires CodeRabbit's approving review is rewritten as "CodeRabbit's approval is the merge signal; its request for changes is a gate": read from the API on 2026-10-31, the ruleset requires no approving review (required_approving_review_count: 0, require_last_push_approval: false, unchanged since 2026-10-20), only resolved conversations, the 15 required checks on an up-to-date head, and no deleting or force-pushing main; waiting for CodeRabbit's approval is the repository's practice. A CodeRabbit review that asks for changes still blocks the merge until a newer review replaces it, even once every thread is resolved (the merge API refused #1460 with 1 review requesting changes by reviewers with write access), so the bullet says to ask for @coderabbitai review rather than dismiss the review. The option F history of 2026-10-18 (#972) is kept in brief, and making the approval a gate again is left to the owner on #1455. A new section, "Lines a session does not cross", adds the rules that were written only in the root MEMORY.md and earlier private notes: never print or ask for a secret, no real tenant, subscription or application id in a public file, an image-publishing tag needs the owner's yes and a green dry run on the exact commit, no admin bypass or force-push to main, and no terraform apply or Azure delete unless asked; it points to the rules the file already had (safety-check refusals, merging your own pull requests, CodeRabbit's boxes, one pull request at a time on the same lines) instead of repeating them. MEMORY.md now points to .claude/CLAUDE.md for those rules instead of listing them, and the comments in .coderabbit.yaml no longer call CodeRabbit's approval the ruleset's required review (comments only; no setting changed). It also records the owner's instruction of 2026-10-31 to call out every discrepancy a session notices (a repository, file or setting that differs from its siblings, the documentation or the ticket), with its link, whether it looks deliberate, and the fix. It also warns that the worktrees agents use share one git stash list, so agents set work aside with a temporary commit instead (two agents' uncommitted changes were swapped that way on 2026-10-31 and recovered), and that files in the shared scratchpad get task-unique names.

  • Root travel handoff and completed-work cleanup (#1450). Adds a dated MEMORY.md with merged work, verified application deployments, workflow behavior, and resume commands. Permits that exact root filename and includes it in redaction checks on future edits. Removes generated local artifacts and old task files, deletes local branches verified as merged (including squash merges), and prunes remote/worktree references while retaining installed tools and dependencies.

  • The root MEMORY.md now also carries the infrastructure, CI and add-on state (#1450). A second session wrote the section "Infrastructure, CI and add-on work" into the same file instead of adding another one: what was merged today with links, what to do next in order, the dates to remember, and the few things only the owner can do (each with its command and what a good result looks like). The file is still scanned by the redaction gate, so it holds no identifiers, local paths or private repository names.

Fixed

  • minimatch moved to 11.0.4 in frontend, functions and scripts by an npm override, closing Dependabot's three medium alerts for GHSA-g3hv-75wg-m249 (globparts amplification) as far as they can be closed. No consumer (eslint 10.14, @eslint/config-array, @typescript-eslint/typescript-estree) asks for the fixed major yet, so the override carries it; lint and the full test suites pass in all three packages on it. The one copy that cannot move is eslint-plugin-jsx-a11y's minimatch 3.1.5, kept by the scoped override from #220 because the plugin imports a default export that minimatch 7 and later do not have; that copy has none of the code the advisory describes (the preprocessing it exploits arrived with minimatch 7), so the frontend alert, if it stays open on that copy, is one for the owner to dismiss as not affected.
  • The Docker Scout workflow test no longer pins docker/scout-action to one commit, so Dependabot's bumps of it stay green (#1502). The test held the action's commit as a literal and went red on the first bump; it now checks the shape, a full commit with the release as its comment, and the workflow's prose comment no longer repeats a version Dependabot cannot edit.
  • The waiting-to-go-live tile no longer shows "0h" over an item it cannot date, "Published, Not Live (6 h+)" is counted instead of sampled, and the Decision Center words the expected secrets the way the Health page does (#1516). After #1513 shipped, the Operational Signals tile read 0h over "1 approved or published, not live, not scheduled": its ordered read takes only items with a string updatedAt, and the one waiting item has none. The snapshot now also reads the waiting items without one (up to 50) and ages them by the first time they carry, Firestore-style { seconds } times included; when items wait and none carries any time, the tile says no recorded time instead of 0 h and the card turns Degraded, as a read that could not count does. An empty updatedAt no longer sorts first and hides the real oldest item. The publishing watchdog now filters for published, not live and unscheduled in the query (absent Live included) and reads up to 1,000 candidates, so its count is exact below that and shown with a "+" at it (on the alert too); it used to read 250 of every non-live item in no order. Its overdue-schedule read now also treats an item with no Live field as not live, like the new read. The Decision Center lists AZURE_SPEECH_KEY, CLOUDFLARE_ANALYTICS_TOKEN and CODER_BROKER_TOKEN as "unresolved on purpose, waiting on you", low priority, with each one's reason, and any other unresolved reference as a high-priority fault. The Telegram /queue reply says when items wait with no time rather than "Nothing is currently stuck in staging".
  • The Cost page's budget bar and budget alerts count this month's spend even when Azure's budget figure lags. On 2026-11-03 the Application subscription's card showed site-monthly-budget at "0% used" while the cost query had USD 7.33 of its USD 150 (about 5 %): Azure updates the spend it keeps on a budget on its own schedule, and the bar and the 50/75/90 % alerts both read that figure, so they would have warned late. When a scope's cost query is read, the budget's spend is now the higher of Azure's figure and the queried month (functions/src/lib/cost/azure.js withSpendAtLeast), and the Azure tab says so under the bar when the queried figure is the one shown. Azure's own forecast is unchanged.
  • The Cost page shows each subscription its own budget, not a resource group's. On 2026-11-03 the Application subscription's card showed site-foundry-monthly-budget ($75), the Microsoft Foundry resource group's budget, with the group's spend as the subscription's. Listing a subscription's budgets also returns the budgets of its resource groups, and the reader took the first by name. It now keeps only the budgets set on the scope itself, told apart by each budget's id (functions/src/lib/cost/azure.js summarizeBudgets), so the subscription shows site-monthly-budget and the Overview's Azure total and budget add up the right budgets.
  • The Cost page shows each subscription's month from its budget when the cost query is throttled, and asks for last month by its dates. With each failing read now named (2026-11-03), the Azure tab showed two causes. Last month's total was refused with 400 BadRequest when asked with the TheLastMonth shortcut, so it is now asked with a Custom period of last month's dates and granularity None, the form the cost analysis measured September with. Both subscriptions' cost queries were still throttled (HTTP 429), and the budget list, read after them, was never reached. The budget list now comes first (it is the Consumption API, which the Query API's allowance does not cover), and when the cost query is refused the card shows the budget's own month-to-date spend and forecast with a note instead of nothing (functions/src/lib/cost/azure.js). A code that only repeats the status ("HTTP 429 429") is no longer shown twice. The runbook records the 400 and explains the new note.
  • The Cost page says which Azure read failed and with what code, and one refused read no longer blanks a card. After the rate-limit fix went live on 2026-11-03, the Azure tab said only "Azure answered HTTP 400" for every scope, which named neither the read nor the reason. Each refusal now names the read (the budgets list, this month's costs, or last month's total) and Azure's own error code when it is code-shaped; Azure's sentence is still never copied (functions/src/lib/cost/azure.js describeArmStatus, azureErrorCode). This month's costs are read first, and if the budgets list or last month's total is refused the card still shows the month with a note saying what could not be read. The runbook's table has a row for the 400.
  • The Cost page reads Azure within Cost Management's rate limits, and keeps its last good figures when it cannot. On 2026-11-03, right after the owner granted the role, every scope answered HTTP 429: Cost Management allows each Azure directory 12 query units every 10 seconds, 60 a minute and 600 an hour, shared with Cost Analysis in the portal (Microsoft Learn), and a refresh sent all six of its queries at once. The reader (functions/src/lib/cost/azure.js) now asks one question at a time, waits as long as Azure's qpu-retry-after header says (at most 30 seconds) and tries once more, asks for last month's total only once a month, and, when a refresh is still refused, keeps this month's last good figures with a note saying when they were read (an info alert says so too). The Azure tab no longer tells the owner to grant the role when the reason is something else; it points to the runbook, whose table now explains the 429.
  • The Health page stops calling the three expected unresolved secrets Critical, and its two "staged" numbers say what they count (#1512). The list of Key Vault references that are unresolved on purpose (today the fallback speech key, the Cloudflare analytics token and the Coder broker token, each with its reason) moved from the nightly monitor script into functions/src/lib/expected-unresolved.js, and the monitor imports it from there, so the two read one list. The ops snapshot now splits unresolved names into expectedUnresolved (with reasons) and unexpectedUnresolved; Pipeline Readiness lists the expected ones under Unresolved on purpose, each with its reason, and both Key Vault probes name them as "unresolved on purpose, waiting on the owner"; the card, the Runtime config chip and the probes stay Critical, by name on the card and the probes, for anything off the list. The live probe states its one assumption: it reads a count, not names, from the anonymous endpoint. The public /api/health count is unchanged. On Operational Signals, the oldest-staged tile was the oldest of 50 items read in no order and then filtered for Live, so it showed whichever old item happened to be in the sample; it now reads the oldest Approved or Published item that is not live and not scheduled for later, ordered by its last edit (only rows with a date take part in the order), with the count waiting, under the name Oldest Waiting to Go Live. The Publishing card's Staged Too Long is renamed Published, Not Live (6 h+), which is what the watchdog counts.
  • The Cost page draws before its first reading instead of showing "This page hit an error". Right after the deploy of 2026-11-01, before the Azure and AI figures had ever been read (or while Azure cannot be read), the summary carried no AI month and no Azure totals, and the Overview, Azure and AI tabs read both without checking, so the page stopped with Cannot read properties of null (reading 'monthToDate'). The server now always answers a zero-filled AI month (functions/src/lib/cost/snapshot.js), and the page fills in empty figures itself (frontend/src/lib/cost.js withEmptyFigures), so it also works against the server already deployed. A new test renders every tab with the server's real answers for those two cases, generated from the backend's own code; it fails with the same error without the fix.
  • The Cost page runbook's one-time grant now works for an account in several directories, and says when it fails (runbook). On 2026-11-02 the owner's first run signed in to the account's home directory, not the one holding the production subscriptions, so Azure refused both grants with InvalidAuthenticationTokenTenant, yet the command still printed done for each, because it printed that unconditionally. A new step signs in to the directory that owns the application subscription, looked up at run time so no directory id is written down, and the grant line prints FAILED when Azure refuses a step and stops when the app's identity cannot be read. With both fixes the owner's run granted Cost Management Reader on both subscriptions.
  • The Health page counts an unresolved Key Vault reference once, not twice. On 2026-11-01 the Pipeline Readiness card and the public /api/health count read 6 for 3 references: App Service puts every app setting into the process twice, as NAME and as APPSETTING_NAME, and secrets-health.js counted both. It now leaves out the APPSETTING_ twin when the plain name is present. The three that remain are unresolved on purpose (the fallback speech key, the Cloudflare analytics token and the Coder broker token, all waiting on the owner). In the same change the expected-unresolved list in scripts/check-unresolved-secrets.mjs drops the two Podbean values and the StreamYard key, which the owner has seeded and the monitor was reporting as resolving-but-still-listed, and the Terraform comments beside those three settings say so. The edge and DNS runbook's health step, which still called unresolvedSecrets: 2 the healthy reading, now points at the list for the number.
  • The restore-drill check opens its issue again, and the Key Vault check says when it could not tell. These are two more workflows with the same flaw the delivery monitor had (the entry below). A workflow step with no shell: runs as bash -e, which stops at the first failing command, so a line such as code=$? written after a script never ran when the script failed. In check-drill-age.yml (the Monday check that the Cosmos restore drill is at most 100 days old), an overdue drill or an unreadable Drills table ended the step before it recorded the result, so the steps that open the "Cosmos restore drill overdue" issue and fail the run with the reason were skipped. That happened on 2026-10-12 (run 38279975200): the drill was overdue and no issue was opened. The drill was run on 2026-10-13 and stays current through 2027-01-21, so nothing is due now; the first Monday check that would open the issue is 2027-01-25, and it now will. The step also writes a short summary of its own if the script ever crashes before writing one, because a missing summary file would skip the issue in the same way. In monitor-unresolved-secrets.yml (the check that the Function App's Key Vault references resolve), a payload the script did not understand ended the step with code 2 before it printed the lines saying the check could not tell; the run was red either way. Both now start the code at 0 and read it as command || code=$?, which -e does not stop. scripts/workflow-exit-code-steps.test.mjs runs both steps under bash -e with stand-ins for node and az, and scans every workflow for the old shape, so a new step cannot bring it back. It failed on the old steps before the fix, and run against the workflows as they were before #1488 it also finds the delivery monitor's two steps.
  • The delivery monitor prints its report on a red run, not only on a green one. In monitor-deploy-drift.yml the steps "Measure the drift" and "Read the run history" captured the script's report with report=$(node ...) and read the exit code with status=$? on the next line. Neither step sets shell:, so GitHub runs it as bash -e {0}, and under -e (stop at the first failing command; set -uo pipefail does not turn it off) a failing capture ends the step on that line. So when the script found drift (exit 1) or could not read (exit 2), the step stopped before it printed anything, and the log and the step summary were blank in exactly the runs that went red: runs 41219040554, 41238281521, 41258066344, 41272679445 and 41325654390 each go from the env block straight to Process completed with exit code 1., while green run 41306150963 prints the table. Both steps now set status=0 and capture with report=$(...) || status=$?, which -e does not cut short (the shape ci.yml and monitor-functions-registered.yml already use), so the report is printed and the job still ends red. The new scripts/monitor-deploy-drift-steps.test.mjs takes each step's shell out of the workflow and runs it with bash -e against a stand-in node that exits 0, 1 and 2, checking the report reaches both the log and the summary and the exit code is kept; it also proves the old shape printed nothing. check-workflow-health.invocation.test.mjs now pins the new line instead of the old status=$?.
  • Koda's key is read in the shapes connectors send (#1477). Muse's first calls on 2026-11-01 carried a working key and were answered 401, because the Agent Content API read the key only from Authorization: Bearer <key>. The guard (functions/src/lib/agent-content/guard.js) now also reads it as the bare Authorization value and from an X-API-Key or X-Agent-Key header, never from the address; Authorization wins when both are present, and a value that is not shaped like a site key is still refused without an audit row. The runbook, ADR 0037 and the API contract say so.
  • The lab agent guard's refusal record no longer becomes an unhandled rejection when Cosmos refuses the write (#1476). The guard that checks the lab agent's token writes one row to admin_audit_logs for every request it refuses, and calls that writer without waiting for it, so a write refused under throttling (Cosmos answering 429) rejected with nothing handling it; Node treats that as fatal by default. default-agent-guard.js now writes through writeDenialAudit, the shared sink the admin and workspace guards got on #1465, which awaits the write inside its own try/catch and logs one content-free line (the error's code, never its message); the row also carries action and timestamp, the two names the container indexes (infra/cosmos-containers.json), like every other guard's row. The guard's 401 and 403 answers are unchanged. A new default-agent-guard-denial-audit.test.js drives the real guard and real wiring with a plain-function Cosmos double (a vi.fn() spy attaches its own handler and hides the rejection), listens for unhandledRejection, and failed on the old code with the rejected write in its list. Every other upsertDoc audit writer in functions/src/lib/ awaits its write, either inside a try/catch or where a failure becomes the handler's own 500; the PR lists them.
  • Visitor lab sessions: the last follow-ups from the two reviews are done, and the door is declared in Terraform (#1400, #1403). LABS_VISITOR_SESSIONS_ENABLED, the switch that opens the visitor lab sessions, was read by functions/src/lib/labs/visitor-session.js and declared by nothing in infra/, so a copy set by hand would have been removed by the next apply. infra/functionapp.tf now sets it from the new variable labs_visitor_sessions_enabled (infra/variables.tf, false by default: the door stays closed, as an absent setting kept it, until the owner sets the variable in the hcw-azure workspace), scripts/assert-expected-plan.mjs declares the setting appearing as "false", and a test in functions/src/lib/secret-catalog.test.js ties the name the code reads to the declaration and its default. The sweep (reapExpiredVisitorSessions) now stops taking up sessions 260 seconds after it started (SWEEP_DEADLINE_MS): the claim it puts on a session lasts 5 minutes from the run's start, and ten slow sessions could have outlasted it, so a second run could have taken a session over while the first was still working on it and spent a try on a second, harmless delete. What is left waits for the next run, and the log says how many. The three queries on lab_jobs are bounded with TOP (the sweep reads no more than the 10 sessions it will take up; the status and start routes read at most 100 records), with a parameter, as Microsoft Learn's "Parameterized queries" page shows it; the test store's runCosmosQuery (functions/test/cosmos-predicate.js) reads TOP the same way and refuses one that is not a whole number above zero. The remainder of item 2 (a user create that got no answer, and a session that could not be saved after both Coder calls worked) and the wording point on enabled_timers in docs/standards/required-inputs.md were already on main from #1464 and #1422, checked against the code and the tests rather than assumed; the Coder page for the user routes was read again on 2026-11-03 and still says {user} is "User ID, name, or me" for the delete. Item 4 of #1400 (a session belongs to an address, not to a secret) is accepted as it is, because the lab is free, time-boxed to an hour and holds nothing of the visitor's; the two ideas that wait (a circuit breaker while Coder is down, an alert on the sweep's Warning lines) are recorded on #1403 with why. Documentation: the lab host runbook ("Visitor lab sessions"), Required inputs §4.1 and §4.5, Variables and secrets, ADR 0032 (amendment of 2026-11-03). Terraform: a plan that adds one app setting, closed; the apply is the owner's in HCP Terraform, and nothing here was applied or run against Azure, Cosmos or Coder. The live check of #1400's item 5 (the deployed app refusing a request that did not come through Cloudflare, and the Cloudflare rule itself) stays the owner's: az functionapp config appsettings list -g rg-web-site-prod-cus -n func-site-prod-cus-01 -o json | ConvertFrom-Json | Where-Object name -in 'NODE_ENV','CF_ORIGIN_SECRET' | Select-Object name lists the two settings by name only.
  • The Cosmos request log is filtered at ingestion again, by a rule Azure actually runs, so the Log Analytics workspace stops hitting its daily cap every evening (#1530). The filter PLAT-8 (#968) wrote was a transformKql property on the CDBDataPlaneRequests table, and the Tables API has no such property: Azure kept the retention it was sent beside and dropped the filter without a word, so the table carried every read and write on every container, the workspace hit its 0.25 GB cap between 20:30 and 21:50 UTC every evening from at least 2026-10-27, and nothing was logged until 08:00 UTC, the 03:00 UTC Cosmos export's completion event included (which is why alert-cosmos-export-daily fired every night from 2026-10-29). The filter now lives where Azure runs one from: a data collection rule of kind WorkspaceTransforms (dcr-logs-transforms-prod-cus, infra/observability.tf), named as the workspace's default rule (infra/main.tf). Both are azurerm resources that read their state back, so a rule edited in the portal or a link removed shows in the next plan, which the azapi write never did. The table's retention moves to azurerm_log_analytics_workspace_table, which reads it back too. The filter keeps every request on ambassador, doc_vault and brain_records (the personal and private data), every write on every other container, and drops reads of ordinary content and everything on leases and brain_search, the change-feed bookkeeping and the hourly-rebuilt search index: measured over seven days on 2026-11-03, 95% of the table's volume (9.48 MB an hour in the hours before the cap, 0.48 kept). The runbooks and ADRs 0022 and 0028 say so instead of describing the filter that never ran.

Changed

  • The Second Brain validation record carries the restore drill of 2026-11-04 (#1409). brain_records was restored with Azure Cosmos DB point-in-time restore into a new, isolated account (cosmos-site-sbx-cus, deleted afterwards), its 84 documents matched the production count, a selected note matched its own revision chain, brain_search was rebuilt there from the restored notes plus made-up fixtures, and search and backlinks answered through the site's own code; 36 minutes to a verified restore. docs/architecture/second-brain-validation.md records the counts, the Learn pages and what was not verified: the nightly export manifest read (line 1 of the issue) still waits for the functions deployment that carries the Health page's Nightly export tile.
  • CI: the frontend unit suite and the browser suites each run as two shards, so a frontend pull request no longer waits on one runner for either (#1545). On the frontend-only #1557 the frontend row took 280 s (192 s of it the vitest run with coverage) and e2e (playwright) 247 s (195 s the browser suites), 302 s from push to all-green against the issue's target of under 240 s, with no placeholder job left to remove. A shard is one part of a suite run on its own runner beside the other part. The unit suite left the frontend row (which keeps lint, format, route validation and the build) for a frontend-unit job of two shards: each runs npm run test:shard -- --shard=N/2 (a new frontend/package.json script), which writes its half's results and coverage as a blob report and checks no threshold, because vitest checks a shard's coverage like any other run's and half a suite is always under the ratchet (shard 1 of 2 measured 48% of lines against 75.5%); a frontend-unit-merge job downloads both blobs and runs npm run test:merge (vitest run --merge-reports --coverage), where the thresholds in frontend/vitest.config.js are checked once on the whole suite and, after a merge to main, the merged lcov.info is what coverage (qlty) uploads. The browser suites run as two Playwright shards of the e2e job, each with its own build, preview server and failure report. gate needs every new job, and a skipped shard counts as skipped. scripts/ci-workflow-steps.test.mjs holds the shape; the required checks runbook has the section "The two slow suites run as shards".
  • CI: the verify matrix is built from the path filter, so a row whose files did not change starts no runner (#1545). The five rows of the verify job (frontend, functions (azure), scripts (operations), vps-agent, edge (availability probe)) each started a runner on every pull request and, when their files had not changed, exited after 8 to 13 seconds, because a matrix row cannot carry an if: of its own (five such rows on the docs-only #1555). The changes job in .github/workflows/ci.yml now decides each row's filter in a detect step of its own and builds the matrix as JSON in a last step, "Build the verify matrix", which the verify job reads with fromJSON; a row left out is never created. The scripts (operations) row is always created, because three of its steps run on every pull request whatever changed, and its answer travels as matrix.relevant to decide its lint and tests. The filters moved with their comments and are unchanged; scripts/version-floors.json reads the functions row's Node.js 24 line from its new place. scripts/ci-workflow-steps.test.mjs runs the five detect steps and the build step with bash and checks the matrix for a docs-only change (the scripts row alone), a frontend change and a push against the same answer worked out in JavaScript; docs/runbooks/required-checks.md describes it.
  • Lab image: the Azure CLI moved to 2.92.0 and the alz provider to 0.22.1, and the CLI's pip override went (#1519). The weekly scan's bump was red because Azure CLI 2.92.0 now ships pip 26.2.2 itself, the version AZURE_CLI_PYTHON_OVERRIDES in lab-image/versions.env pinned, and the full build refuses an override the CLI already carries. That line is deleted; certifi, pyjwt and urllib3 stay, the CLI still ships older ones. The alz checksum was read again from the Terraform registry and matches the bot's reading.
  • The lab host's PyYAML pin moves to the CPython 3.15 wheel (#1505). #1468 moved the lab image and the lab host to CPython 3.15.2 and merged before #1508, so main briefly pinned the cp314 wheel's SHA256 on a 3.15 host, which bootstrap.sh would have refused at the hash check (the host names the pyyaml-6.0.3-cp315 release from its own Python line, and that asset is another file). PYYAML_WHEEL_SHA256 in lab-image/versions.env is now the wheel the first publish-pyyaml-wheel.yml run on main after that move attached to the pyyaml-6.0.3-cp315 release (run 41557242736), verified by hand with gh attestation verify --bundle --cert-identity against the workflow's identity on main. No host run happened in between.
  • The lab host now takes the published PyYAML wheel (#1505). PYYAML_WHEEL_SHA256 in lab-image/versions.env is set to the SHA256 of the wheel the first run of publish-pyyaml-wheel.yml on main attached to the pyyaml-6.0.3-cp314 release (run 41555864123; the asset verified by hand with gh attestation verify --bundle --cert-identity against the workflow's identity on main). From the next bootstrap.sh run the host downloads that wheel and its bundle, checks the SHA256 and the provenance with cosign, and installs it in place of PyPI's wheel; the one-line change makes a new generation of the ansible-core environment, which is the first live proof of the host's path.
  • CodeRabbit reviews are asked for, not automatic (#1487). Owner decision 2026-11-01, on the free plan. .coderabbit.yaml now has reviews.auto_review.enabled: false: every push and every "Update branch" to an open pull request spent one of the five included reviews an hour, and on 2026-11-01 three pull requests used the hour between them while #1485 waited for the review that would lift a stale request for changes. A push or an "Update branch" now starts no review and spends none of the allowance; only an explicit ask does. A session comments @coderabbitai review once the required checks are green on the head it wants read, which works whether or not automatic reviews are on (CodeRabbit: automatic review controls). .claude/CLAUDE.md says so in "Each bot's first review is waited for, and CodeRabbit's is asked for" and in the merge-signal bullet; the rest of the review practice (three rounds, every thread answered and resolved, CodeRabbit's approval of the current head as the merge signal) is unchanged. No paid add-on is turned on.
  • The plan check's declared changes are pruned to what is still pending (#1479). scripts/assert-expected-plan.mjs lists the one-off changes a plan may carry, and the file's own rule is that an entry is deleted once its change has applied. The plan check on the run for 1ccc72a (actions run 41200891363) showed about twenty entries as "declared, not in this plan". The ones that infra/ sets unconditionally, so that no diff means applied, are deleted: the Second Brain's two containers and five settings, the Doc Vault container, the Cloudflare analytics settings, the two timer flags catalogued disarmed, the Coder broker token reference, alert-timer-overdue's three added rows (now part of the applied query the file compares against) and the Code Conductor retirement itself, which applied in run-BN7KzWpfvXom9SFC. The three arming plans, the SSH-alert loopback entries and the Web Analytics site create stay, because each waits on a value the owner sets in the workspace and "not in this plan" cannot tell that from applied. Tests follow: the entry-by-entry cases for the deleted declarations are gone, a prune test pins what left and what stayed, and a new case checks the applied timer rows match the tree while no change to the rule is declared.
  • AWS Learn catalogue: SAP-C02 and SAP-C03 re-read ahead of their day, so the build stays green on 2026-11-17 (#1497). On that day the catalogue alarm (frontend/src/data/education-catalogues.test.js, which fails on every branch the day a stored status is overtaken by its own dates) would have named both rows of frontend/src/data/aws/certifications.js. Re-read on 2026-11-03 against AWS's September announcement (September 2026 new offerings, whose Key dates still read "November 16, 2026 – Last day to take SAP-C02 / November 17, 2026 – SAP-C03 GA delivery begins"), the exam page and the hub. SAP-C02 is stored retired with its last day kept and SAP-C03 active from 2026-11-17, each with statusReadDate 2026-11-03, so under rule 2 of deriveStatus the badges keep saying "Retiring" and "Coming" until AWS's day and nothing moves early. The SAP-C03 exam guide and the DVA-C03 exam guide resolve now that registration has opened, so both rows link them (they were 404 on 2026-10-05). The header records the pages and wording, and a new test block holds the two rows and the days they turn.
  • The dependency-review runbook says where the Python add-ons' base-image pull requests come from, and why the website's weekly run does not open them (#1386). That issue recommended that lab-supply-chain.yml open a pull request in each Python add-on when the website's base moves. Before that was built, a read of the add-on repositories on 2026-11-03 found each already carries a daily base-update workflow (added 2026-10-27, the day after the issue) that opens that pull request with the add-on's own App token and has opened real ones that a person then merged (Cloud Assessor #55 and #57), so a second opener here would have opened a competing pull request in each add-on. Nothing is built here; the runbook now records the mechanism, the one move it refuses (a Python minor line, which is why the three add-ons' daily runs have been red since the website moved to 3.15 on 2026-10-31, for example Cloud Assessor run 41597788687) and the read-only check. The paragraph above BASE_IMAGE in lab-image/versions.env was read again against the pinned digest and records that the stale wording the issue named had already gone with #1468.
  • The plan check forgets the five declared changes the #1530 and #1469 applies consumed (#1530, #1469). scripts/assert-expected-plan.mjs lists the one-off changes an HCP Terraform plan may carry, and its own rule is that an entry is deleted once its change has applied, so a later drift of the same thing is caught instead of waved through by a stale line. Two runs applied on 2026-11-04: run-fQRFPxBy7CowV6zG, for #1535 (the Cosmos request-log filter: the delete of the azapi table resource and the creates of the azurerm table, the workspace transformation rule and alert-logs-overquota), and run-vUQrQLSGJh4J5HZ2, for #1531 (the Function App's Cosmos data role on the hcw database). Those five entries are gone and the file's running list of what is in state names them with their runs. The tests follow: the #1469 case now holds that the applied hcw grant is not declared again while any database key added later must be, and a new case holds that the four #1530 declarations are gone while their three resources are still in infra/.
  • CI runs only the jobs a change needs, and one gate check stands for all of them (#1545). Owner reading of 2026-11-04: too many workflows, placeholder jobs, and every merge re-running every other open pull request. ci.yml now starts with a changes job that diffs the pull request once and answers whether the browser suites, the lab-host Ansible checks, the Coder template checks and the Python harness checks are needed; those four jobs read the answer in a job-level if: and are skipped outright (no runner) when it is no, instead of starting a runner to post their name, as five jobs did on every pull request. e2e (playwright) now runs only when frontend/ changed. The verify matrix rows keep their in-job detect step, because a matrix row cannot carry an if: of its own. A final gate job needs every other job, runs even when some were skipped (if: ${{ !cancelled() }}), and fails when any result is failure or cancelled; it is the one CI check the ruleset will require, in place of ten single job names, and the ruleset's new list of five (payload, gh api line, read-back and rollback) is in the new required checks runbook; it takes effect when the ruleset is updated, and until then every old name is still a job name, so pull requests merge under either list. The separate Coverage workflow is gone: the frontend and functions (azure) rows run their suites once, with coverage (the thresholds in each vitest.config.js are the ratchet), and a coverage (qlty) job uploads both reports with GitHub OIDC after a merge to main only, since a pull request runs only the rows its files touch and a report with one package missing would read as a drop. The Complexity Delta workflow is a complexity delta job of CI, unchanged in what it does. CodeQL stays on pull requests until the ruleset no longer requires its three contexts; the merge-queue question is answered in the runbook (not yet, with the reasons). scripts/ci-workflow-steps.test.mjs runs the four moved detect steps with bash as before and holds gate to needing every job; scripts/ruleset-contexts.test.mjs holds the new list and the old one. ADR 0026 has the addendum.
  • The plan check forgets the #1469 step 2 delete its apply consumed (#1469). scripts/assert-expected-plan.mjs lists the one-off changes an HCP Terraform plan may carry, and its own rule is that an entry is deleted once its change has applied, so a later drift of the same thing is caught instead of waved through by a stale line. run-DvAMt2zE3XU4es4h applied the merge of #1544 on 2026-11-04 at about 13:55 UTC: the plan was the one declared destroy, azurerm_cosmosdb_sql_role_assignment.func_cosmos (the Function App's account-wide Cosmos Data Contributor grant), beside the standing permanent diff, and the identity now holds the database-scoped grant alone. That entry is gone and the file's running list of what is in state names it with its run. The test follows: the #1469 case now holds that neither the account-wide grant's block in infra/ nor a declaration of its delete exists, where before it held that the delete was declared while the block was gone.

Removed

  • The Code Conductor brain is taken back out of the site (#1479). The owner's direction of 2026-10-31: the site's Second Brain is for content, ideas and publications, and the engineering brain, the one that holds decision records, belongs to Code Conductor (Proj-Code_Conductor), which is its own product. So the second workspace that #1465 added this morning (the code-brain module and its six routes under /api/cms/code-brain/, the shared-workspace refactor of the creator brain made for it, the export and restore additions, the sync-client script change, the runbook and the contract entries) is reverted. That code did reach production, still switched off: the second attempt of Deploy Functions run 41195200540 shipped 0b3576a7, which contains dfac635, finishing at 2026-11-01 05:03 UTC (its first attempt, at 2026-10-31 11:02 UTC, was refused because a Terraform apply was running, and deployed nothing). It stayed live until run 41279867586 deployed 14b1f423, which carries this revert, from 06:12 UTC the same day. (An earlier version of this entry said it was never deployed; it was deployed, still switched off.) The creator Second Brain is exactly as it was before #1465 and its tests pass unchanged. ADR 0036 stays as a withdrawn record. The Terraform #1465 added goes too (the hcw_code_brain database and container, their variables, the five CODE_BRAIN_* Function App settings, the guard test and the plan-assertion entries), because the owner confirmed that the run for dfac635 was applied: the next confirmed apply in HCP Terraform destroys that empty database and container, and pressing confirm is the owner's step. The plan check (scripts/assert-expected-plan.mjs) declares those two deletes and the five setting removals, so tfc-plan-check.yml reads that run as expected. The restore script (scripts/restore-cosmos-export.mjs) now restores only into containers the current spec provisions: a retired container that an older export set still lists is named and skipped, never written into the creator database.

Features

  • Koda, the owner's content agent, gets an Agent Content API and a Koda tab on the AI Engine (#1477, ADR 0037; built from the owner's proposal of 2026-10-31). Koda runs in the Muse service and can only sign in with one fixed key, so the site now issues one: made on the new Koda tab (/admin/ai-engine?tab=koda), shown once for the owner to paste into Muse's secure field, and kept only as a SHA-256 hash in one admin_config document. The routes under /api/agent-content/v1/ (status, the Drafts routes by id, and the published reads) are the Drafts stage's own handlers created with the key guard in the role guard's place, so a draft Koda writes gets the same field allow-list, 256 KB cap, etag protection, title dedup, refusals for anything live and audit rows as one typed on the Drafts page, with koda (agent) as its author and draftOrigin: 'agent'. The key is checked in constant time with no cache, limited to 120 calls a minute with the public routes' compare-and-increment counter, refused everywhere under /api/cms/, and every refusal is an audit row without the key. Five switches on the tab (read drafts, create and edit, send to review and back, delete, read published) narrow what the key may do; there is no publish, unpublish, slug, settings or secrets route for it to call. Rotate and Revoke cut the old key on its next request. The base path is agent-content, not the proposal's agent, because /api/agent/* is the lab agent's. The runbook is Agent Content API: connect Koda, rotate and revoke; the open point for the owner is the host Muse registers (api-azure.hybridcloudworks.com, where the API answers).

  • Blogs hub and editable source intake (#1443). Blogs now sits directly below Coder Corner with article review/edit links, source search/provider filters, editable RSS and listing sources, pause/resume, bounded intake limits, per-source manual jobs and run status, and gallery/URL fallback images. Adds Kiro’s official RSS feed to AWS. Both scheduled intake paths honor runtime settings; stable cache identities, conflict detection, audited editor-only saves, untrusted custom sources, and guarded RSS downloads preserve the existing publishing workflow. Inspection prefers article/social images over source defaults; listing links now resolve relative to their page path, and listing ingestion supplies the article-ID generator missing from production wiring. See the blog workflow.

  • The visitor session broker's Coder token, CODER_BROKER_TOKEN, is now declared in Terraform as a Key Vault reference to the secret CODER-BROKER-TOKEN, the way every other credential the Function App reads is (the declaration half of arming the sweep timer; follow-up to item 1 of #1403) (pull request). The code had read CODER_BROKER_TOKEN since the visitor sessions were built, but nothing in infra/ declared it, so a copy set by hand would have been removed by the next apply, and switching the sweep timer on would have done nothing useful. infra/functionapp.tf gains the reference. functions/src/lib/secret-catalog.js lists it, so the Integrations → Keys page offers it in the Labs group, and functions/src/lib/credentials/register.js registers it as a hand-rotated credential with no rotation rule yet. scripts/check-unresolved-secrets.mjs lists it as expected-unresolved until the owner seeds it (the cap in its test moves from 5 to 6), and scripts/assert-expected-plan.mjs declares the new app setting so tfc-plan-check reads the plan as DECLARED. While the secret does not exist the app behaves as it did before: the broker reads the token as not set, so a session cannot start (the visitor routes answer VISITOR_NOT_CONFIGURED once LABS_VISITOR_SESSIONS_ENABLED is on and Turnstile is seeded), and the timer logs one Warning and calls nothing; applying the reference first is safe because Terraform writes only the reference and never reads the vault. New tests tie the setting names the code reads (CODER_BROKER_TOKEN_SETTING, TURNSTILE_SECRET_SETTING) to the references in infra/ and fail when the broker's reference is removed. Documentation: Required inputs §4.1 (the enabled_timers row now says plainly that the timer keeps cleaning up existing sessions while LABS_VISITOR_SESSIONS_ENABLED is off, the wording point CodeRabbit raised on #1412), §4.5 and §4.6; Variables and secrets (32 references); the lab host runbook ("Arming it" step 1 and the new "Seeding the broker token", with how to take it away again); ADR 0032's amendment; the cold rebuild list; the rotation schedule and the compromise tiers. Still open in #1403 and not done here: which Coder user and permissions the token is made with (item 4), and LABS_VISITOR_SESSIONS_ENABLED, which infra/ still does not declare. Nothing was applied or run against Azure, HCP Terraform or Coder.

  • A timer, sweepVisitorSessions, runs the visitor lab session sweep every two minutes on its own, so a Coder clean-up that Coder has not confirmed no longer waits for the next visitor (item 1 of #1403; the owner answered timer) (pull request). reapExpiredVisitorSessions ends visitor lab sessions past their hour and retries the clean-ups of the workspace and user that Coder has not yet confirmed (#1388). It ran only when a visitor read the lab's status or started a session, so with no visitors, or with LABS_VISITOR_SESSIONS_ENABLED off (the status route answers before it sweeps), a waiting session, and the Coder user and host workspace slot it held, waited. The new timer (functions/src/functions/visitor-session-sweep.js, one run in functions/src/lib/timers/visitor-session-sweep.js) calls the same sweep with the real store, fetch and readBrokerConfig. Why every two minutes: a refused clean-up is tried again after 2, 4, 6 and 8 minutes, a dead claim is taken over after 5 and a session lasts 60, so two minutes puts each retry within one tick of being due (five minutes would turn the 2-minute retry into a 5-minute one; one minute would double the runs, because a visitor's own request already runs the sweep at most once a minute). A run with nothing to do is three queries on lab_jobs and no call to Coder, about 2,160 queries a day; request-unit cost was not measured against the live account. It is bounded: at most 10 sessions a run (MAX_SWEEP_SESSIONS) and a 10-second limit on each Coder call. It never creates a session or a Coder user: it is given only the query, read and conditional-replace operations, a test watches every operation that would create or overwrite a document, and the only thing it asks Coder to build is a workspace delete. It runs whatever LABS_VISITOR_SESSIONS_ENABLED says, because that switch closes the door to new visitors and turning it off while sessions exist must not strand their Coder users and workspaces. It does nothing at all, not even a query, when Coder is not set up (CODER_URL and CODER_BROKER_TOKEN; readBrokerConfig no longer falls back to the read-only status token, which cannot delete anything): the sweep would otherwise spend one of a waiting session's five tries on every tick, and five tries end a session as cleanup_failed. It logs counts and kinds only (reaped, finished, pending, failed; the kind of an error; never a session, a Coder id, or any text Coder or Cosmos sent), as a Warning when the run did something and as Information otherwise (host.json ships only Warning), and it never throws out of the handler: a failed run is one Warning line and the next tick tries again. Because a run that cannot sweep still counts as a success to alert-timer-overdue, the runbook says to look for that Warning. The trade-off: before the timer, tries were spent only when a visitor arrived; now they are spent on the clock (at about 0, 2, 6, 12 and 20 minutes, a retry slipping a tick when a run starts late), so a Coder or token outage longer than about twenty to thirty minutes leaves the waiting sessions as cleanup_failed, for a person to finish by hand. No alert reads those lines yet; that is in #1403. Two runs at once cannot double-call Coder: the Functions host runs one copy of a timer across all instances and does not start it while a run is still going (Microsoft Learn), and, without relying on that, the sweep claims a session with a conditional write before it calls Coder; tests run two timer runs, and a timer run with a visitor's sweep, at the same moment and count the Coder calls. One limit: a claim lasts 5 minutes from the start of a sweep, and ten sessions whose Coder calls all run to their 10-second limit can use all of it, so a claim taken over then can lead to a second, harmless delete (Coder answers 404 or 410 for what is gone) that spends one try; the margin is an open item in #1403. The sweep in the visitors' requests stays, with its once-a-minute limit, as the only sweep while the timer is off. Hardened in the same change, because the timer runs unattended with the broker's token: readBrokerConfig no longer falls back to the read-only status token (every create and delete with it would be refused, and each refused clean-up try is spent), so without CODER_BROKER_TOKEN the broker is not configured; an id or name saved in a session record is written into a Coder URL only when it is plainly an id (not .., nothing with a slash, ?, # or %, and not me, which Coder reads as the caller's own account); a record's own endStatus and try count are read only when they are one of the two final statuses and a whole number; and a log line names an error only by its status, system code or class. Infrastructure (plan only, nothing applied from here): the flag SWEEP_VISITOR_SESSIONS joins local.timer_catalogue (infra/functionapp.tf) and the enabled_timers allowlist (infra/variables.tf), so the Function App gains FEATURE_FLAG_SWEEP_VISITOR_SESSIONS = "false", off until the owner adds the name to enabled_timers; sweepVisitorSessions joins local.timer_schedules (infra/observability.tf), so alert-timer-overdue watches it; and scripts/assert-expected-plan.mjs declares both changes for the plan check, in each state the alert's timer list can be in. Registered in functions/src/functions/index.js, function-inventory.json, the registration-shim list and the route inventory (now 29 timers). The runbook (docs/runbooks/labs-host.md, "The sweep timer") says what to check, how to arm it, and how to turn it off or roll it back; ADR 0032 (a dated amendment), .azure/api-surface.json and docs/standards/required-inputs.md say it too. docs/standards/required-inputs.md also records that CODER_BROKER_TOKEN is declared nowhere in infra/ (MISSING), so a copy set by hand would be removed by the next apply: declaring it is its own change, tracked in #1403. Tests fail on the tree without the timer (6 of the 7 timer test files), on the lab sweep code as it was before the hardening (25 tests), and on 38 hand-made changes, all caught, among them: the timer not registered, the gate inverted, the schedule changed, an error let out of the handler, createDoc handed to the sweep, the master switch ignored, the sweep waiting for the visitor-session switch, a run that logs the broker token or the error text, a run that sweeps with no Coder settings or with only the status token, a path that takes .. or me, and each of the infrastructure and plan-check edits left out. What is still open in #1403 (items 2 to 9, and the new items found while building this: a time-based try budget, an alert on the Warning lines, the lease measured from the start of a sweep) stays there. Not live until the owner has merged, applied the Terraform run, deployed the functions, declared CODER_BROKER_TOKEN in Terraform (it is not today) and added SWEEP_VISITOR_SESSIONS to enabled_timers; nothing here was applied or run against live Azure, Coder or Cosmos.
  • Cloud Crawler's catalogue row is available, so /tools/cloud-crawler frames the tool once the site is deployed (the second of two pull requests for #1357; the first was #1411). The host has run the v0.2.0 image since the owner's playbook run of 2026-10-26 (the playbook is the script that sets up the lab host), and the site's status address then answered configured: true, reachable: true. The row lets the tool's pane (the framed window where it runs) save files (downloads) and copy its report (clipboard-write), and does not let it ask the page to move (navigate); the pane's code, read on 2026-10-26, never asks to (#1354 is the rule). The tool's address joins the site's frame policy (frame-src) and the clipboard and full-screen lists of Permissions-Policy in frontend/staticwebapp.config.json, as for the other three tools. The row's wording drops "or your Terraform" and the terraform technology entry, because the engine the lab runs cannot read Terraform yet; put both back when the add-on ships an engine that does (tracked in HCW-CloudCrawler_AddOn#29; HCW-CloudCrawler_App#60 is merged, but the App has no release after v0.1.0). Smaller fixes: a test title and an API-surface note that still said the setting was not provisioned; the scripts (operations) CI job now also runs when functions/src/lib/addons/registry.js changes, which scripts/assert-expected-plan.test.mjs reads; and the plan check's entry for ADDON_CLOUD_CRAWLER_URL is deleted now that the setting is applied. The pair page, the program README, the root README, the labs host record, the Scout runbook and the integration standard now say four tools, TODO.md loses its "one AddOn row is coming" row, and the labs host runbook gains "Cloud Crawler: the first real upload". The add-on's integration guide is updated in its own repository (HCW-CloudCrawler_AddOn#30) so its copy of the row equals this one field by field. Not verified by any session: the tool inside the site's frame in a real browser, and one real upload with the human check. lab-host/ is untouched on purpose (a merge there makes the site ask for a playbook run).
  • The dossier's wording check (OSS-9) now reads every Markdown file, NOTICE, .env.example and the issue forms of a repository, not only README.md and CLAUDE.md (#1405, for HCW-NetworkEyes_App#318, item F-2). docs/standards/oss-docker-app-dossier.md has a new one-line PowerShell command that takes the newest commit on main, lists that commit's files through GitHub's tree call, reads each of those documents at the same commit and prints a line for each hit of "customer", "client", "pricing" or "edition". It stops with an error instead of printing a partial list if a file cannot be read or GitHub cut the file list off (CodeRabbit's review of the pull request). The rule now says what to keep (technical terms, dated history, a quote of another party), to write down the commit id the command printed, and when OSS-9 is met. The template line and the Network Eyes worked-example row are updated to match. The command was run, copied out of the page, in PowerShell 7.6.7 and Windows PowerShell 5.1 against Network Eyes on 2026-10-26 (233 hits in 14 files on main at 19759c6a). The row says "part" until HCW-NetworkEyes_App#320, which rewords the documents and adds a guard test, merges.
  • The lab host is ready to run Cloud Crawler and the site's backend is ready to reach it; the catalogue row stays coming (first of two pull requests for #1357; this one is #1411). lab-host/ansible/group_vars/all.yml gains the cloud-crawler row: docker.io/hybridcloudworks/img-cc-lab-addon v0.2.0, the first image with the upload route, pinned by its index digest sha256:5a43c90bda9bed1d3472847f492b7c05405e6a6ce2d4aa7f4d9b519bb72daf28, on loopback port 18084, 512 MiB, 256 processes, the add-on's CC_ environment (the same frame-ancestors, site-origins, proxy and limit settings as the other three rows, the widget's public site key, and CC_MAX_STORE_MB at its documented 64), and TURNSTILE_SECRET from vault_addon_cloud_crawler_turnstile_secret. The digest was read on 2026-10-26 from Docker Hub's public tag API and from docker buildx imagetools inspect, which agreed; gh attestation verify against HCW-CloudCrawler_AddOn exited 0 and the signing certificate names publish-images.yml at refs/tags/v0.2.0 (run 40318349978), which the row's attestation_identity_regexp matches. infra/functionapp.tf declares ADDON_CLOUD_CRAWLER_URL, and scripts/assert-expected-plan.mjs declares it as exactly one added app setting, as #1089 did for the other two; three new tests in scripts/assert-expected-plan.test.mjs hold the declaration to the Terraform value, to a plan that adds that one setting and nothing else, and every ADDON_*_URL name in infra/ to a name the Function App's registry reads. The vault-key table in lab-host/README.md and the runbook's list of keys set by hand now include the key (seven become eight; the vault holds sixteen keys, not fifteen), and the table gains the two earlier add-on keys it had omitted. The runbook's new "Cloud Crawler: the host steps" gives the owner the Terraform confirmation, a names-only vault check, the playbook run and the status check, in order, with what to see at each. The pair page, the app-to-addon README, TODO.md, the labs-host architecture page and the scan runbooks' add-on counts (three become four) follow. Not changed, and unverified: the catalogue row, frame-src and the add-on's integration guide (the second pull request, after the status route answers configured: true, reachable: true); whether the vault key is on the host (the owner states it is; no session can read the host's vault); whether the container starts and the certificate answers for the name (the host has not run the row; the wildcard name resolves, read on 2026-10-26); and that this host has memory for a fourth 512 MiB container beside five workspaces and Coder (ADR 0035 notes the ceiling).
  • Second Brain is implemented inside HCW admin (#1395). The owner-only workspace connects private Markdown notes, revisions, topics/projects, creator references, search and a keyboard-accessible map. Existing editors can capture research and create reviewed drafts with private provenance. An on-demand Foundry assistant cites workspace sources. The Windows Obsidian companion uses a dedicated brain_sync client, encrypted token storage and recoverable conflicts. Private records join authored backups; indexes can be rebuilt. Deployment starts with both feature flags off, with owner configuration and real-vault enrollment governed by the rollout runbook.
  • The website now counts the lab sessions it starts for visitors, and the Labs hub Dashboard has a card that shows the count (editors and admins only; nothing public). Each time the visitor session broker creates a new session (functions/src/lib/labs/visitor-session.js), it adds one to a per-lab, per-UTC-day tally (functions/src/lib/labs/lab-starts.js): one lab-starts:YYYY-MM-DD document { day, total, byLab } and one lab-starts:meta marker { since } in the system container, each written only if nobody else changed it since it was read (an ETag check), so two overlapping starts do not lose a count. The counting documents hold nothing that identifies a person (no IP address, client key, session id, user id, token, query string or time of day), and lab-starts.test.js fails if a stored document gains any other key. A repeat request that returns a live session, a page view, a refused request and a failed provision are not counted; a counter fault is logged by code only and never reaches the visitor. New editor route GET cms/labs/starts (cmsLabsStarts) returns today, the last 7 days, the last 30 days, the total and a per-lab breakdown; the card shows them with the day counting began, says in plain words that it counts sessions and not people, and has Copy as text for a letter, which writes one plain sentence with the real numbers and the date range (no vendor names). Counting is on from the deploy: the visitor session records are deleted about two hours after their last update, so there is no earlier history and none is estimated. Signed-in member starts inside the lab pane and the public validation runs are not visitor sessions and are not counted. The broker's lab check now uses an own-property test, so ids such as constructor or toString are refused instead of passing. docs/runbooks/labs-host.md has the new "Counting lab starts" section. Not live until the functions and the frontend are deployed.
  • A standard and a fill-in dossier say what an open-source Docker app from Hybrid Cloud Works must look like to qualify for Docker's free open-source programme (DSOS), with Network Eyes as the worked example (ticket #678). docs/standards/oss-docker-app-dossier.md quotes Docker's four rules from its programme and application pages (read 2026-10-25) and says what the pages leave out: whether an organisation that already pays for a Docker plan can be accepted, which the page tells the reader to ask Docker in writing. It lists thirteen checks (OSS-1 to OSS-13: public source and images, an OSI-approved licence with the same SPDX id in LICENSE, the Dockerfile label, NOTICE and the README, the copyright holder, OIDC-only publishing with provenance, an SBOM and a Scout check, the img-<code>-<lab-addon|appliance-app> name, no paid edition and no customer or client wording, active development, community files, the two rulesets, and Dependabot, CodeQL and dependency review), each with a PowerShell command and the result to expect, a template block to fill for the next product, and a short paragraph on moving a product to its own Docker organisation if it ever gets a paid edition. The Network Eyes example uses values read from the repository, Docker Hub and the registry on 2026-10-25 and 26. It records what HCW-NetworkEyes_App#316 fixed while the page was being written (the copyright holder in LICENSE and NOTICE, and the "client" and "customer" wording) and the gaps that remain on main: "lab edition" wording in the README and CLAUDE.md, no licence label on the lab add-on image, a stored Docker secret that no workflow uses, and a tag ruleset that lists the repository-admin role as a bypass actor. The page is registered in mkdocs.yml and linked from section 2.11 of the field application standard. No guard script was added, because this repository has no existing test that checks standards pages. Nothing was sent to Docker.
  • A test now fails any pull request that puts a draft article back into docs/content (PR 4 of the move of pre-publish drafts out of the repository; owner rule 2026-10-25, follows #1381). functions/test/docs-content-guard.test.js lists docs/content (sub-folders and letter case included) and fails if any blog-*.md file other than blog-template.md and blog-machine.md is there; its failure message explains in plain words that the repository and the docs site are public and that drafts belong in the owner's private vault and on the admin Drafts page. The functions (azure) row of ci.yml now watches all of docs/content/ instead of the two contract files, so the guard also runs on a pull request that only adds such a file. .claude/CLAUDE.md gains "Pre-publish content never goes in this public repository", and the "why the notes stay outside" paragraph of docs/runbooks/obsidian-idea-notes.md names the guard. Checked by adding a docs/content/blog-test.md: the test failed with that message, then passed once the file was removed.
  • The 17 pre-publish drafts are removed from the public repository and the docs site, together with the "Import from docs/content" route that read them (PR 3 of the move of pre-publish drafts out of the repository; owner rule 2026-10-25, follows #1379 and the promote-script change). Every docs/content/blog-*.md draft except the two contracts (blog-template.md, blog-machine.md) is deleted, as are their mkdocs.yml navigation lines and five matching wiki-redirect entries; each file was first compared byte for byte with its copy in the owner's private vault. Four of them (blog-how-to-01, blog-how-to-02, blog-build-log-01, blog-build-log-02) were linked from the docs front page's "Start here"; those links are gone and the section now points only at the naming convention, so they can be restored from the vault if the owner wants. POST /api/cms/drafts/import-repo, its handler, drafts-import.js, the admin "Import from docs/content" button with its hook and result list, the client timeout entry, the .azure/api-surface.json row, the cmsDraftsImportRepo entry in functions/function-inventory.json and the narrowed ci.yml path filter go with it; route-inventory.test.js now asserts the route is absent. repo-draft.js (the parser behind the markdown upload) and the "From docs/content" label for rows already in the database stay; repo-draft-source.js stays as the pinned GitHub fetch that the lab-host drift check and the delivery monitor use, without its draft listing and file reads. Tests that read real draft files now read synthetic fixtures in functions/src/lib/cms/__fixtures__/, written to the blog template, and the test that pinned "seventeen articles" is deleted. docs/content/blog-machine.md and docs/runbooks/obsidian-idea-notes.md no longer tell anyone to import from docs/content. After this deploys, the docs page /content/blog-lab-01-landing-zone/ returns 404 and POST /api/cms/drafts/import-repo returns 404 instead of 401.
  • The idea-note promote script now saves a draft into the owner's private vault, never into the public repository (PR 2 of the move of pre-publish drafts out of the repository; owner rule 2026-10-25, follows #1379). scripts/ideas/promote-idea.mjs wrote docs/content/blog-<name>.md into the working tree; it now writes <vault>/Drafts/blog-<name>.md, where the vault comes from --vault <folder> or the HCW_VAULT environment variable, and stops with a plain-language error (exit 2) when neither is given or the vault folder does not exist. --out-dir is removed. It refuses (exit 1, nothing written) any output folder that resolves inside the repository, whether given as a relative path, with .., in a different case on Windows or macOS, or through a symlink or junction (links are followed with the real path, and the comparison runs again after the folder is made). The file format is unchanged, and functions/src/lib/cms/idea-promotion.test.js still reads the output with parseRepoDraft and now also with parseMarkdownDraft, the reader behind the Admin → Drafts upload. New tests cover a path inside the repository, relative paths, .. tricks, a junction and symlink into the repository, a link that points elsewhere, a missing vault and HCW_VAULT, and assert that no test run leaves a file under docs/content (they fail against the old script). docs/runbooks/obsidian-idea-notes.md is rewritten so the steps are promote into the vault Drafts folder, then upload on Admin → Drafts, with every instruction that sent a draft through a pull request removed and the reason explained; the Drafts stage section of docs/content/blog-machine.md now says the Drafts page is the source of truth and the import-from-repository route is being retired. Not removed here (a separate change): the 17 existing docs/content drafts, their mkdocs entries and the import route.
  • Admin → Drafts can take a whole markdown file, or several at once, straight from the owner's private vault, so a pre-publish draft no longer has to be committed to the public repository (PR 1 of the move; owner decision 2026-10-25). POST cms/drafts/from-markdown takes { markdown, origin } (origin is vault or paste) at the editor role, reads the text with the docs/content import's own parser (parseRepoDraft, which needs no repository path, so the import's path checks are untouched), checks it with the rules a typed draft goes through (validateDraftFields, 256 KB body cap, 272 KB for the whole file, the request body counted as it is read and abandoned past 545 KB, with or without a Content-Length) and writes a drafting, never-public draft with draftOrigin vault or paste (functions/src/lib/cms/drafts-markdown.js, drafts-handlers.js). The document id comes from the normalised title, so a second upload of the same article, even a double click, answers ALREADY_IMPORTED (with the stage it is at) instead of duplicating, and a title a published article already carries is skipped; a file that is not a draft is a 422 with the parser's reason. The markdown is never logged and never put in an audit row. The Drafts page gains a Paste or upload a markdown draft panel: a picker for many .md files sent one request at a time with a result line per file (created, already present, or the reason it was not added; one bad file does not stop the rest) and a closing total, plus a box for one pasted draft; the list labels drafts From my vault and Pasted in. Registered in the route inventory, function-inventory.json, .azure/api-surface.json and the client timeout table. The docs/content import is unchanged and stays until a later change retires it. docs/runbooks/obsidian-idea-notes.md and the Drafts stage section of docs/content/blog-machine.md describe the new path. Not yet live: the owner deploys the functions and the frontend, and the admin sign-in path is not exercised by tests.
  • The owner's standing orders for sessions are written into the project instructions: the session runs the work and is the owner's last gate, not the first. .claude/CLAUDE.md gains "You run the work; the owner is the last gate, not the first" (do every ticket you can, a missing permission is asked for and never a reason to hand work back, GitHub administration is the session's, use the agents, the board is the list of work, naming conventions, and the link hook). Written after the owner repeated these in one day (2026-10-25).
  • An add-on's integration guide is checked against the website's catalogue by a reusable workflow, so a guide that has drifted fails a job instead of waiting for a review (website half of #1356). scripts/addons/check-guide-drift.mjs (Node built-ins only, tested by scripts/addons/check-guide-drift.test.mjs) reads the add-on's row from frontend/src/data/addons/catalogue.js, the row pasted in the guide's "Catalogue row" block (evaluated in an empty node:vm context) and, when the guide states one, the port from lab-host/ansible/group_vars/all.yml, and fails naming each of twelve fields (and the port) that differ with both values; it also fails when the site no longer lists the id or the guide has no readable row. .github/workflows/addon-guide-check.yml is the workflow_call wrapper: permissions: {} with contents: read on the job, no secret, every action pinned to a commit, the script fetched at the workflow's own commit (job.workflow_sha) and the data from main, the result in the job summary; scripts/addon-guide-check-workflow.test.mjs holds it to that. docs/runbooks/addon-guide-drift.md has the caller each add-on repository adds and how to read a failure. The add-on repositories' callers are still to be added, so the issue stays open.
  • One shared standard for the four field application repositories is written down, with a conformance table and the gaps as proposed tickets (PF-2, HCW-NetworkEyes_App#255). docs/standards/hcw-field-application-standard.md lists what NetworkEyes, CloudCrawler, CloudAssessor and MigrateOrchestrator must, should and may do (identity, documentation, community files, labels and the board, rulesets, the self-merging pull request action, CI, Dependabot, security scanning, releases, container images, secrets, agent instructions), with the evidence for each row read from the default branches, the settings API and a repository-admin dry run on 2026-10-25, a per-repository conformance table, seven recorded exceptions and sixteen proposed tickets. Findings that need the owner soon: NetworkEyes' repository-admin workflow lists 7 required checks while the live ruleset has 10, so applying protect-main would drop three; NetworkEyes' CodeQL default setup scans only the workflows, not its Python or TypeScript; and no repository has the P0 or severity:critical labels the board offers.
  • The Scout runbook records that the copyleft policy is left as it is (#1367). docs/runbooks/docker-scout.md said loosening it was still open; the owner decided on 2026-10-25 to leave it, and the page now says so with a link to the decision.
  • Two standing rules for sessions: every message links what it names, and the specialist agents are used for review and audit work. .claude/CLAUDE.md gains "Everything named to the owner carries its link" (each message links every issue, setting, decision or file it names, in the same sentence, with a check before sending; owner instruction of 2026-10-25) and "Use the specialist agents" (which agent for which kind of review, how to brief one, and to say in the closing summary which was used). The wording was reviewed by the documentation archivist agent before this change; its nine wording findings were applied.
  • The lab host's migration add-on moves to v0.3.2 (digest sha256:bd3e0a16…), and only the new repository name is accepted for its attestation (#1367). The add-on's v0.3.2 release was published to Docker Hub on 2026-10-25 and gh attestation verify against HCW-MigrateOrchestrator_AddOn passed (signer: its publish-images.yml at refs/tags/v0.3.2). lab-host/ansible/group_vars/all.yml now pins that digest and the attestation pattern no longer accepts the old repository name, as its own comment said to do once the pin moved; scripts/lab-host-addons-attestation.test.mjs now asserts the old name is refused (shown failing against the old pattern). Not yet live: the owner runs the lab host bootstrap after the merge.
  • How the nine repositories' dependencies are kept under watch is written down (#1367). docs/runbooks/dependency-review.md records, per repository, which tools actually run (read from each repository's workflow files on 2026-10-25, not guessed from file names), a one-line weekly check of open Dependabot and code-scanning alerts, a proposed rule for how fast to act on each severity, and the open findings, including the Network Eyes app's missing review gates and the Migrate Orchestrator app's 17 code-scanning alerts, which belong to the apps' workstream. The Scout runbook no longer tells the reader to widen the token on an entitlement error: it checks the Docker ID and token first.
  • Docker Scout runs in CI, with one stored Docker token (#1367). The owner decided on 2026-10-25 to store a Docker credential for it after both token-free ways failed (docker/scout-action and the Scout command line each needed a signed-in Docker user). .github/workflows/addon-image-scout.yml reads the three add-on digests the lab host runs (scripts/lab/read-addon-pins.mjs) and, for each, prints Scout's policy table and fails on a CRITICAL or HIGH vulnerability that has a fix, when a pull request changes the pins and every Wednesday. The credential is the repository variable DOCKER_SCOUT_USER and the secret DOCKER_SCOUT_PAT (a personal access token with the lowest permission, Repo Public Read-only), read in two places only; every job holds contents: read; a fork's pull request is skipped, not handed the token; a missing token is a warning on a pull request and a red run on the weekly one, never a quiet pass. scripts/addon-image-scout-workflow.test.mjs pins each of those; seven deliberate breaks of the workflow were each caught, and the file was restored byte for byte. docs/runbooks/docker-scout.md has the steps to create, store and rotate the token. The weekly Trivy scan stays as the check that needs no credential and opens the issue.
  • The weekly vulnerability scan now covers the three add-on images, and the way to use Docker Scout is written down (#1367). lab-supply-chain.yml scanned only the two lab images with Trivy; its scan step now also reads the digests the lab host runs from addons[] in lab-host/ansible/group_vars/all.yml (scripts/lab/read-addon-pins.mjs, tested by scripts/read-addon-pins.test.mjs, and scripts/lab-pin-pr-workflows.test.mjs pins that the scan includes them and refuses an empty read) and scans those too, opening the same weekly issue when a HIGH or CRITICAL vulnerability has a fix. No new workflow, no third copy of the Trivy pin, no stored credential. Scout itself is not in CI: both docker/scout-action v1.27.0 and the Scout command line, tried on 2026-10-25, stopped ("user githubactions not entitled to use Docker Scout" and "Log in with your Docker ID"), because Scout only answers a signed-in Docker user, and storing a Docker token for it is the owner's decision. An earlier draft of this change claimed Scout needed no login; that was wrong (it had run on a desktop where Docker Desktop is signed in) and is corrected. docs/runbooks/docker-scout.md says what the plan includes (Docker Team, unlimited Scout-enabled repositories), the four checks and which is for what, why Scout is not in CI and what adding it would take, how to turn Scout on, the desktop commands, how to read the result, and the Docker Hub housekeeping facts. The decisions that remain are on #1367.
  • The labs host's Applied state records the run of 2026-10-25 (#1355). Read over SSH: /etc/hcw/applied-commit.json says the host converged at 2026-10-25T04:44:25Z from d2b207b4, which was main's head, and docker ps shows the migration, network assessment and cloud assessment add-ons Up (healthy) on the img-mo-lab-addon, img-ne-lab-addon and img-ca-lab-addon images. The ADR 0035 row and the Configuration row of docs/architecture/labs-host.md now say so, with how each fact was read, in place of "not recorded" and the stale 2026-10-14 commit.
  • The three tool add-ons' images follow the owner's naming convention img-<short name>-<job>-<type>. lab-host/ansible/group_vars/all.yml now pulls docker.io/hybridcloudworks/img-mo-lab-addon (migration), img-ne-lab-addon (network assessment) and img-ca-lab-addon (cloud assessment), and the tag is the full release tag (v0.3.1, v0.1.0, v0.1.0). The digests are unchanged: the owner copied each image and its signed build attestation to the new repository on 2026-10-25, and the registry shows the same index digest and the attestation on every new name. The container names (hcw-addon-<id>) are not changed. The old hcw-addon-* repositories stay until the host has run from the new names. The runbooks, the host README and the app-to-addon pages follow.
  • The migration add-on's repository is now HCW-MigrateOrchestrator_AddOn, and this repository's living references follow. The owner renamed it on 2026-10-24 from HCW-AzMigrateOrchestrator_Addon (its App repository had already become HCW-MigrateOrchestrator_App). The three links in the catalogue, the integrations directory and the admin service registry, the add-on pair pages, the integration standard, the lab-host comments and the bootloader header now use the new names; GitHub redirects the old ones. The architecture decision records and docs/architecture/app-to-addon/final-package.md keep the names that were true when they were written. The lab host's attestation pattern for the migration image accepts both names, from #1362, until the pin moves to an image built under the new name.
  • The migration add-on's image attestation pattern accepts both repository names, ahead of its rename. The owner is renaming HCW-AzMigrateOrchestrator_Addon to HCW-MigrateOrchestrator_AddOn (the App repository is already HCW-MigrateOrchestrator_App). The lab host verifies each add-on image's build attestation against a pattern that names the building repository, and an attestation cannot be edited afterwards: the digest pinned today was built under the old name, and the first image built after the rename will carry the new one. lab-host/ansible/group_vars/all.yml now accepts HCW-(AzMigrateOrchestrator_Addon|MigrateOrchestrator_AddOn) for the same workflow and tag refs only, and scripts/lab-host-addons-attestation.test.mjs pins that a branch ref or a lookalike name (...-fork) is still refused. The old name comes out of the pattern once the pin has moved to an image attested under the new one.
  • The notes DNS record applied (#1350), so the plan checker forgets it. #1360's create applied on 2026-10-24 (run-bb1XtRGJEbuK1Cec; tfc-plan-check read it as applied, carrying the known permanent diff and nothing else). notes.hybridcloudworks.com then resolved to Cloudflare addresses on 1.1.1.1 and answered HTTPS with Server: cloudflare and a CF-RAY, status 404 because the name is not yet claimed in Obsidian; there was no redirect loop and no 525 or 526, so the zone's SSL/TLS mode suits the record. One correction to #1360's description: it expected the plan to read 1 to add, 0 to change, 0 to destroy, but the run's own counts were 4/1/3, which is the permanent diff (the counts are not the check, the addresses are) plus that one create.
  • notes.hybridcloudworks.com, prepared for Obsidian Publish (#1350). infra/frontend.tf declares cloudflare_dns_record.notes_obsidian_publish: a proxied CNAME from notes to publish-main.obsidian.md, which is what Obsidian's Custom domains page asks for (read 2026-10-24): proxy on, and the zone's SSL/TLS mode "Full". It is the zone's second proxied hostname after api-azure, because Cloudflare, not Obsidian, presents the certificate for the name. The plan checker declares the one create, and docs/runbooks/edge-dns-verification.md gains the row, the count (seven records) and what a redirect loop or a 525/526 means. terraform fmt -check and terraform validate pass and the checker's 66 tests pass; nothing is applied. The owner confirms the HCP Terraform run of hcw-azure (expected plan: 1 to add, 0 to change, 0 to destroy), then claims the name in Obsidian.
  • Add-on panes get the site's theme, and the network and cloud assessments are granted what they asked for (#1353, #1354). Owner decisions 2026-10-24, from the owner's phone. Theme: the frame now loads <origin>/?theme=dark or ?theme=light (addonFrameUrl), read from the dark class on the document when the page first renders. It is read once and never updated, because changing a frame's address reloads it and would drop the report open in the pane; a toggle reaches an open pane at the next page load. Checked in a real browser, framed by the live site on 2026-10-24: the network assessment and the cloud assessment follow the parameter (dark turns the page dark, light keeps it light, with the browser itself set to light); the migration add-on ignores it and stays white, which is an AddOn-side gap. Capabilities: the owner said yes to the guides' request. The rows now hold downloads and a new clipboard-write capability, replacing navigate and downloads. The site's clipboard capability grants read and write (allowFor), which the issue text had understated, and the AddOns only need write for "Copy report", so a write-only capability was added instead of granting clipboard; navigate goes because neither add-on sends one. The standard (sections 8 and the helper list), the two pair pages and the accepted-risk row in TODO.md say so. Seven tests were written or changed; with the old code put back they fail, with the new code all 76 pass. Not checked: the live panes after the next deploy, and whether "Copy report" works inside the sandboxed frame.
  • Cloud Crawler joins the add-on suite as a coming row, and the suite's documents are brought in line with the live site. Owner request 2026-10-24 ("introduce the rest of the add-ons … the site needs to know what backend integrations"). The fourth pair, HCW-CloudCrawler_App and HCW-CloudCrawler_AddOn, gets a catalogue row (cloud-crawler, coming since 2026-10-24), a server registry entry (ADDON_CLOUD_CRAWLER_URL, unset, so the status route answers { configured: false } with no read and no network, proven by a new test), the route /tools/cloud-crawler and its explainer page, and a pair page, docs/architecture/app-to-addon/cloud-crawler.md. It is not in the Tools menu, has no CSP entry, no app setting, no group_vars row and no port: the App has not tagged a release and the AddOn is a skeleton with no image (read 2026-10-24). Public copy names nothing behind the site. Checked on a production build in real Chromium, dark and light: the page shows the title, a "Coming soon" badge, the summary and the reason, frames nothing, has no accessibility violation (axe, whole document), and no link to it appears on /azure. The documents that had drifted since the network and cloud assessments went available on 2026-10-12 (#1089) are corrected: the program's index now lists four pairs with their real status and adds a table of the backend each add-on is configured to use (Coder for the network assessment; human verification for all; nothing else), read from group_vars; the network and cloud pair pages are rewritten as available; migration.md no longer quotes an image digest, which had gone stale (digests are written down only in group_vars); labs-host.md no longer calls ports 18082 and 18083 reserved; the README.md no longer says the network and cloud assessments are "on the way"; and the TODO.md accepted-risk row about coming rows and [VERIFY] values is replaced by one about Cloud Crawler. Not changed, and unverified: labs-host.md's "Applied state" table still says the add-on role is not yet on the host, while the live status route shows all three add-ons reachable; only the host's own record (/etc/hcw/applied-commit.json) settles it.
  • Idea notes: an Obsidian second brain for content, with a format, a check and a promote-to-draft script (#1343). Owner request 2026-10-24. The vault lives on the owner's computer, outside this public repository; the scripts take its folder as an argument and only ever read it. An idea note is a note whose properties are type, theme, format and status (and optionally title, subtitle, tags, reading, part, date). scripts/ideas/check-idea-notes.mjs <folder> reports what is wrong with each note and a count by status and type. scripts/ideas/promote-idea.mjs <note> turns an outlined blog idea whose format is how-to or build-log into docs/content/blog-<name>.md, in the shape the admin's Import from docs/content takes; it never overwrites, never edits the note, and refuses what can only be private notes or only work in Obsidian ([[wikilinks]], %% comments %%, Dataview and Tasks blocks). The properties are read by a small reader of our own, with no new dependency, because Obsidian writes lists in the block form and the import's reader takes only the inline form. Runbook: docs/runbooks/obsidian-idea-notes.md, written for someone who has never used Obsidian. Checked: 36 scripts tests on temporary vaults (never a real one); functions/src/lib/cms/idea-promotion.test.js reads each promoted draft back with the import's own parser (parseRepoDraft) and holds the format's limits and content types to the import's and the publisher's, seen failing when either was changed; the runbook's steps run once in PowerShell against a temporary vault, with the outputs it shows. ci.yml's functions filter now names scripts/ideas/idea-notes.mjs. Not done: an admin screen for the backlog; promoting types other than blog; any look at the owner's real vault.
  • A sample interview, so the Interviews pages show up and there is a file to copy. Owner request 2026-10-24. frontend/src/data/interviews/example-interview.json is a made-up guest ("Sample Guest") whose page, bio and subtitle all say it is a demonstration and that no real person said any of it. It adds the Interviews section to the Azure landing page, the list page and /azure/interviews/example-interview, and puts the first two in the sitemap. Delete it when the first real interview is published (docs/runbooks/add-an-interview.md). Checked on a production build served locally: the page is prerendered with an Article record and appears in sitemap.xml; in real Chromium at 1280 px in dark and light, axe found no violations on the article, every book spread and the list page, with no console errors; the book offers no button on a 390 px phone or with reduced motion; /aws/interviews and an unknown interview are not-found pages. Not checked: the live site (not deployed), and a screen reader.

  • Interviews on the Azure section: a readable article page per interview, with a book view that turns its pages. Owner request 2026-10-24. An interview is one JSON file in frontend/src/data/interviews/; adding a file adds the interview, with no list to edit, and a file whose name starts with an underscore is a draft the site ignores. docs/runbooks/add-an-interview.md has the format, an example and the steps. Each interview gets /azure/interviews/<name> and a card on /azure/interviews. The page is the whole transcript as plain text, with a title, a description, an Article record and a link in the sitemap, and that is what the build pre-renders for search engines, so no separate transcript generator was needed. Book view is an extra the visitor chooses on a wide screen when their device is not set to reduce motion; it is a stack of CSS leaves that turn, with the two buttons beneath it as the way in (the arrow keys, Home and End work when a button has focus), a spoken position ("Pages 3 and 4 of 6"), and every page except the two on show unreachable and hidden from screen readers. No flip-book library: the one the owner's plan named was last released in April 2021, and the owner chose to build the turn here instead. Nothing is published yet: no interview exists, so there is no Interviews card on the Azure landing page, no list page in the sitemap and no pre-rendered route until the first file is added (a first version put the link in the top menu; see the Fixed entry below for why it moved). The build and its tests stop on a bad file with a sentence naming the file and the field. Checked in a real browser, dark and light, wide and phone width, with axe including colour contrast, which found and fixed four things the unit tests could not see: the site theme repainting the book's white pages and its links, a second <main> landmark, and the site's global <header> styling leaking onto the article heading. Obsidian as a second brain for the content creator is a separate piece of work and is not in this change.

  • go.hybridcloudworks.com for the Linkie link-in-bio page. Owner request 2026-10-24. Linkie serves a custom hostname that CNAMEs to app.linkie.bio and proves ownership with a TXT record, both of which its Domain settings page lists once the name is saved there. infra/frontend.tf now declares the two records the way the podcast one is: a DNS-only CNAME go, and the TXT _cf-custom-hostname.go with the value Linkie generated. Declared as Terraform records, and not added by hand or through Cloudflare's API, so the daily Cloudflare DNS audit sees them as the workspace's own and the next plan shows no drift. Linkie's page also lists rows named @ and www; it does not say whether those names are relative to go or to the zone, and read as the zone a CNAME at either would hand the site to Linkie, so only what its help article asks of a subdomain is declared: the CNAME and the TXT (an early draft of this entry, the runbook and the Terraform comment stated the zone reading as fact; it is unverified, and the wording now says so). The plan checker declared the two creates and the edge DNS runbook gains both rows and the go checks. Applied by the owner on 2026-10-24 (HCP Terraform status Success), after which both records answered on 1.1.1.1 and 8.8.8.8 and the site's own apex and www were unchanged. The first browser request got Cloudflare error 1014 "CNAME Cross-User Banned", which Cloudflare's page says is how a cross-account CNAME answers until the target's owner (here Linkie) has registered the hostname with Cloudflare for SaaS. The declared creates are deleted from the plan checker. Linkie's status moves from Failed to Live when it has registered go, which its article says can take up to 48 hours.

  • A Telegram message when the delivery monitor goes red, and the monitor fires at 8 hours instead of 24. Owner decision 2026-10-23. The "Monitor delivery health" check went red on 2026-10-22 because one line on a frontend page had waited more than a day for its by-hand deploy, and the only notice was GitHub's failure email. Now checkAgentHealth (the five-minute timer that is already running) reads that workflow's runs from GitHub's public API once an hour and sends one Telegram message titled "DELIVERY MONITOR RED", naming the job that failed and linking the run. A reminder goes again if it is still red a day later, and one "back to green" message follows when a run passes. If the bot cannot read GitHub 24 times in a row (about a day), it says so once, in a separate "WATCH IS BLIND" message, because a watch that has stopped reading looks the same as a green monitor. The repository is public, so this reads with no token: no GitHub secret was added, and the bot token stays in Key Vault. The cost is that it cannot speak while Azure is down; GitHub's own failure email still does. The threshold now lives only in scripts/check-deploy-drift.mjs (8 hours, was 24 there and again in the workflow). Functions need a deploy for the message to start; the threshold takes effect on merge. (functions/src/lib/timers/delivery-monitor-watch.js)

  • A monthly audience figure the site can state: a Cloudflare Web Analytics beacon at the edge and a monthly read into the Ambassador hub (#1302). Owner decision 2026-10-20: the Docker Captain application goes in on the Thought Leader track, which asks for 5,000 monthly views, and the Microsoft MVP form asks for impact metrics, so the site has to be able to state a monthly audience figure. Nothing is shown to visitors. infra/frontend.tf creates a Cloudflare Web Analytics site for hybridcloudworks.com with Cloudflare's lite option (no cookies, no client-side state, no fingerprinting; the option asks Cloudflare not to measure visitors in the EU, which Cloudflare documents for the snippet its proxy injects and which is unverified for the hand-placed one this site carries, so the figure is aggregate page views, read as a floor), counted on the new cloudflare_account_id workspace variable so that a plan without it stays whole, and the cloudflare_api_token gains Account → Account Settings → Write for the one account-level create. web_analytics_site_token and web_analytics_site_tag are outputs and not sensitive, because the token ships in every page. The frontend build (frontend/scripts/cf-web-analytics.mjs) puts the beacon before </head> only when VITE_CF_WEB_ANALYTICS_TOKEN is set — deploy-azure-frontend.yml reads it from the CF_WEB_ANALYTICS_TOKEN repository variable — leaving index.html byte for byte unchanged without it and failing the build on a value that is not token-shaped; connect-src gains https://cloudflareinsights.com. The ambassadorAudienceMonthly timer runs daily at 06:45 UTC, because alert-timer-overdue cannot express a calendar month, and records the previous month once: page views from Cloudflare's GraphQL Analytics API with the read-only CLOUDFLARE_ANALYTICS_TOKEN (Account Analytics: Read, a second token and never the workspace's) and podcast downloads from Podbean through the existing client ("podcast downloads unavailable" in the row until that credential is seeded), as one manual evidence row per programme under Docker Captain's and Microsoft MVP's reach requirement, dated the last day of the month and idempotent by a deterministic id. Two riders: the footer carries a "GitKraken Ambassador" text link, and the Landing Zone Builder's credit reads "based on HashiCorp's validated pattern", the wording HashiCorp's trademark policy allows, with the pattern linked. The privacy policy names Cloudflare Web Analytics as a processor and says we have asked Cloudflare to exclude EU visitors, not that they are excluded. Owner steps, in order: add Account → Account Settings → Write to the hcp-terraform-cloudflare-dns token; set cloudflare_account_id on hcw-azure and apply; confirm the EU option on the new site's Manage page in the Web Analytics dashboard (https://dash.cloudflare.com/?to=/:account/web-analytics) and record the date in the cloudflare_account_id row of required inputs §4.1; set CF_WEB_ANALYTICS_TOKEN from the web_analytics_site_token output and deploy the frontend; mint the read-only token and seed CLOUDFLARE-ANALYTICS-TOKEN on Integrations → Keys, then take CLOUDFLARE_ANALYTICS_TOKEN off EXPECTED_UNRESOLVED in scripts/check-unresolved-secrets.mjs; add AMBASSADOR_AUDIENCE_MONTHLY to enabled_timers. Required inputs §4.1, §4.2, §4.5, §4.6, §4.8 and §4.10 carry the rows.

  • GitKraken article series: three how-to drafts for the Q4 2026 Ambassador quarter (#1078). docs/content/blog-gitkraken-01-commit-graph.md reads this repository's 1,193 commits in GitLens 19.4's Commit Graph without an account; blog-gitkraken-02-mcp-server.md explains what the GitKraken MCP server's Git and pull-request tools do inside an AI coding session on a platform that ships every change through a pull request; and blog-gitkraken-03-stacked-prs.md sets the working agreements' one-conflict- at-a-time rule and the CHANGELOG.md collision beside GitLens 19.3's Rebase Stack and Desktop 12.6's stacked pull requests. Every repository figure was read on 2026-10-20 and every product claim is quoted from the release notes of that day, with the unverified items listed in each file's trailing Sources comment. Listed in mkdocs.yml as drafts like the lab and Docker series. The owner imports them once from Admin → Drafts, reviews, publishes, and logs each on the programme's quarterly check-in by 2026-12-31; the programme asks for two pieces a quarter.

  • Portainer read-only check on the lab host, reported into the Health Hub (#1282). The portainer role installs hcw-portainer-check: every 15 minutes, and once on each bootstrap.sh run, it reads Portainer on the host's loopback with the owner's access token (GET only; the token root 0600, handed to the unit through LoadCredential, never on a command line) and hands one content-free report of counts, states, version and licence dates to the site through the lab agent's new bin/report-host-check.js and POST /api/agent/reportHostCheck, kept in admin_config/host_checks. The Health Hub's Portainer card (Enhanced) judges it: degraded on an unhealthy or restarting container, an image with an update or a licence within 30 days of its end; critical when Portainer or its local environment is down, the check failed, or no report has arrived for 45 minutes. A failed run raises the existing LAB-2 alert; no new rule. The token is the owner's own (owner decision 2026-10-19), set with hcw-vault-set as vault_portainer_api_token; until it is set the check is skipped and the card says so.

  • Visitor lab sessions: brokered ephemeral Coder workspaces for visitors outside the org (#1016).

  • Implemented functions/src/lib/labs/visitor-session.js to broker ephemeral Coder users (v-<hex>) and workspaces from template hcw-lab without granting permanent sign-up or accounts.
  • Enforced strict fail-closed constraints: origin lock against site apex/www, Cloudflare Turnstile token validation (action: visitor-session), per-client daily quota (2 sessions/day), global concurrency limit (LABS_VISITOR_MAX_CONCURRENT=2), and a 60-minute hard duration limit.
  • Added self-healing opportunistic reaper cleaning up expired workspaces and users from Coder and Cosmos DB lab_jobs.
  • Registered anonymous API endpoint public/labs/visitor-session supporting GET (status & active session), POST (start session), and DELETE (stop session), registered in .azure/api-surface.json and functions/function-inventory.json.
  • Updated frontend publicApi.js and LabPanePage.jsx to render visitor session controls with Turnstile verification, frame active visitor workspace URLs with an "End session" control, and display clear capacity notices when full.

  • A private "Doc Vault" in the admin, for the owner's own templates, applications and letters, kept apart from the Ambassador Evidence tab (owner request 2026-10-26). /admin/doc-vault (Admin → Spotlight → Doc Vault, publisher and above) holds documents of six categories (Template, Application, Letter, Notes, Reference, Other) with tags (12 × 40), an optional https or /admin/ link and up to 256 KB of plain text or markdown, shown exactly as written in a read-only box and never as a web page. Add by Paste a document or by Upload files (several .md, .markdown, .txt files, sent one at a time with a result line each; a failing file does not stop the rest; front matter title, category, tags including Obsidian's one-per-line form, and link are read, every other key is named in a warning and not kept). Each document has a Copy button (the text as saved) and Copy as plain text (without markdown marks), falling back to the older copy command where the clipboard is blocked. Adding a document with the same category and title as one already here is a new version, never a duplicate (identical text writes nothing); the newest ten versions are kept, each can be read and restored, and a restore is itself a new version. Move to Trash keeps a document and its history for 30 days (put back, or Delete forever from Trash), and the list load removes what is older. Nine routes under cms/doc-vault (functions/src/lib/doc-vault/), all at the publisher role, reads included: viewer is monitoring only, editor is the role an invited co-author holds, publisher is the first role trusted to put things in front of the public, which is the bar for the place that must stay out of it. Its own Cosmos container, doc_vault (/id, the text not indexed, no TTL; backed up as authored work, infra/cosmos-containers.json, the plan checker declares the one create), created by a Terraform apply that stays the owner's: until then every route answers 503 NOT_PROVISIONED and the page says the one step. Document ids are an HMAC of category and folded title under a salt item kept in the container, so an id in an audit row, a URL or telemetry cannot be checked against a guessed title. Every write and every read of a document writes a content-free audit row (id, version, category, source, size, admin id); text, title, tags and link never reach a log line, an audit row or an error, and answers are Cache-Control: private, no-store. Writes carry the etag of the version on screen (412 CONFLICT otherwise), the old text is archived before it is replaced, and a permanent delete marks the document purging before it removes anything so it cannot lose a live document's history or leave an orphan. Registered in function-inventory.json, .azure/api-surface.json, route-inventory.test.js (publisher on every route, not public), the client timeout table and the export classification. The byte-counting JSON body reader moved from drafts-handlers.js into functions/src/lib/http/bounded-body.js so both read a body the same way (behaviour unchanged, the Drafts tests pass). Never public is held by functions/src/lib/doc-vault/no-leak.test.js: no module outside the vault names the container, a vault document fails every public clause in every state, every public route run against a store that answers every read with a vault document serves none of it, and the page is under /admin only. A security review of the diff (the read-only typescript-security-reviewer agent) found no Critical or High issue; its six findings were acted on (salted ids, safe delete order, read auditing, no-store, backup wording, spell check off in the text box) except that the publisher role is shared by every publisher, which is stated in the runbook with the one-line change to super_admin. Checked: 259 new backend tests (four files under functions/src/lib/doc-vault/) and 99 new frontend tests, with the full suites green (functions 7,044 tests, frontend 5,444); mutation checks (19 deliberate breaks of the real code, each caught by the new tests, including a quadratic scan and a public-shape document) and the linear-time test failing on the first draft of the plain-text converter before its scan was bounded. docs/runbooks/doc-vault.md explains what goes here and what does not (and the difference from Evidence), who can use it and why, and the one-time setup. Not yet live: the container does not exist until the owner confirms the HCP Terraform run, the owner deploys the functions and the frontend, and the copy buttons were tested in a test browser, not in Edge, Chrome or Firefox on Windows.

  • Admin → Calendar: the owner's own dated alerts, on the existing Calendar page, with a pop-up on sign-in and the Docker-Sponsored Open Source application (due 2026-10-30) as the first (#678). Owner request 2026-10-26, "Calendar Alerts when I log into the Admin portal", then "we already have a calendar in the admin portal so maybe we can build off of that": so there is no page or menu entry of its own. An alert is a title, a UTC day, an optional UTC time, an optional link (an https address or an /admin/... page) and a note of up to 500 characters; it is open, snoozed until a day, or done. On the Calendar (/admin/calendar) an alert is an orange "Alert" chip with a bell on its due day (a new kind alert in the GET cms/calendar read, one more collector in COLLECTORS, functions/src/lib/calendar-alerts/collector.js; same editor role, read-only, handed no write function), filterable with the toolbar's Kinds and Status; pressing it opens Done or Reopen, Snooze, Open link, Edit and Delete, and none of the reschedule, unschedule, share or cancel actions a content item has. A Calendar alerts panel under the grid lists every alert with Open, Due soon, Snoozed, Done and All filters and adds, edits, finishes, snoozes and deletes them; a change in the panel makes the grid read again and a change from a chip reloads the panel. Download .ics builds a calendar file in the browser and saves it to the owner's disk, so it has no web address. On sign-in an editor or above sees a dialog (focus goes to the dialog, not to a button; focus kept inside it; Esc closes it; role dialog) listing every open alert due within 7 days or overdue, each with Done, Snooze (1 day, 3 days, next week) and Open link; it opens at most once per browser session and once per UTC day, only inside the admin layout, and never blocks sign-in: a failed read shows a small notice with Try again, and a failure to even load the pop-up renders nothing. Backend: GET and POST cms/calendar-alerts and PATCH and DELETE cms/calendar-alerts/{alertId} (functions/src/lib/calendar-alerts/), editor and above, registered in function-inventory.json and .azure/api-surface.json. The alerts keep their own store, one document admin_config/calendar_alerts (no new container), and not a content or social record, because the Calendar owns no storage and the records it shows are the ones the scheduled publisher (scheduledPublishDate), the Publer sync (scheduledAt) and the newsletter send act on: an alert stored as one could be published or posted. It is written under its ETag with a retry on a lost race, and is read by no public route, the content manifest, the prerender, the sitemap, the newsletter or the social queue. The starting alert is made by the server on the first list read after the deploy, create-only under the fixed id docker-dsos-application, with the ids of every alert ever made recorded in the same document, so a second read changes nothing and an alert marked done or deleted is never made again (the Calendar read adds it in memory, writing nothing, if it is read first). Titles, notes and links are never logged, and the audit row holds the alert id, its state and, for an edit, the names of the fields changed, never their values. Links are printable ASCII only and may not point at an OAuth route; the link button names the site it goes to; a stored row is checked again on the way out, and the .ics builder drops control characters and the other line-end characters so no title or note can add a calendar property. Runbook: Calendar Alerts. Separate on purpose from the Telegram reminders sheet. Landed in #1390.

Quality & Verification

  • Every workflow job that runs this repository's code now has a time limit, and a test keeps it that way (#1441). A job with no timeout-minutes gets GitHub's default of six hours, so a test stuck in a loop (one vitest's own timeout cannot interrupt) would leave a pull request on "pending" for hours instead of failing. Each job that checks out the repository and then runs a shell step now carries a limit a few times its slowest successful run in October 2026, read from the Actions history (for example the verify rows 20 minutes against a slowest of 3.4, ansible-lint 35 against 10.8, the lab image build 60 against 17). The e2e limit goes from 25 to 60 minutes, because its slowest normal run took 19, 16 of them installing Chromium. Two jobs are sized for waits they are built to do rather than their run time: monitor-functions-registered (40, as it may wait up to 20 minutes for a deploy to finish) and publish-lab-image's publish job (60, as it rebuilds as slowly as the build job when the cache is empty). scripts/ci-workflow-steps.test.mjs now reads every workflow and fails when such a job has no limit, or one that is not a plain number within GitHub's cap; the three jobs its rule leaves out (addon-image-scout scout, dependency-review, docs-pages deploy) are listed in the test with the reason. The two jobs that open a firewall window in Azure for their runner and close it in an always() step (deploy-functions and publish-content-manifest's build job) also give every step up to the last close a limit of its own, adding up to at least five minutes less than the job's limit (55 of 60, and 19 of 25). A stuck step there now fails on its own limit, the close steps run as after any failure, and the job's limit can never cancel the job while a window is open; the test checks this too. Each step that adds a rule now writes the name its close step needs before the add rather than after it, and each close step checks the rule is there before removing it and says which case it found, so an add that succeeds in Azure and then hangs until the step's limit stops it still gets cleaned up. The four close steps also no longer give up at the first error. GitHub runs a step's shell with bash -e, which stops at the first failing command, so a single failed read, or a Deny that Azure refused with 409 Conflict because the open step's update was still running when its limit stopped it (Resource Manager runs one update to a resource at a time), used to skip the removal and leave the window open. Each close step now turns -e off and runs every Azure call through window_retry (scripts/lib/window-retry.sh), retrying with growing pauses (the storage Deny for up to about two and a half minutes); when it cannot read whether the rule is there it removes it anyway; and it ends by reading the result back and failing with an error if the default action is not what it should be or the rule is still there. The storage and SCM close limits go from 3 to 5 and 4 minutes to fit the retries, and Install dependencies, Azure Login and Assert every expected function is registered give back a minute each (2, 2 and 5 minutes, against slowest runs of 12 s, 17 s and 44 s), so the deploy's step limits still add up to 55 of its 60. scripts/ci-workflow-steps.test.mjs runs each close step against a stand-in az to hold this. No job or check name changed.

  • The frontend's Doc Vault "linear time" test no longer fails by chance (#1433). The test (frontend/src/pages/admin/docVault/docVaultModel.test.js, "is linear: pathological input grows in step with its size") checks that the browser's "copy as plain text" converter takes about four times as long for four times as much text, not sixteen times, which would mean it had become quadratic (slower and slower as the text grows). It timed the converter the same weak way as the server's twin of the test, fixed in #1434 for #1431.

  • The problem. The large input was timed as five single conversions, all inside a few milliseconds, against a small input with a 0.02 ms floor under it. A slow spell on a busy machine slows all five large timings at once, and nothing cancels it. Run alone here, the old test's ratio reached 8.76 against its limit of 9, and in 30 runs on Node 26 (the version CI uses for this job) it failed once, with angle: expected 10.9 to be less than 9. The floor also hid the real time of the "ticks" shape (0.005 ms), so its ratio was always read as 1.02.
  • The fix. The server test's method, copied (the two packages share no test helpers): each batch repeats the conversion until it lasts at least 4 ms, the two sizes are timed in alternating rounds, each keeps its fastest of nine rounds, there is no floor, and a failure prints both timings. Two additions: the repetition count is the first at which the faster of two timed batches lasts 4 ms, so one slow first batch (the converter not yet compiled, or a pause) cannot settle on too few (raised in review); and the small batch reads at least as much text as the large one, so both batches last about as long. Without it, "brackets" timed one conversion of 11 ms against one of 47 ms, and in a whole-suite run with other programs keeping the machine about six times busier than it has processors, it read 9.5. The six input shapes, their sizes and the limit of 9 are unchanged; each shape is now its own test, so a failure names it.

  • How it was checked. 30 readings of each shape: highest ratio 8.76 before and 4.66 after. 30 runs of the file each way, taken in turns on Node 26: 1 failure before, none after, and the group takes 1.7 s instead of 2.2 s. It still catches a slow converter (a temporary local change, not committed): letting the link scan read to the end of the line, the mistake the converter's own comment warns about, fails "brackets" (17.2) and "links" (15.8); a slowdown in the square of the text length fails all six; at 1/1,024 of that cost it fails "ticks", as the old test did.

  • The Markdown draft parser's "linear time" tests use the same steadier timing (Refs #1431). The tests (functions/src/lib/cms/repo-draft.test.js, "the parser stays linear on hostile text") check that the parser for drafts pasted or uploaded as Markdown takes about four times as long for four times the text (the limit is 8 times), so a hostile file cannot keep the server busy.

  • The problem. Each size was timed as the best of three single runs, one size after the other, with a 1 ms floor under the small timing. While another pull request ran the whole functions suite on a busy machine, "finds links in a body of repeated ](a" failed once with 11.0x. Four of the six cases also took under 1 ms, so the floor hid their real ratio.
  • The fix. The method of the entry above: that of #1434, plus its two additions: the count of repetitions is confirmed by the faster of two batches, and the small batch reads at least as much text as the large one (here "bare ](" timed one run of about 10 ms against one of 40 ms). The first of the two is also made in #1434's own test, functions/src/lib/doc-vault/markdown.test.js, which had the same calibration. The inputs, the limit of 8 and the separate 15-second ceiling test are unchanged.

  • How it was checked. On Node 24 (the version CI uses for this job), 30 readings of each case: highest ratio 5.50 before and 4.36 after, and the four cases under the floor now read 3.6 to 4.1 instead of 0.15 to 0.93. 30 runs of the file each way passed; the group takes 2.4 s instead of 0.6 s. It still catches a slow parser (temporary local changes, not committed): putting back the old link pattern fails all three link cases (17.4, 16.0 and 63.2 times) as the old test did; a slowdown in the square of the input length fails all six, where the old test missed the cubic case; at 1/16,384 of that cost it fails two, where the old test failed none.

  • A test now fails when a comment beside a lab host pin names a version or digest that the weekly bump will move, and the 23 comments that did are reworded (#1444, found on the bot's pull request #1423).

  • The problem. Every week a bot (.github/workflows/lab-supply-chain.yml, running scripts/lab-pins-upstream.mjs --bump host) opens a pull request that moves the lab host's pins (the exact versions and checksums it installs; a checksum is a fingerprint of a file, and a digest is the same for a container image). It moves the values and, on purpose, leaves the sentences in the comments beside them alone, because a script cannot rewrite prose safely. So a sentence such as "2.11.4 is the newest release" was true when written and false after the first bump. CodeRabbit (the review bot) asked for changes on #1423 for three of them (Caddy, Coder, Node.js), and there were more: the test reported 23 on main, 21 in lab-host/ansible/group_vars/all.yml and 2 in lab-host/bootstrap.sh. Fifteen already contradicted their pin (Docker Engine, Caddy, Coder, PostgreSQL, Portainer, Vault, Node.js, cosign), three named PostgreSQL's current tag and would have been false after its next bump, and five came from one comment that sat above the wrong pin (the add-on comment, which says "no automation moves these", sat over cosign_version, which the bot does move).
  • The fix. A comment beside a pin that the bump moves now names no exact version and no digest. It says "the pin below", or names a release line (the first numbers of a version: 2.38, or 18), and it keeps its history as dates ("read 2026-09-26", "released 2026-10-06") and as what was checked. The bot is not changed: it still edits values only, which is safe once no comment repeats them. scripts/lib/lab-pin-comments.mjs is the rule in code. "Beside" means the comment lines directly above the pin's group of lines, a comment at the end of the pin's own line, and every comment in the caddy test (a short script whose header describes its pin from some lines away). An exact version is one with as many dotted parts as the pin (29.9.2, or 18.6 for PostgreSQL), so release lines, 26.04, HTTP/1.1 and dates are left alone, and a 26.04 beside a two-part pin such as 18.6 is flagged. The test also fails on a comment that names the pin's current value, because that comment is true today and false after the next bump, and this way it fails when it is written and not in the bot's next pull request. The failure message names the file, the line, what the comment says and what the pin holds, says how to fix it and points at lab-host/README.md, "Bumping a pin".
  • What was reworded. The Docker, Caddy, Coder, PostgreSQL, Portainer, Vault, Node.js and cosign comments in group_vars/all.yml, and the cosign comment in bootstrap.sh. The dates, what each digest was read from, the Coder 2.38 breaking changes, the PostgreSQL 16 to 18 rehearsal and the Portainer and Vault support lines are kept. Two things moved so each comment sits over what it describes: PostgreSQL's part of the Coder comment now sits above PostgreSQL's own lines, and the cosign pair moved above the add-on comment. A note at the top of group_vars/all.yml and a paragraph in lab-host/README.md, "Bumping a pin", state the rule. The same class of sentence in the documentation now points at the pin instead of repeating it: the component table at the top of lab-host/README.md (which named Docker Engine 29.8.1, Caddy 2.11.4, Coder v2.37.3 and Vault 2.1.1 against pins that had moved on), the Docker CLI note in lab-host/README.md and roles/labs_agent/README.md, the example output lines for Portainer, roles/coder/README.md, roles/vault/README.md, roles/portainer/README.md and docs/runbooks/labs-host.md. Dated measurements ("measured on v2.38.0 on 2026-10-17") and examples in argument_specs.yml are kept as they are. The docstring of roles/coder/files/hcw-coder-automation.py (the text at the top of a Python file that says what the file does) is reworded (documentation text only: its code is unchanged, which a comparison of the parsed syntax confirms), so the next bootstrap.sh run lists that one file as changed and does nothing else for it.
  • What does not change. No pin value: the parsed group_vars/all.yml (read with a YAML parser, 87 top-level keys) is identical before and after, in the same order, and bootstrap.sh has the same lines apart from comments. No workflow or ruleset (the repository's merge rules) is edited, so the names of the required checks (the checks a pull request must pass before it merges) are the same, and nothing was run against the lab host.

  • How it was checked. 20 new tests, 35 in scripts/lab-pins-upstream.test.mjs instead of 15. Sixteen check the rule on small texts (a different version, the pin's own value, a v-prefixed tag, a digest prefix of eight hex digits or more (hex digits are 0 to 9 and a to f, the characters a digest is written in), what is allowed, which comment belongs to which pin, every line of a long comment, a nested key, a shell file, the $comment of a JSON file (JSON has no comments, so the Docker CLI fixture keeps its notes in a field of that name), the caddy test, two files whose comments sit on the same lines, a pin listed twice and Windows line ends, the message and the page it points at, a pin that cannot be found). Four run the real bumpers (the functions that make each edit, through planHost, the one that plans the whole weekly edit, with every watched product behind and fake publishers, that is publisher answers made up for the test so that nothing goes over the network) over the real files: the list of all 29 lines the bump edits is written out in the test and must match exactly, no comment names a version today, none contradicts a pin after every value has moved, and a comment that still names the old value after a bump is caught with its file and line. With the new test and the unchanged files from main, the two real-file tests failed with the 23 reports. Twenty-three deliberate breakages (three comments put back, twenty changes to the rule, its message or a bumper, such as a bumper that stops editing vault_checksum) were each caught by at least one test, then undone. Run for real against the publishers in a scratch copy of the changed files, --bump host moved nine value lines, the same nine as #1423, and no comment line, and the rule reported nothing on the result. An independent read-only review of the first version found parts of the rule that no test pinned (the v prefix, the lines of a long comment run, JSON, the eight-hex-digit limit for a digest, the file name in the key that groups pins) and the list of lines the bumpers edit; each has a test now.

  • The CI path-filter guard now follows what a module in another package imports and reads, so a change to a file two or three steps away from a test can no longer skip that test's job (#1428). Part 1 of that ticket: the check that each test job's path filter lists every file the job's tests need. Parts 2 and 3 are in #1430.

  • The problem. Each test job in .github/workflows/ci.yml runs on a pull request only when the pull request changes a file that the job's filter (a pattern for file paths) matches. scripts/ci-filters.test.mjs is the guard: it fails when a filter misses a file the job's tests need from outside their own package (a folder with its own package.json and tests, such as frontend/ or functions/). Until now it counted only the one outside file a test imports, not the files that file imports in turn, or reads. For example frontend/src/lib/aiEngine.test.js imports functions/src/lib/ai/router.js, which loads ai/select.js, cosmos-client.js, secret-vault.js and about twenty more files. A pull request that changed only one of those did not run the frontend job, although the test loads it.
  • The fix. The guard now opens every module a test reaches, in any package, and follows the chain to its end. It reads each module's imports from its code: import ... from and export ... from (a name passed on from another file), import('x') (loads a file while the program runs) and require('x') (the older way of loading a file) when the path is written out, and vi.mock('x') (a test's way of replacing a module, which still names it). A file it cannot parse, such as TypeScript, is read as plain text. For each module it also works out the files the module reads when a test runs it, with the analysis it already used on the test files, and it follows an import() or require() whose path is worked out as the program runs when that names one file. It leaves out what a pull request cannot change: package names such as lodash, anything under node_modules (where installed libraries go) or .git (git's own data), and the dist and coverage folders that sit beside a package.json (a package's build output and coverage report). A coverage or dist folder lower down, such as docs/coverage/, counts like any other. The guard is also faster: it keeps the answers the disk gives and no longer parses a file twice. A parsed file serves one analysis only, because the analysis writes notes onto it; a second round parses it again (a review found that reusing it could make a read drop out unseen).
  • What it found. 54 files for the frontend row (one entry in the list of test jobs in ci.yml) and 36 for scripts (operations), and none for functions (azure), vps-agent or the edge probe. A reviewer counted 53 and 36 by hand for #1427. The 54th, functions/src/lib/ambassador/model.js, is imported by ambassador/readiness.js, which #1438 added to the filter one step short of it. All of them are plain imports, and none of those modules reads a file from disk, so the fix is in the filters. frontend now lists the ai/, cms/ and labs/ folders of functions/src/lib/ whole, because the tests load eight or more of their code files (20 of the 25 non-test code files directly in ai/, 10 of the 23 in cms/, 8 of the 19 in labs/), and 23 single files. scripts (operations) lists cms/, labs/, http/ and cloud-tools/ whole and 12 single files. Each group has a comment naming the test behind it.
  • What it costs. A whole folder also matches the test files in it, which nothing loads (66 of them in the three frontend folders), so a change to one of them runs the job needlessly. Replayed over the last 150 merged commits with grep -E (a text search) as the workflow runs it, frontend would have run for 75 instead of 69 (two of the six are changes to files the tests really load, four are changes to test files in functions/src/lib/cms/), scripts (operations) stays at 141 and the other rows do not change. Over the last 400 commits it is 227 instead of 211 for frontend and 375 instead of 374 for scripts (operations).
  • What to do when the guard fails. The message names the file and the chain of modules that leads to it. Either add the file to the row's filter in ci.yml (a whole folder when many of its files are listed), or stop the test reaching it. Two new reasons to fail: an import in a module outside the package whose file cannot be found, and a read the guard cannot work out in such a module. For these a person writes an entry in REVIEWED_READS, the list in the test file where the reason a read is fine is written down; for a missing import the entry names it as import '<path>'. An entry may now name a module outside the package, and is stale only when no package's tests use it.

  • How it was checked. 77 new tests, 255 in the file instead of 183. 73 of them build small temporary repositories for each way a chain can go (a module that imports another, a re-export, import(), require(), a read with join or resolve, an import cycle that must end (two files that import each other), an import that cannot be found, a package name, node_modules, coverage below a package's top, a second round of the analysis); 71 of those 73 fail on main's guard code and all pass on this one, and the other two only guard what must not change. 50 deliberate breakages of the new code (a folder rule dropped, a kind of import no longer counted, a used syntax tree used again, and so on) were run one at a time against the tests: 49 each make at least one test fail, the 50th (removing the guards against import cycles) makes the run never end, and each breakage that first got through now has a test. The test file takes about 12 seconds on the machine it was timed on against about 20 for main's, and its peak memory is about 1.2 GB against 0.7 GB. The "what it does not see" note at the top of the test no longer lists this limit.

  • The CodeQL (code security scan) and Terraform/Trivy (infrastructure) checks now fail loudly when their path filter cannot be read, instead of being switched off (#1429). The same weakness #1428 fixed in ci.yml.

  • The problem. A job decides whether a pull request needs it by running grep -E (a text search) with a filter (a pattern for file paths) over the files the pull request changes. grep answers 0 (a file matches), 1 (none does) or 2 (it could not read the pattern, for example because a parenthesis is never closed). The change-detection steps of .github/workflows/codeql.yml (Detect language changes, once for each of the three languages) and .github/workflows/iac-validate.yml (Detect infra changes, in the terraform job and in the trivy job, each deciding infra/ and infra-lab/) tested that with if grep ..., which treats 2 like 1. A broken filter would therefore have skipped a CodeQL analysis, or the Terraform or Trivy checks, as "no relevant change" while the required check stayed green.
  • The fix. The steps now use the same few lines as ci.yml: 0 writes the "relevant" answer, 1 writes "not relevant", and anything else fails the step with a message that names the job and the filter. A diff that cannot be computed still runs everything, and a push to main still runs everything. The two patterns that iac-validate.yml kept inside the script now sit in INFRA_FILTER and LAB_FILTER settings, with this workflow's own file added by the script as before.
  • What does not change. The job ids, the job and matrix names that the main ruleset (the repository's merge rules) requires, the triggers, the permissions and the if: conditions are untouched, and a step that fails fails the same check, which is the point. The paths that count as a change are the same: the old and the new pattern picked identical paths for all five patterns, over 22,686 paths (the 3,084 the repository tracks and the rest made up: infra, infra-lab and infra-labs folders, .js, .jsx, .py and .pyc endings, near misses of the workflow names).
  • The tests. The new scripts/codeql-iac-workflow-steps.test.mjs runs each step with bash in a temporary git repository (a matching file, a file that does not match, three filters grep cannot read, the workflow's own file, a base commit that cannot be fetched, a push to main), checks that the lines that decide are the same in all seven change-detection steps of the three workflows, and scans every file in .github/workflows/ for if grep. Nine lines in five other workflows remain, each listed in the test with its reason: four are negated (if ! grep, which means "stop unless it matches"), so a grep that cannot read answers "stop"; five refuse a bad patch with a fixed pattern over a file that another command in the same step also reads, and that command fails the step if the file cannot be read. A new if grep fails the test. Against the old codeql.yml and iac-validate.yml, 14 of the new tests fail: the old CodeQL step printed "no python change, skipping analysis" and exited 0 on an unreadable filter.
  • Housekeeping, and one detail. The code that cuts jobs, steps and matrix rows out of a workflow, the git settings the tests use and the if grep check now live in scripts/lib/workflow-text.mjs, shared with ci-workflow-steps.test.mjs. In iac-validate.yml the infra/ answer is written before the infra-lab/ filter is read, so if only the second filter is broken the first answer is already written; the step still fails, so no later step runs and nothing is skipped.

  • Decision record: 0026.

  • A CI job whose filter grep cannot read now fails loudly instead of being switched off, and the filter guard now runs on every pull request (#1428). Two follow-ups to #1427. First, each job in .github/workflows/ci.yml decides whether a pull request needs it by running grep -E with the job's filter (a pattern for file paths) over the files the pull request changes. grep answers 0 (a file matches), 1 (none does) or 2 (it could not read the pattern, for example because a parenthesis is never closed). The four change-detection steps (the verify matrix, ansible-lint (lab-host), coder (lab-host) and harness (python)) tested that answer with if grep ..., which treats 2 like 1, so a broken filter would have skipped its job as "no relevant change" and left the pull request looking green. The steps now keep the answer: 0 writes relevant=true, 1 writes relevant=false, and anything else fails the step with a message that names the job and the filter. A diff that cannot be computed still runs everything, as before. The three steps that kept their pattern inside the script now take it from a FILTER setting like the matrix step, so the lines that decide are the same in all four, and the filter is put in brackets before ci.yml is added to it, so a stray backslash at the end of a filter cannot join onto the | and hide the mistake. The new scripts/ci-workflow-steps.test.mjs takes each step's shell out of ci.yml and runs it with bash in a temporary git repository (a matching file, a file that does not match, three filters grep cannot read, ci.yml itself, a base commit that cannot be fetched, a push to main); on the old ci.yml 20 of its tests fail, the broken filter ending with exit code 0 and relevant=false. Second, the guard (scripts/ci-filters.test.mjs) lives in the scripts package, so it ran only when the scripts (operations) row ran, and a pull request that changed only a frontend or functions test first met it on main after the merge. A new step, "CI path filters list every file the tests read", runs it on every pull request, outside the filter, like the two checks beside it (oidc-subjects.test.mjs is the model), and the scripts row now installs its packages on every pull request so the step can run; the same new test file keeps the step outside the filter and works out which steps each row runs for a pull request that changes only frontend/src/x.test.jsx. Replayed over the last 150 merged commits on main, the guard ran for 141 (94%) before and runs for all 150 now; adding the other packages' test files to the row's filter would have reached 143 and adding their folders whole 145, and both keep a list that someone has to keep up by hand. On the runner, in this change's own CI run, the install took 5 seconds and the new step 12 (about 15 seconds for the guard on its own on the machine it was first timed on), and the scripts row is not on the longest path: on the last run on main scripts (operations) took 53 seconds and frontend 165. codeql.yml and iac-validate.yml still used the if grep shape when this was written and were not changed here (fixed in the entry above, under #1429). Decision record: 0026.

  • The Doc Vault "linear time" test no longer fails by chance (#1431). The test (functions/src/lib/doc-vault/markdown.test.js, group "linear time") checks that the reader for uploaded Markdown files takes about four times as long for four times as much text, not sixteen times, which would mean the reader had become quadratic (slower and slower as the text grows, so one big file could keep the server busy). It does that by timing the reader with a clock. On 2026-10-27 it failed in CI run 40484952326 with a ratio of 11.9 (the limit is 9) on a pull request that changed only documentation, so the cause was timing noise on the shared test machine and not the reader. The failure could not be repeated here (0 failures in 300 runs of the file on an idle machine and in 100 runs with 48 busy programs running beside it), so the cause was read from the timings instead. The large input was timed as five single parses of about a tenth to a quarter of a millisecond each, all inside about two milliseconds. Across 8,144 local readings of the old measurement the ratio of the four shapes that do real work was normally 3.5 to 4.1 and reached 6.9 at most; in the worst cases whose individual timings were recorded (ratios of 5.8 and 6.3), all five large timings were slow together (1.6 to 1.9 times their usual) and the small timings, taken earlier, could not cancel that. Three of the eight shapes also sat under a floor of 0.02 ms that was 3 to 14 times their real time (0.0014 to 0.007 ms), so their growth was never measured. Now every timing repeats the parse until it lasts at least 4 ms, the small and large inputs are timed in alternating rounds so a slow moment lands on both and cancels out, each keeps its fastest of nine rounds, and the floor is gone. The check is still the ratio, still under 9, on the same eight shapes, and a failure now prints both timings. Measured side by side in one process on an idle machine (2,000 readings each): the highest ratio fell from 5.04 to 4.25, and the gap between the median and the 99th percentile for the four shapes that do real work fell from 0.58 to 0.29. It still has teeth: against a reader made quadratic on purpose (a temporary local change, not committed) the old and the new test both fail on all eight shapes when the slowdown follows the text length, and on exactly the shapes a slowdown touches when it follows the line count (3), a value's length (4) or a heading's length (1); for a mild slowdown (1/1,024, 1/16,384 and 1/262,144 of that cost) the new test fails on 6, 3 and 1 shapes against 4, 2 and 0 for the old one. The price is time: the group takes about 1.0 s instead of 0.06 s on an idle machine, and 2.9 s instead of 0.26 s with the busy programs. Not verified: Node 24 (CI's version; this machine has 26.5) and a CI run. The same timing method in frontend/src/pages/admin/docVault/docVaultModel.test.js (the plain-text converter) has the same weakness and is not changed here.

  • The CI path-filter guard now works out which files a test reads, so a read written with join(...) can no longer slip past it (#1414). Each test job in .github/workflows/ci.yml runs on a pull request only when the pull request changes a file its filter matches, and scripts/ci-filters.test.mjs fails when a test reads a file outside its package that the filter misses. The first version cut each read off at its first comma, so readFileSync(join(HERE, '..', 'functions', 'x.js')) was read as join(HERE and dropped, which is how the Cloud Crawler registry file was missed in #1413. scripts/lib/js-file-reads.mjs now parses the test with espree (the parser ESLint already uses; it becomes a direct dev dependency of scripts/) and follows join, resolve, new URL, variables, loops and helper functions, including helpers in other files. A read it cannot follow, or a use of the file functions it cannot see through, fails the test instead of being skipped; the only way past is a reason written in REVIEWED_READS, and an entry that no longer matches a read fails too. Run against main's filters it reports 40 gaps: 33 files and 4 folders that tests read, and 3 .nvmrc files that the version-floors checks read. The filters now cover them. functions (azure) also runs for any change under frontend/src/, frontend/scripts/, frontend/public/ and scripts/, because the Doc Vault test reads every source file in them: over the last 150 merged pull requests it would have run on 111 instead of 66, frontend on 67 instead of 64, and scripts (operations) on the same 141. What the guard still cannot see is listed at the top of the test.

  • CI now runs npm run validate:routes on every pull request that touches the frontend (#1345). The validator fails when a page spells a provider path by hand (href="/azure/...") instead of building it from the route factory, which stays correct until a route is renamed and then breaks that page silently. It was not run by CI, so #1337 merged two such paths and only a hand run found them (fixed in #1338). It is a new Route validation step in the frontend row of .github/workflows/ci.yml, after Format check, switched on by a validate key on that row so no other package runs it. Seen both ways locally: exit 0 on the current tree, and exit 1 with the file and the path named when a component with a hard-coded href="/azure/..." was planted, then removed. Not yet seen in a CI run; this pull request's own frontend job is the first.

  • The AVM versions bot's check step reads the pins instead of remembering them (#1287). update-avm-versions.yml failed on 2026-10-06 and 2026-10-13 in its own vitest step after a bump: seven expectations in labImage.test.js, hcl.test.js and LzLabValidate.test.jsx carried the pinned versions (0.21.0, 0.9.0, 0.22.2, the lab's 0.15.0, 0.22.2, an absent 0.23.0) as literals that -u cannot refresh. They now derive the pin from AVM_MODULES, what the lab carries from LAB_IMAGE_AVM, and an absent release as one patch above the highest carried; the constraint evaluator's table keeps a fixed shelf of its own, since its rule is not about the pins. The same fix moved the path filter in front of -u: vitest run -u <path> took the path as the flag's value and ran the whole suite, refreshing every snapshot in the tree; and the bot's plain check now runs the lab pane's suite as well. Proven by applying the bot's 2026-10-13 bump (avm-ptn-alz 0.22.0, the connectivity module 0.17.7, the spoke module 0.22.3) locally: the seven failures reproduced before the change and the three suites passed after it.
  • The three LAB-2 lab-host rules are on the Fire record (#981). The owner ran the three tests from main on 2026-10-18: alert-lab-unit-failed fired at 14:46Z (ten minutes after the one syslog line), alert-lab-heartbeat at 15:36Z (the agent stopped 14:52Z, started again and active), alert-lab-disk at 16:20Z (a 138.9 GiB file to 87%, removed, 15% again). Every rule the lab host carries has now been seen to fire; the runbook's Fire record rows replace the three not yet rows.
  • Fire tests for the three LAB-2 lab-host rules that had never fired (#981, LAB-2). scripts/alerts/Invoke-AlertFireTest.ps1 gains lab-unit-failed, lab-heartbeat and lab-disk, in the script's own shape: a stop condition read over ssh hcw-lab, one staged change, a bounded wait on Azure Monitor's fired alerts, and a put-back in a finally that runs on failure or Ctrl+C and names the exact ssh line when it cannot confirm.
  • lab-unit-failed writes one syslog line with logger -t hcw-unit-failed -p daemon.err, exactly as the hcw-unit-failed@ notifier does, naming a hcw-fire-test-<UTC stamp>.service unit that does not exist; it refuses while the hour's window already holds a notifier line, and has nothing to put back (stateless rule, six-hour mute). Up to 35 minutes.
  • lab-heartbeat stops azuremonitoragent (never himdsd) until alert-lab-heartbeat sees an empty 30-minute window, and starts it again in the finally; it refuses when ssh cannot reach the host, a labjob- container is running, or the agent is not active beforehand. Up to 55 minutes, during which the host ships no Heartbeat, Perf or Syslog rows.
  • lab-disk reads df -B1 on the host, refuses past 80% used or when 87% would leave under 10 GiB free, allocates /var/tmp/hcw-fire-test-disk.img sized on the host from df at that moment (rounded down to a MiB) so the root reads 87%, holds it for alert-lab-disk's hourly evaluation, and removes it in the finally; the lab DCR's 60-second % Used Space sample sets the floor. Up to 80 minutes.
  • The runbook's fire-test and recovery tables carry the three, with by-hand blocks and not yet Fire record rows that name the test; Invoke-AlertFireTest.Tests.ps1 renders each under -WhatIf, refuses on each stop condition, and holds the runbook's table to the script's -Test list.
  • Credential-roll code scanning alerts closed (#1247, #1248 follow-up). The AWS roll's Key Vault rollback no longer re-checks currentAccessKeyId, which the function already requires (CodeQL js/trivial-conditional, alert 1416); the fetch stubs in functions/src/lib/credentials/google-roll.test.js and scripts/roll-cloud-credentials.test.mjs match a probe by new URL(url).hostname rather than by a substring anywhere in the URL (CodeQL js/incomplete-url-substring-sanitization, alerts 1409–1415), so a stub can no longer answer for a host that merely embeds the vendor's name.
  • Deployment verification: fixed post-deploy frontend probe and deployed smoke test RPC check.
  • In scripts/smoke-deployed.mjs, moved POST /api/aiProxy into the live admin guard verification (asserting 401/403 for unauthenticated callers) and verified that unmapped RPCs return 404 without cross-root file imports.
  • In .github/workflows/deploy-azure-frontend.yml, added DEFAULT_HOSTNAME_OVERRIDE defaulting to calm-ground-0d0e6a010.7.azurestaticapps.net (with fallback to vars.STATIC_WEB_APP_DEFAULT_HOSTNAME), enabling the post-deploy probe to verify HTTP 200 without requiring Microsoft.Web/staticSites/read permissions on the deployer identity.

  • In .github/workflows/deploy-azure-frontend.yml, updated probe_url to stream curl bodies directly to a temporary file, eliminating SIGPIPE (Broken pipe) under pipefail when inspecting large HTML responses.

  • Validation & regression test plan: completed contract, accessibility, and infrastructure test gates (#982).

  • Contract: added scripts/ruleset-contexts.test.mjs asserting strict 1:1 parity between GitHub Ruleset 20680114 required status checks and workflow job contexts declared on pull requests.
  • Accessibility: extended frontend/src/pages/admin/adminA11y.test.jsx with full vitest-axe validation for HealthPage ensuring zero violations.
  • Infrastructure: added infra/tests/guards.tftest.hcl with mocked providers verifying absence of keyless AzureWebJobsStorage, environment-only deploy identity subjects, Function App scoping of Cosmos DB data roles, and network isolation, wired directly into .github/workflows/iac-validate.yml.
  • CI: the lab-host role tests are their own job, a workflow-only change runs the lint-class jobs only, and CodeQL runs after the merge (#1545, second pull request). The owner's question of 2026-11-04 was whether ansible-lint (lab-host) had to run every time: it took nine minutes on #1549, a pull request that moved no lab-host file, because the job also ran every role test in sequence and because ci.yml was in every job's filter. Three changes in .github/workflows/ci.yml. ansible-lint (lab-host) keeps the lint, the playbook syntax check and the argument-spec check (about a minute) on lab-host/ changes; a new lab-host role tests job runs the twenty-six role tests, only when lab-host/ansible/roles/, group_vars/, inventory/ or one of the files outside them that the tests assert on change (lab-host/coder/, lab-host/bootstrap.sh, lab-host/README.md, two infra files, the labs-host runbook and three vps-agent files, each listed with its test in ROLE_TEST_READS in scripts/ci-workflow-steps.test.mjs, which holds the filter to that list), and on every push to main; gate needs both. Each changes step now says with WORKFLOW_SELF_TRIGGER whether a change to ci.yml alone counts: yes for the lint, the harness and the matrix rows, no for the browser suites, the role tests and the Coder template checks, which read no line of the workflow; scripts/ci-workflow-steps.test.mjs pins the list and runs each step with bash against a change to ci.yml alone and beside a matching file. The ufw closed-ports test stays as it is: its 195 s are thirty-five ansible-playbook runs that each change the one scratch ufw the next one reads, so they cannot run side by side without a namespace per run; its header says so. With gate the only CI context the ruleset requires (applied by the owner on 2026-11-04), codeql.yml runs on pushes to main and weekly, not on pull requests, with its detect step and scripts/codeql-iac-workflow-steps.test.mjs (now iac-workflow-steps.test.mjs) gone; the trade-off (an alert lands after the merge, no code scanning comments on the pull request) is in docs/runbooks/required-checks.md, and .claude/CLAUDE.md says the five required checks are in force and that a pull request title is one line of at most about 72 characters in the shape <area>: <what changed> (#<issue>).

Security

  • Lab image: a newer Azure CLI can no longer be quietly downgraded, and the review findings on #1466 are tidied (#1426). The full build now stops, naming the line to delete, when the Azure CLI already ships a package at or above the version AZURE_CLI_PYTHON_OVERRIDES lists (the weekly bump moves the CLI alone, so a newer CLI would otherwise have been put back on the older package). The replacement now happens in the same step that installs the CLI, so no image layer keeps the old, flagged files, and the ansible-core install runs with no network. An empty or deleted overrides block now builds. smoke.sh signs RS256 and PS256 sign-in assertions with msal, reads a token's tenant id with PyJWT and fetches a page over requests and urllib3 on 127.0.0.1, all with no network, and a new scripts/lab-image-azure-cli-overrides.test.mjs checks every override line's shape. versions.env now says truthfully why cryptography stays at 48.0.1 (Microsoft still pins pyOpenSSL 26.2.0; newer pyOpenSSL lifts the cap but removes APIs the CLI's certificate commands use, and neither open cryptography advisory is reachable from the CLI) and that the base image's patch-tag digest will move once more when its riscv64 build lands. lab-image/README.md explains the overrides block and that urllib3 2.8.0's proxy change leaves the CLI's https:// proxies checked against REQUESTS_CA_BUNDLE as before.

  • The lab image's critical Python findings are fixed, and the base moves to CPython 3.14.10 (#1426). Every CRITICAL and HIGH Python finding the weekly scan reported in the full image was inside the Azure CLI's own bundled Python under /opt/az, and Azure CLI 2.91.0 is still Microsoft's newest release. The build now replaces certifi (2026.10.23), PyJWT (2.15.1), urllib3 (2.8.0) and the CLI's pip (26.2.2) there, from hash-checked wheels, and runs pip check on the result; smoke.sh checks each version and that the CLI's sign-in libraries still import. lab-image/versions.env (AZURE_CLI_PYTHON_OVERRIDES) records what waits on a publisher: cryptography 48.0.1, which pyOpenSSL in the same bundle caps below 49, oauthlib 3.2.2, whose fix is a new major version Microsoft has not adopted, and the copies of msgpack, setuptools and urllib3 that pip keeps inside itself. The base is python:3.14.10-slim-trixie by its own index digest (the line tag differed only by a riscv64 image still on 3.14.9), and terraform 1.17.1, kubectl 1.37.2 and azurerm 5.12.0 move to their newest releases, each sum read from the publisher.

  • Sign-in library @azure/msal-node no longer forced; @azure/identity 4.14.1 asks for it itself (#1346). The 2026-10-24 entry below added an npm overrides line (a line that makes npm install a version the parent package did not ask for) to force msal-node 7.1.1, because @azure/identity 4.14.0 still asked for ^6.0.0 and every 6.x is open to GHSA-xcx6-jx8p-x3h5. @azure/identity 4.14.1, released by Microsoft, now asks for ^7.1.0, so functions, scripts and vps-agent move to ^4.14.1 and the override line is gone from each package.json. The installed version does not change (npm ls @azure/msal-node still shows 7.1.1 in all three, now under a range the identity package declares), and the accepted-risk row in TODO.md is removed. Not tested: real Azure sign-in, meaning Azure Arc on the lab host and the workload-identity login in CI; confirm after the next deploy.

  • Sign-in library @azure/msal-node forced to 7.1.1 in functions, scripts and vps-agent, closing three High Dependabot alerts (#163, #164, #165). Owner decision 2026-10-24. The alerts are for GHSA-xcx6-jx8p-x3h5 (CVSS 7.5): in msal-node's authorization-code flow, parameters can be sent to an attacker-controlled endpoint; fixed in 7.1.0. @azure/identity 4.14.0, which pulls the library in, still asks for ^6.0.0, so Dependabot could not fix it by itself; an npm overrides entry in each package.json now makes every copy 7.1.1. Before opening this, the managed-identity path (the one the Function App uses for Cosmos, Blob and Key Vault) was run against a stand-in for the App Service identity endpoint on 6.0.1 and again on 7.1.1, in all three packages: the same six checks passed both times (system-assigned and client-id tokens, the token cache, concurrent calls, DefaultAzureCredential, and a refused request). All three test suites pass. Not tested: the real Azure endpoint, Azure Arc on the lab host, and the workload-identity login in CI; the owner deploys the functions and watches the delivery monitor. The override is recorded as an accepted risk in TODO.md with its removal trigger.
  • Coder rotation credential: measured, and kept unscoped with the reason (#1037). On the live Coder 2.38.0 on 2026-10-19, with throwaway tokens of hcw-status: a token with coder:templates.author, organization:read and user:read publishes the template and cannot mint keys; a token with api_key:create and user:read mints tokens with scopes it does not hold and those tokens use them, so the inheritance rule in Coder's current docs is not enforced on this version; scopes are enforced on reads by filtering (workspace:read alone sees no templates), and an out-of-scope write answers 500 rather than 403. Since any credential that renews the status token can mint any scope, narrowing the rotation credential contains nothing against a leak; it stays coder:all and containment stays on the account. The results are the second table under "The status token for the site" in lab-host/README.md; the role README and the runbook point at it. Three tokens of hcw-status that nothing held (two hand-made status tokens superseded on 2026-10-10, and the old host's rotation credential restored by the snapshot) were deleted.
  • LAB-9 rebuild rehearsal done on the live lab host: 66 min 42 s (#955). 2026-10-18, VM 939861 recreated through Hostinger's API on the same plain Ubuntu 26.04 template; Vault unsealed on the snapshot's data 66 min 42 s after the click, Coder back on the restored database at 58 min. The runbook's "Reinstalling the host" step 7 is rewritten in the order the rehearsal proved: the nine vault keys set by hand (three add-on Turnstile secrets, not one), Arc's two runs and then Connect-Lab.ps1 -Arc before any further play (the post-onboarding run closes port 22), the hcw-azure run for the new machine identity, bootstrap.sh --skip-tags coder (Coder's first account cannot be made by password on this build, and the restore brings the accounts back), the Coder restore, the seal check, the agent go-live as the publishing run, Vault init and restore in one on-host command whose root token never reaches a screen, the generate-root seed, the Coder automation's rotation credential, Portainer 2.45.2's self-created local environment. The Caddy token's client-IP filter now names both of the host's addresses, because the rebuilt host reaches the Cloudflare API over IPv6 and issuance failed until the /128 was added; the token is rolled, not re-created, on a rebuild. Required inputs record a Cloudflare token naming rule (where it lives, what uses it, what it reaches) after the audit poller's token was revoked in place of the production-zone Caddy token on the cut-over evening; both corrected the same night, the poller reading again with 0 unexpected.
  • Lab DNS-01 by delegation into a dedicated lab zone (LAB-1, #948). Owner decision 2026-10-18: the lab zone is hybridcloudinsights.com, a domain the owner already held, connected to the same Cloudflare account that day (Free plan, nameservers moved at Hostinger, zone emptied). infra-lab/dns.tf adds _acme-challenge.lab and _acme-challenge.coder.lab as DNS-only CNAMEs to _acme-challenge.lab.hybridcloudinsights.com (variable lab_acme_challenge_target, output lab_acme_delegations, test asserts); roles/caddy gains caddy_dns_challenge_override_domain, rendered as the Caddyfile's dns_challenge_override_domain, because certmagic finds the zone for a challenge name by SOA walk and does not follow a CNAME. The runtime token is re-issued with DNS edit on the lab zone alone and rotated with hcw-vault-set + bootstrap.sh; the lab-host README has the token form, the forced-issuance proof (the stored certificate copied aside first, with the restore line) and the revocation of the production-zone token. Completed the same evening (21:04Z): the hcw-lab workspace applied the two CNAMEs once its VCS source was repointed to the moved repository (it had planned nothing since 2026-10-07), the token was rotated, a forced issuance obtained a new *.lab.hybridcloudworks.com certificate through the delegation, and the production-zone token was revoked; ADR 0032's LAB-1 interim is closed with the date. The Cloudflare DNS audit's caddy-acme allowlist entry is dated to match (CADDY_PRODUCTION_ZONE_ACCESS_ENDED, 2026-10-18T20:53Z, the minute the host switched tokens): a challenge TXT in the production zone from the lab host's address at or after that instant has no credential that could have made it and reads UNEXPECTED, with a regression test (CodeRabbit on #1280). Every other document that still said in the present tense that the token writes the production zone now says until when it did.
  • Cloudflare DNS audit: an apply that errored or was cancelled is still an apply (#948). The first live run of the apply-correlated hcp-terraform entry (2026-10-18 14:09Z) read 44 applies and still left four account-token edits UNEXPECTED — 2026-10-06 00:10Z and 2026-10-10 21:40Z, HCP Terraform runs whose apply stopped early after writing, which filter[status]=applied excluded. fetchTerraformApplies now reads every run, keeps each one whose apply began (applying-at) and closes its window at the first of applied-at, errored-at, canceled-at or force-canceled-at (the end of the read window for an apply still running); plan-only and discarded runs carry no applying-at and are skipped. The report names each apply's outcome.
  • Cloudflare DNS audit: the first live run's lessons (#948). The workflow's audit step caught the script's exit code on the command itself (the runner's bash -e ended the step before $? was read, so the first run failed instead of recording exit 2); a manual run can untick open_issue to report without escalating; and the report prints, for each UNEXPECTED change, the entry's shape (actor field names, request method, user agent; never values), because the first run showed the HCP Terraform token's edits arrive as account/api_token with no token_name. The allowlist is rebound to what the log carries: hcp-terraform is an account-owned token with the Cloudflare Terraform provider's user agent, not the lab host's address, while an apply of hcw/hcw-azure was in progress by HCP Terraform's own run record (read with the TFC_TOKEN secret; a user agent alone is a header anyone can send, as CodeRabbit noted); a new terraform-owner-token entry keeps the owner's own workstation terraform runs (the 2026-10-06 00:26Z edits) visible under their own name; caddy-acme is told by the lab host's address in the new CLOUDFLARE_CADDY_HOST_IP variable, the one address the token's client-IP filter allows, with the token name kept as a second key should the log ever carry it.
  • Cloudflare DNS audit: a daily poller of the account audit log stands in for the notification Cloudflare does not have (LAB-1 interim, #948). Owner decision 2026-10-18: Cloudflare has no notification type for audit-log or DNS-record events on any plan, so the interim for the Caddy token's DNS edit on the production zone is .github/workflows/cloudflare-dns-audit.yml. Daily at 05:50 UTC it runs scripts/cloudflare-dns-audit.mjs, which reads Audit Logs v2 (GET /accounts/{account_id}/logs/audit, every page) from two hours before the previous successful scheduled run (at least 26 hours, at most 14 days; manual runs do not move the resume point), so a failed run widens the next window rather than leaving a gap, keeps the create, update and delete events on DNS records in hybridcloudworks.com, and judges each against an allowlist carried at the top of the script with the reason for every entry: the hcp-terraform-cloudflare-dns token's edits, the Caddy token's TXT _acme-challenge.*.lab edits (an A record from it, or a challenge for a site name, is the finding), and the owner's own dashboard edits. An UNEXPECTED change opens Cloudflare DNS audit: unexpected record change (<date>), labelled Security and live-check, or comments on the open one; a read that fails (no token, a non-200, a body that is not the success: true envelope, more than twenty pages) fails the run rather than passing quietly. Content-free: the token never leaves the environment, record contents are not read into the report, and e-mail addresses are masked. scripts/cloudflare-dns-audit.test.mjs covers each allowlist entry, the refusals, two-page reads, the non-200 and the empty window from hand-written fixtures in the documented shape. The runbook section is docs/runbooks/edge-dns-verification.md, "The daily DNS audit"; ADR 0032's LAB-1 bullet now names this as the interim. The owner steps are the CLOUDFLARE_AUDIT_LOG_TOKEN secret (Account → Account Settings → Read, the permission the endpoint accepts) and the CLOUDFLARE_ACCOUNT_ID variable, with the two optional allowlist variables, in docs/standards/required-inputs.md §4.2 and §4.3.
  • CodeRabbit's approval is the required review (#972, option F). Owner decision 2026-10-18: .coderabbit.yaml turns on reviews.request_changes_workflow, so CodeRabbit requests changes while a finding is open and approves the head once every thread is resolved and its pre-merge checks pass, and sets reviews.allow_author_approval: false, so @coderabbitai approve from the author (sessions author as the owner's account) cannot stand in for it. The Default ruleset's pull_request rule takes required_approving_review_count: 1 with require_last_push_approval by the owner's PUT; the rollback is the same PUT with the count at 0.
  • Lab host: a missing sshd drop-in keeps sshd on the loopback while the Arc agent is Disconnected (#1263 follow-up, CodeRabbit on #1263).
  • The gap: in its hold state, the Arc gate keeps sshd's bind as the drop-in has it. With no drop-in to read, it set hardening_sshd_loopback_only false, so sshd listened on every address while ufw's rule stayed held.
  • The fix: a missing drop-in now keeps the role's default, the loopback alone.
  • Tests: tests/ufw-closed-ports.test.sh covers the case, and stops on a drop-in it cannot read (55 checks).
  • Lab host: sshd's loopback bind follows the Arc gate, as port 22 does (#1257 follow-up, CodeRabbit on #1261). #1260 bound sshd to the loopback on every run. Opening 22 at ufw then no longer gave a path in, which broke two documented routes: a rebuilt host's onboarding over 22 (Register-LabArc.ps1), and the Web Console reopen in "When Arc SSH is down".
  • One decision: the hardening role's new tasks/arc_gate.yml decides once, before sshd.yml and on every run whatever the tags, from the Arc agent. ufw's 22 rule and sshd's ListenAddress lines both follow it: closed and on the loopback while Arc SSH is ready, open and on every address while it is not, and left as they are while the agent reads Disconnected.
  • The listener check: privilege_checks_public_tcp_ports admits 22 only on a run where sshd listens on every address.
  • The arc role's guard: it now also requires a live listener beyond the loopback before Arc SSH is turned off, and that listener must answer with an SSH banner.
  • The sshd -T read-back: while sshd should listen on every address, it requires an address beyond the loopback.
  • Tests: ufw-closed-ports.test.sh has 53 checks, with real listeners standing in for sshd and a bounded wait for each, and sshd-apply.test.sh has 24. The runbook's reopen now rebinds sshd too.
  • Lab host: turning Arc SSH off never leaves both SSH paths shut (#953 follow-up, CodeRabbit on #1258). Port 22 has been closed at ufw while Arc SSH is ready since #1258. A run with arc_incoming_ssh_enabled false could close 22 in the hardening role, then clear the Arc agent's incoming ports later in the same run.
  • Hardening: hardening_arc_ssh_wanted, which follows that switch in group_vars, makes the gate open 22 in such a run.
  • Arc role: the new tasks/ssh_off_guard.yml refuses to turn Arc SSH off while ufw keeps 22 closed, as in a run narrowed to --tags arc. It reads ufw conservatively: an installed ufw must answer, any DENY or REJECT naming 22 refuses, and only a general allow from Anywhere counts.

  • Tests: tests/ufw-closed-ports.test.sh runs both halves against the real ufw (46 checks).

  • Lab host: sshd listens on the loopback only (defense in depth after #953 stage 3, #1257).

  • In 00-hcw-hardening.conf.j2, added ListenAddress 127.0.0.1 and ListenAddress ::1 so sshd binds to the loopback alone.
  • On Ubuntu, sshd-socket-generator derives ssh.socket's ListenStream= lines from these directives at systemctl daemon-reload; added daemon_reload: true to Reload ssh and added Restart ssh.socket handler.
  • Removed port 22 from privilege_checks_public_tcp_ports (defaults: [80, 443]), turning the loopback bind into a verified assertion in tasks/listeners.yml.

  • Added listenaddress checks in tasks/sshd.yml against sshd -T, updated sshd-apply.test.sh and listeners.test.sh, and updated documentation in docs/runbooks/labs-host.md and role READMEs.

  • Lab host: port 22 closed to the internet once Azure Arc SSH is ready (#953 stage 3, LAB-7). The hardening role's ufw tasks (now tasks/ufw.yml) delete the allow of each port in hardening_ufw_closed_tcp_ports, 22 on the lab host. A run then reads ufw status and fails if any ALLOW or LIMIT rule still names a closed port: alone or in a list, on any interface or address, or as Ubuntu's OpenSSH profile. hardening_ssh_arc_gate closes 22 only while the Arc agent is Connected with incoming connections on for port 22. While the agent is Disconnected (a drop, or a reset agent being onboarded again) it leaves 22 as it is. Otherwise, as on a rebuilt host that Register-LabArc.ps1 onboards over 22, it keeps 22 open, and the run says which. tests/ufw-closed-ports.test.sh runs the tasks against the real ufw in a scratch network and mount namespace with a stub azcmagent, 36 checks, in CI's ansible-lint (lab-host) job. alert-lab-ssh-login now pages any login that does not arrive through Arc (lab_ssh_loopback_only defaults to true). The query change is declared for the plan check, and Arc SSH's two applied creates are pruned from it. sshd still listens on every address behind ufw, so 22 stays in privilege_checks_public_tcp_ports until #1257 binds sshd to the loopback. The runbook adds "Closing port 22: the first run" and "When Arc SSH is down", the Web Console reopen.

  • Lab host: the play also fails if the host daemon publishes a port beyond the loopback (#958 follow-up). A socket read sees a published port only through Docker's userland proxy, a daemon setting. So privilege_checks/tasks/listeners.yml also reads the nat table's DOCKER chain, IPv4 and IPv6, and refuses any DNAT rule without a loopback destination. On 2026-10-17 the host's five rules were all -d 127.0.0.1/32. tests/listeners.test.sh now has 23 checks, and the check refuses a Docker firewall backend other than iptables, which its rule read does not cover.

  • Lab host: the play fails if anything listens beyond the loopback but sshd and Caddy (#958, LAB-12). The last check of every run (roles/privilege_checks, tasks/listeners.yml) reads every TCP and UDP listener with ss, the rootless sandbox's forwarded ports included.
  • What fails: anything not on 127.0.0.0/8, ::1 or an IPv4-mapped loopback address, unless it is TCP on privilege_checks_public_tcp_ports (22, 80 and 443). The failure names each one.
  • UDP and port 22: no UDP port is public now that Caddy serves no HTTP/3, and port 22 leaves the list when it closes (#953, stage 3).
  • The isolation chain: the other half of LAB-12's acceptance is the host firewall's read-back (#1236), which fails the play when the live chains differ from the files.

  • Tests: tests/listeners.test.sh (15 checks, in the ansible-lint (lab-host) job). On the lab host's own ss output of 2026-10-17 it refuses only Caddy's UDP 443, which the next bootstrap removes, and passes without it. The runbook gains "Listeners beyond the loopback".

  • Audit finding PLAT-8: Cosmos DB export immutability, dedicated state container, 90-day security log retention & PII data-plane auditing (#968).

  • Added time-based WORM immutability policy azurerm_storage_container_immutability_policy.cosmos_export in infra/storage.tf enforcing 35-day retention matching the expire-cosmos-export lifecycle policy so backup blobs cannot be deleted or overwritten by the application identity or non-admin roles. Parameterized locking via var.cosmos_export_immutability_locked.
  • Added dedicated mutable container azurerm_storage_container.cosmos_export_state (cosmos-export-state) for change-feed continuation tokens (state/${container}.json), preventing daily delta continuation token overwrites from failing against WORM container retention (BlobImmutableDueToPolicy). Updated createExportBlobStore in functions/src/lib/backup/cosmos-export-edges.js to route state/ continuation blobs to cosmos-export-state.
  • Increased Log Analytics workspace retention from 30 days to 90 days via var.logs_retention_in_days (default 90) in infra/main.tf and infra/variables.tf, satisfying compliance retention requirements for diagnostics, security tables, syslog, and activity logs.
  • Enabled Cosmos DB DataPlaneRequests in azurerm_monitor_diagnostic_setting.cosmos in infra/observability.tf, with an ingestion-time table transformation (azapi_update_resource.cosmos_data_plane_table) filtering telemetry to personal data containers (ambassador), providing a compliant 90-day PII access audit trail without exhausting the workspace daily ingestion quota.
  • Documented quarterly out-of-subscription export strategy, immutability verification commands, and state container isolation in docs/decisions/0028-cosmos-out-of-account-export.md and docs/runbooks/cosmos-restore.md.

  • Added unit tests in scripts/backup-immutability-retention.test.mjs, functions/src/lib/backup/cosmos-export-edges.test.js, and updated scripts/assert-expected-plan.mjs.

  • Coder: one running workspace per learner, and cloud sign-ins kept in memory (#954, LAB-8). Owner decision 2026-10-16, in all three templates (hcw-lab, amo-guided-lab, na-guided-lab).

  • Sign-ins in memory: a 64 MiB tmpfs at /tmp/cloud-creds (0700, noexec, nosuid, nodev), with AZURE_CONFIG_DIR, AWS_CONFIG_FILE, AWS_SHARED_CREDENTIALS_FILE and CLOUDSDK_CONFIG pointing into it. A cloud sign-in never reaches the home volume, and ends when the workspace stops.
  • One running workspace: on a start build only, the template reads the owner's workspaces from Coder on the loopback with their session token (hashicorp/http 3.6.2, locked), failing closed unless Coder answers 200. A precondition refuses the start while another holds the limit: a start pending, starting, running or cancelling, or a failed stop. A failed start does not hold it.
  • The launcher and the site: a start refused for another lab names that lab, frames its Stop, and starts this one once the other stops. A limited state reads "Stop your other lab to continue", and the ten-minute cap does not run during that wait. The site's lab panel says one workspace runs at a time and a sign-in lasts until the workspace stops.
  • Which builds it binds: builds of a template version that carries it. Coder Community's dashboard Start rebuilds a workspace's own version, so the runbook has a one-time owner step after the first publish.

  • Tests: template.test.mjs (28) and lab-host-launcher.test.mjs, and the coder CI job now runs fmt, init and validate on every template.

  • Audit finding PLAT-6: Rebuild inventory outside Terraform, custom roles bootstrap, secret escrow & origin-secret rotation (#966).

  • Parameterized Azure custom role definitions: created scripts/bootstrap-roles.ps1 and scripts/bootstrap-roles.sh allowing subscription ID injection (-SubscriptionId, -ManagementSubscriptionId) across all five custom roles (keyvault-secret-writer, function-config-refresh, function-settings-reader, static-web-app-deployer, alert-rule-reader), replacing hardcoded subscription GUIDs.
  • Added unit test assertion in scripts/terraform-role-definitions.test.mjs verifying that bootstrap scripts cover all registered custom role definitions.
  • Documented authoritative Where to Re-obtain paths in docs/standards/required-inputs.md §4.6 covering all 30 Key Vault secrets in functions/src/lib/secret-catalog.js (including portal URLs, CLI generation commands, Bitwarden item names, and auto-renewal tools). Updated §4.6b to document the five custom roles and bootstrap scripts.
  • Authored docs/runbooks/origin-secret-rotation.md detailing the 3-way lock between Key Vault CF-ORIGIN-SECRET, HCP Terraform cloudflare_origin_secret, and Cloudflare Transform Rule x-hcw-origin-secret, complete with zero-downtime execution order, verification curls, and rollback steps.
  • Authored docs/runbooks/cold-rebuild.md providing a single ordered, copy-pasteable disaster recovery guide from bare accounts to full production estate operation within the 8-hour RTO objective.

  • Updated mkdocs.yml navigation under Runbooks:.

  • Secrets: Google API keys and AWS price-list IAM access key auto-renewal (#1028).

  • Implemented automated Google Cloud API key renewal in functions/src/lib/credentials/google-roll.js via Google Cloud API Keys API v2 (apikeys.googleapis.com) with verify-before-delete semantics for GEMINI-API-KEY, YOUTUBE-API-KEY, and GCP-BILLING-API-KEY.
  • Implemented automated AWS IAM access key renewal in functions/src/lib/credentials/aws-roll.js using pure SigV4 signing (node:crypto) with verify-before-delete semantics for AWS-ACCESS-KEY-ID and AWS-SECRET-ACCESS-KEY scoped to pricing:GetProducts.
  • Enforced verify-before-delete: newly minted credentials are functionally probed against their respective cloud services (Gemini models list, YouTube videos list, GCP Billing services list, AWS Pricing GetProducts) AND committed to Azure Key Vault before old credentials are deleted. If a probe fails, the newly minted key is deleted and the existing key remains active and undisturbed.
  • Added CLI orchestrator scripts/roll-cloud-credentials.mjs supporting --target <gemini|youtube|billing|google-all|aws|all>, --dry-run, and --check modes with zero CLI argument secret leakage.
  • Updated functions/src/lib/credentials/register.js to set renewal: 'automation' with rotation commands for all five credentials.

  • Added 65 new unit tests across google-roll.test.js, aws-roll.test.js, roll-cloud-credentials.test.mjs, and register.test.js.

  • Coder: only the lab-learners team signs in, and no sign-in creates an account (#954, LAB-8). Owner decision 2026-10-16.

  • The allowlist: the coder role renders CODER_OAUTH2_GITHUB_ALLOWED_TEAMS=HybridCloudWorks/lab-learners, and coder_oauth2_github_allow_signups is false.
  • The checks: tasks/signin_policy.yml refuses an empty team list, or a team that is not <organisation>/<team-slug> in an allowed organisation spelled the same way, since Coder v2.38.0 checks a team only inside an organisation the learner already passed.
  • Who is affected: existing accounts, the owner's included, keep working, and the team is checked at each GitHub sign-in. A learner's first sign-in matches only an account the owner made with their primary verified GitHub email; the runbook's new "Coder sign-in: adding and removing a learner" has the lines, and a suspended learner's workspaces stop within a minute. Sign-ups off is the normal setting now, not a kill switch: group_vars, ADR 0032 and docs/standards/required-inputs.md say so.
  • No account, no route: sign-ups off does not cover a deployment's first account, which v2.38.0 lets through from GitHub or /setup and makes the Owner. Before Coder starts, the role counts the accounts in its database. With none, it takes Coder's Caddy route down and reloads Caddy then and there. Once Coder is up, a 404 from /api/v2/users/first fails the play. The runbook's "A Coder with no account yet" has the owner's measured recovery, and scripts/lab-host-coder-first-user.test.mjs holds it.
  • Runbook token lines: they strip the carriage return with tr -d \\r; one backslash had deleted every r from the token.

  • Tests: roles/coder/tests/coder-signin.test.sh (23 checks) in the ansible-lint (lab-host) job.

  • Secrets: Cloudflare API token auto-renewal via roll-token API (#1027).

  • Implemented automated Cloudflare API token renewal in functions/src/lib/credentials/cloudflare-roll.js and CLI orchestrator scripts/roll-cloudflare-tokens.mjs.
  • Supports all three estate targets: tfc-azure-cloudflare-api-token (hcw-azure HCP Terraform workspace), tfc-lab-cloudflare-api-token (hcw-lab HCP Terraform workspace), and lab-caddy-cloudflare-api-token (vault_cloudflare_api_token in lab host Ansible Vault).
  • Enforces verify-before-apply semantics: verifies the newly generated token secret against Cloudflare's /user/tokens/verify endpoint before committing to HCP Terraform or the lab host vault.
  • Updates HCP Terraform sensitive workspace variables via the JSON:API endpoint (application/vnd.api+json).
  • Updates the lab host Ansible Vault via hcw-vault-set, piping secrets strictly to standard input.
  • Updated functions/src/lib/credentials/register.js to set renewal: 'automation' for the three Cloudflare tokens.

  • Added 33 unit tests across functions/src/lib/credentials/cloudflare-roll.test.js, functions/src/lib/credentials/register.test.js, and scripts/roll-cloudflare-tokens.test.mjs.

  • Lab host: Azure Arc SSH as the owner's way in; port 22 stays open (#953, LAB-7 stage 2). Arc SSH carries the session through an Azure Relay connection the host's Arc agent dials out to, so sshd needs no inbound port. sshd still asks for the pinned key. Port 22 closes in a later change, once the owner has connected this way and through the Web Console.

  • Azure: infra/lab-hybrid.tf declares the Arc machine's default connectivity endpoint and its SSH service configuration, port 22 only (Microsoft.HybridConnectivity 2024-12-01). They are gated on the machine existing, grant no role, and are declared in the plan check.
  • The host: the arc role turns on the agent's incoming connections for port 22 alone. It fails closed unless the extension allowlist is still the Azure Monitor Agent alone, and the nightly drift check sees each setting. arc_incoming_ssh_enabled: false restores LAB-6's state.
  • The desktop: Connect-Lab.ps1 -Arc checks az, the ssh extension, the sign-in and the service configuration before writing anything. Its ProxyCommand helper gets fresh relay credentials on every connection, deletes them from disk before the connection starts, and passes them to the proxy in its environment alone. HostKeyAlias keeps the host key already accepted. Without -Arc the direct block returns.
  • Docs: ADR 0032's amendment of 2026-10-16, the runbook's "Connect through Azure Arc" and the architecture page.
  • Tests: scripts/lab-host-arc-ssh.test.mjs holds the port, names and api-version together across infra, the role and the script, and Connect-Lab.Tests.ps1 runs the generated helper end to end against a stand-in az and proxy.
  • Lab host SSH: one user, one pinned key, public key only, behind a lockout guard (#953, LAB-7 stage 1). Port 22 stays open in this stage; Arc SSH and closing 22 follow.
  • The key: group_vars pins hardening_admin_authorized_keys to the owner's workstation key, and the hardening role writes hcwadmin's authorized_keys from that list alone. Root's is emptied, and the copy-from-root fallback is gone.
  • The guard: before writing, the role stops on an empty list, an entry that is not one public key, or a list without the key of the most recent accepted login (read from 14 days of the journal).
  • sshd: a drop-in adds AllowUsers hcwadmin, AuthenticationMethods publickey, no agent, stream-local or tunnel forwarding (local TCP forwards stay), LoginGraceTime 30, PerSourcePenaltyExemptList for the loopback that Arc SSH will arrive from, and LogLevel VERBOSE. It is validated, read back with sshd -T and put back on failure. sshd is reloaded, never restarted.
  • Order: a pending-reload marker keeps root's keys until sshd has loaded the drop-in, even after an interrupted run.
  • Connect-Lab.ps1: stops writing root's authorized_keys, and prints the group_vars line that pins a key.
  • Accepted risk: passwordless sudo stays, with its compensating controls in the role README; hPanel's Web Console is the break-glass.
  • Tests: admin-keys.test.sh (33 checks) and sshd-apply.test.sh (21, as root) in the ansible-lint (lab-host) job, and lab-host-ssh-hardening.test.mjs. lab-host-ssh-login-alert.test.mjs now holds the pinned keys to the alert's fingerprints unconditionally.
  • Add-on images: SLSA build provenance attestation verification before pull (#1069, part 1). The addons Ansible role (lab-host/ansible/roles/addons) now verifies the SLSA build provenance attestation of each deployed add-on image against Sigstore's public transparency log and the expected GitHub Actions workflow identity (attestation_identity_regexp) using cosign verify-attestation before calling docker_image_pull. A missing attestation pattern, mismatched identity, or unreachable transparency log fails the run closed, preventing untrusted or unauthenticated images from being deployed. Sigstore cosign (v3.1.3) is pinned and SHA-256 verified in bootstrap.sh and group_vars/all.yml and tracked upstream via scripts/lab-pins-upstream.mjs.
  • Visitor lab sessions: the broker sends Coder nothing it cannot vouch for, and follows no redirect (#1403, the items found while the sweep timer was built). Every call functions/src/lib/labs/visitor-session.js makes to Coder is made with redirect: 'error': the broker token travels in a Coder-Session-Token header that fetch keeps on a redirect to another site, so a Coder address that answered with one would have sent the token wherever it pointed, and the sweep now runs unattended with it. A value read from a session record is written into a Coder path only when it has the shape of what it claims to be: a Coder id must be a UUID (isCoderId), a visitor's user name v- and twelve hex characters (isVisitorUsername), and a workspace name one of the labs' (isLabWorkspaceName); the old rule, any one path segment that was not ., .. or me, would have sent a stored value that was another person's name. Before a user is deleted it is read back by its id (GET /api/v2/users/{id}, the page read 2026-11-03), and the delete is sent only when Coder's answer is that id under a visitor's name; Coder having no such user counts as gone, like a delete that found nothing, and is said in the log the same way, and any other answer leaves the user for the next try. The test fixtures' Coder ids are UUIDs now (functions/test/coder-ids.js makes the same one from a label every time), and tests break each rule on purpose: a plausible name as an id, another user answered for the read-back, a name that is not a visitor's, a redirect, a workspace name that is not a lab's.
  • The Docker Sandboxes template gets the same Azure CLI Python fixes as the lab image, and is now built and scanned (#1473). lab-image/sandbox-template/Dockerfile is the recipe the Docker guide page tells a learner to build for "Run an agent against your landing zone"; it installs the same Azure CLI release as the lab image's full target, but until now without the package replacements in AZURE_CLI_PYTHON_OVERRIDES (lab-image/versions.env), so a learner's build still carried the certifi, PyJWT and urllib3 versions the weekly scan flagged, and nothing built or scanned it. The recipe now does what the full target does, in the one RUN that installs the CLI: the downgrade guard, the wheels downloaded with the CLI's own pip in hash-checking mode (it has no fetch stage), an offline install and pip check. publish-lab-image.yml builds it on every pull request that touches lab-image/ and smoke-tests it offline (the replaced versions read back, az and terraform at their pins, the agent user), and lab-supply-chain.yml builds it from main each week and scans it with Trivy beside the published images, saying in its issue when the build itself failed. scripts/lab-image-azure-cli-overrides.test.mjs holds the recipe to the same order and the workflow to the build, and lab-pin-pr-workflows.test.mjs holds the scan.
  • A day on which the Log Analytics workspace hits its cap is a named alert, alert-logs-overquota, and the personal-data access trail is kept in full (#1530). alert-logs-capacity fired at 80% every afternoon that week and was read as weather; nothing said that from 20:33 until 08:00 UTC every log-based alert in the estate was blind. The new rule reads the OverQuota line Azure writes to _LogOperation when collection stops (Microsoft's own daily-cap query), fires from inside a capped workspace because that table is not billable, is stateful across the 08:00 UTC reset, and pages at severity 1 through the existing action group. With the request-log filter above, every read and write on ambassador, doc_vault and brain_records is logged for 90 days with the caller's identity, which the data-subject runbook now points at.
  • The Function App's Cosmos data role gains a grant on the hcw database alone, the first of two steps that narrow it from the whole account (#1469). The backend reads Cosmos through one data-plane role (Cosmos DB Built-in Data Contributor) granted on the whole account. Cosmos can grant the same role on one database instead (scope <account id>/dbs/<database>), and at that scope the SDK can still read the database, list its containers and work inside them, which is everything the app, the six change-feed functions, their leases container and the nightly export do. The move takes two apply runs so the app never has a moment with no grant: this step adds azurerm_cosmosdb_sql_role_assignment.func_cosmos_db["hcw"] in infra/cosmos.tf beside the account-wide func_cosmos, whose address is deliberately untouched (a count or moved on the one grant the site runs on would be an address change the plan check reports, and a wrong move is the gap this avoids); a later pull request deletes the account-wide block. Only hcw gets a grant: hcw_code_brain left with #1480. The plan check declares the one create, infra/tests/guards.tftest.hcl holds the new grant to the database scope and the Data Contributor role while the account grant keeps its scope, scripts/assert-expected-plan.test.mjs holds the declarations to the database map, the deployment runbook carries the owner's steps and checks for both apply runs, and ADR 0036 records the narrowing as in progress.
  • The custom role definitions no longer carry the owner's subscription id, and the redaction gate now reads infra/, the workflows and the operator scripts (#1540). Five of the role files under infra/roles/ named a real subscription in their assignableScopes, in a public repository whose rule is that a real tenant, subscription or application id never goes in a tracked file. The files now carry the placeholder form the sixth file already used (...0001 for the workload subscription, ...0002 for the management one), and scripts/bootstrap-roles.sh and .ps1, which always replaced that field at create time, now also find both real ids themselves: the workload subscription from az account show and the management one from az account list by its name, instead of carrying the management id as a default in the script. Both scripts refuse a placeholder or a non-GUID and end with a read-back table of every HCW role and its real scopes, which is the evidence for the owner's re-run. scripts/docs/check_redaction.py only ever read docs/ and the root documents, which is how this went unnoticed; it now also scans infra/roles/*.json, infra/**/*.tf, infra/**/*.tfvars*, .github/workflows/*.yml and the top-level scripts/*.ps1 and *.sh for non-placeholder GUIDs, with the public constants those files legitimately hold (four Microsoft built-in role and policy ids, one Cloudflare DNS ownership token, the two synthetic browser-test ids) on its allow-list with their reasons, and repository-policy.yml runs it on every pull request (the docs workflow only runs when a docs path changes). scripts/docs-redaction.test.mjs fails on a role file with a real-looking id, on each new scan path, and if that step leaves the policy workflow.
  • The Entra scripts look the API's client id up instead of carrying it, and the redaction gate reads the rest of the tree (#1540). scripts/cutover/01-entra-api.ps1, 02-entra-spa-client.ps1, 03-entra-dev-client.ps1, scripts/rollback/restore-admin-signin.ps1 and scripts/lab/Register-LabAgent.ps1 carried the HCWSite API's real application id as a parameter default, the rollback script the tenant id too, and their tests and two frontend tests held the same values as fixtures, one folder below where the gate read. Each script now takes -ApiAppId, then HCW_API_APP_ID, then the one registration named HCWSite API in the signed-in tenant (scripts/lib/Resolve-EntraApiAppId.ps1; the lab script's own Resolve-LabApiAppId, tested with Pester), stopping when none or several match; the rollback script compares -TenantId or HCW_TENANT_ID only when one is given, and otherwise the lookup is the tenant check. The gate now reads every .ps1 and .sh under scripts/, lab-image/**/*.sh, lab-host/** (yml, yaml, sh, j2), functions/src/**/*.js and frontend/src/**/*.{js,jsx}. Every GUID those paths held was classified: the real ids went; the made-up fixture ids were rewritten to a first group of one repeated digit (99999999-...), which the gate now recognises along with the placeholder and the browser tests' e2e00000-... family; the Defender WSL walkthrough's and simulator's sample device ids became obviously fake; and the four Azure Policy built-in definition ids estate.js excludes, the two UUIDv5 namespaces the draft ids are hashed under, the three UUIDv5 values their tests assert, and three public URL names (Amazon Music's podcast listing, two ElevenLabs preview objects) went on the allow-list with their reasons. The four gitleaks fingerprints for the removed defaults left .gitleaksignore, and the rollback runbook, the lab-host runbook and ADR 0006 say where the id comes from now.
  • The Function App's account-wide Cosmos grant is removed; the identity keeps the hcw database grant alone, the second of the two steps (#1469). Step 1 (#1531) added azurerm_cosmosdb_sql_role_assignment.func_cosmos_db["hcw"] beside the account-wide func_cosmos and applied on 2026-11-04, after which both grants were listed on the identity and the site answered as before. This step deletes the func_cosmos block from infra/cosmos.tf and declares the one destroy in scripts/assert-expected-plan.mjs, so the plan is that destroy beside the permanent diff and nothing else. Before the block went, the code was read for anything that needs the account scope: nothing lists the account's databases (functions/src/lib/cosmos-client.js opens COSMOS_DATABASE by name, the six change-feed functions and their leases container bind to that same database, and the nightly export reads containers through the same client), which is the only thing the account scope adds per Microsoft's data-plane security reference. infra/tests/guards.tftest.hcl now refuses an account-scoped grant for the identity instead of requiring one, scripts/assert-expected-plan.test.mjs holds that the block is gone and its delete declared, and that no Cosmos role assignment in infra/ is scoped to the account id, the deployment runbook marks Step 2 as the current step with the owner's checks and the rollback (revert the pull request with the restored grant's create declared, so its plan is one create the plan check accepts), and ADR 0036 records the narrowing as complete once that apply runs.

Fixed

  • The weekly lab pin bot no longer reports the Python base image as behind when it is current (#1426). The --bump image-base check decided whether the Python line tag (python:3.14-slim-trixie) and its patch tag (python:3.14.10-slim-trixie) agree by comparing their whole image lists (an index, one image per platform). The line tag still lists a riscv64 image on CPython 3.14.9, so the two lists differed and every run posted a "have not settled" note, although the pin was the patch tag's own digest and the linux/amd64 image (the only one the lab image is built for) was the same in both. scripts/lib/lab-pin-bumps.mjs now compares the two tags' linux/amd64 images, says nothing when the pin already is the patch tag's digest, still pins the patch tag's index digest when it moves, and stops with a note when either tag lists no linux/amd64 image. Found by the review of #1466; four cases in lab-pin-bumps.test.mjs.
  • Visitor lab sessions: the follow-ups from two reviews are fixed or decided (#1400, #1403). All in functions/src/lib/labs/visitor-session.js, with LABS_VISITOR_SESSIONS_ENABLED still off. A visitor's daily start is now spent only when a new lab is really made: pressing start again while their session runs hands it back for free, a start refused because the lab is full costs nothing, and a start that fails in Coder gives its start back (giveBackDailyCap in functions/src/lib/daily-cap.js, a decrement that cannot go below zero). One visitor may ask the human verification check at most 10 times and read the status or stop at most 60 times in ten minutes, per Function App instance, before anything is read. The check's failure detail goes to the log, and the visitor gets the lab submission's plain sentence. The three routes refuse a request that did not come through our Cloudflare whatever NODE_ENV says. A start writes a provisioning record before its first call to Coder, so a call that never answers, or a session that cannot be saved after Coder made it, still leaves a record the sweep cleans up from (it looks a user up by the name the broker chose, GET /api/v2/users/{user}). Waiting sessions whose workspace is not confirmed as deleted count against the concurrent limit, drop the lab address, and record whether each delete was accepted (ok) or answered 404 (gone, also logged). After three refusals of the user delete in a row the workspace delete is asked for again, and a given-up clean-up gets another round an hour later, up to 24. The stop answers a cleanup field (done, pending, failed) beside its old body. After review, starts that arrive at the same moment can no longer get past the limits: a per-visitor start lock and one slot document per allowed lab, each taken with a create the database refuses when it exists, and slots that repair themselves after a crash. The list of labs is read again after a sweep the start waited for. A daily start is given back exactly once, with one refund document per session (visitor-refund:<id>, a second write of it is refused) that raises the visitor's limit for the day, instead of a decrement; a give-back that fails is tried again by the sweep before the record is saved as over. After a second review, a start also reads the list again once it holds its lock (so a session finished meanwhile is handed back), counts sessions saved before slots existed (past their hour included), and looks a user up after a Coder 408. After a third review, every holder owns one of the slots: a holding session that no slot names is given one before a start takes its own, and slot documents no longer age out. A user Coder plainly refused is never looked up by name. A clean-up given up for good stops counting against the limit. The pane shows the server's sentence for a refused start or status read, and reads the status again after 409 VISITOR_START_IN_PROGRESS; .azure/api-surface.json describes all of it. Coder's response text no longer reaches error messages or logs. The tests now run the module's real SQL (runCosmosQuery in functions/test/cosmos-predicate.js) and the test store can age records out by their ttl. Still the owner's: whether a session should belong to a secret rather than an address (#1400, item 4), and the broker token's Coder permissions (#1403, item 4). Documentation: the lab host runbook, "Visitor lab sessions". Nothing was run against Azure, Cosmos or Coder.
  • The weekly lab pin bot now says when it could not read a publisher, and tries a failed read again first (#1447). A product whose release list could not be read used to drop out of the host pull request with no word, so a passing hiccup delayed its update by a week unnoticed. It now appears under "Not bumped" in the pull request body and the notes file (which also feeds the weekly issue), with the address and the status it got, so "no newer release" and "could not read" look different. Every read the bot makes (scripts/lab-pins-upstream.mjs and scripts/lib/lab-pin-bumps.mjs) that fails with a server error (5xx), a 406 or no answer at all is tried twice more, after pauses of 2 and 5 seconds, with the same address and headers, before it counts as failed. "No answer" now includes a publisher that has not replied within 30 seconds (each try has its own limit, and one read gives up after 60 seconds in all, so a stalled publisher can no longer hold the job for Node's five-minute default) and a connection that drops while the reply is still arriving. Every such failure, a reply that is not JSON, and a reply with no stable version in it now name the address. When the unreadable publisher is the only host item that week, no pull request is opened (there is no edit to put in one), and the weekly issue carries the note. Tests cover a publisher that fails once and is bumped, and one that keeps failing and becomes a note with no edit.
  • AI Engine model refresh, routing display and cost reporting are corrected (#1437). Provider lists refresh concurrently under one deadline per provider, incomplete pagination preserves the previous list, and routing reloads after catalogue changes. Planned tasks are distinguished from unavailable active tasks; unknown prices remain unknown, small costs remain visible, and Gemini 3.8 Flash TTS is the reviewed multi-speaker recommendation, with structured speaker annotations, verified promotional price expiry, and 25 output audio tokens per second; explicit saved selections remain intact. See model review.

  • Ambassador links stay out of the public footer, and readiness respects qualification periods (#1437). The hard-coded GitKraken Ambassador footer credit is removed. Application dates take precedence over recurring quarter/year windows in the dashboard and workspace; program details use the current upkeep period. Footer links wrap, program cards have more room, and ambassador dialogs and forms fit mobile screens.

  • A visitor lab session is saved as over only once Coder has confirmed the clean-up, and one Coder has not confirmed is kept and retried, not forgotten (#1388). When a visitor pressed End session, or the sweep found a session past its hour, functions/src/lib/labs/visitor-session.js asked Coder to delete the workspace and the user, threw away whether Coder accepted, and saved the record as ended or expired anyway; nothing looked at it again, so a delete Coder refused was never retried. Coder deletes a workspace with a job that finishes later and refuses (HTTP 417) to delete a user who still has a workspace, so the user delete straight after an accepted workspace delete is refused for most sessions: a workspace that was never deleted keeps one of the host's 5 workspace slots, and the user stays behind. Found by two read-only reviews of the #1383 fix, from the code and the Coder source and not run against a live Coder; the refusal of the user delete is not confirmed on the live host. Now a session is first claimed with a conditional write (active to ending, with a lease), so only one request calls Coder (two stops at the same moment make one set of Coder calls). It is saved as ended or expired only when both deletes are confirmed (accepted, or 404 or 410 because already gone). Otherwise it is kept as cleanup_pending, with which half is done (workspaceGone, userGone) so the workspace delete is not asked for twice, and tried again after 2, 4, 6 and 8 minutes by the sweep or by a later stop, then marked cleanup_failed after the fifth try, which nothing retries and which keeps its Coder ids for 7 days for a person to finish. A claim whose holder died is taken over by the sweep (or a later stop) after 5 minutes. The sweep (reapExpiredVisitorSessions) now also lists the waiting sessions, returns { reaped, finished, pending, failed }, takes up at most 10 sessions a time, and runs from the status and start routes at most once a minute per worker. It runs only when someone visits (nothing else triggers it, and the status route answers before it sweeps while the switch is off), so a waiting record is kept for 7 days instead of the usual 2 hours; a timer that runs the sweep on its own needs the owner's decision and is in #1403. A waiting session does not count as active, so the visitor is not shown it and may start another. A session is also no longer saved as over when the broker is not configured but the session has something in Coder to delete. A start that fails after Coder made the visitor's user (the workspace call timed out, which cancels only our wait while Coder can go on and make the workspace, or was refused) is now kept as a cleanup_pending note before the 502 is answered: the user's id and name and the workspace's name, no workspace id, no visitor hash (so no visitor can reach it), and a first try after 2 minutes. For a record with no workspace id the sweep looks the workspace up by owner and name (GET /api/v2/users/{user}/workspace/{workspacename}, Coder's API reference, read 2026-10-26) and deletes it only when its name is the expected one and its owner is that user; one it does not find counts as gone for that try only, and the user delete, which Coder refuses while the user still has a workspace, is the confirmation. Every Coder call now has a timeout, 10 seconds for a delete and 30 for the calls that provision a session, so a hung Coder cannot hold a request open; ids are written into Coder's URL paths as one piece each. Logs say only what Coder answered for each half (for example user HTTP 417) and which try it was, never an id. Tests cover Coder refusing (500, 417, timeout, network failure), a later stop and the sweep each finishing it, 404 and 410 counting as done, the bound and the growing wait, the lease takeover, two stops and two sweeps at the same moment, a Coder that never answers, and the throttle; checked by ten changes to the code, the first of them putting the old behaviour back (each makes at least one test fail). .azure/api-surface.json, ADR 0032's amendment and docs/runbooks/labs-host.md (the statuses, a query for finding stuck sessions, numbered steps for finishing one by hand, rollback) describe it. Two independent reviews before the pull request found no wrong delete, no missed ownership check and no harmful second delete; what they found beyond this change (a delete build that fails after being accepted ends as cleanup_failed, a user create that got no answer leaves no note because its id is unknown, the 404 reading and the token fallback, and more) is in #1403. Built on the #1387 change. Not yet live: the owner deploys the functions. Keep LABS_VISITOR_SESSIONS_ENABLED off until then.

  • Visitor lab sessions now tell visitors apart: each visitor has their own daily limit and can read and end only their own session (#1387). functions/src/lib/labs/visitor-session.js asked the client-identity helper for a fromRequest method that createClientIdentity() has never had (it has anonymousKey(request) and trustedClientIp(request)), so in production the lookup was skipped and every visitor was filed under the text unidentified. With LABS_VISITOR_SESSIONS_ENABLED on, that meant one shared daily limit of 2 for the whole site; GET public/labs/visitor-session answering any caller with the id and address of the live session (it has no Turnstile, origin or rate check); DELETE ?sessionId= then ending that session, and a new visitor being handed the first visitor's session; and Turnstile (Cloudflare's bot check) never being told the caller's address. Found by a read-only review of the #1383 fix, from the code and not run against a live deployment; whether the setting is on in production was not checked. The three routes now read the caller the way lib/labs/public-submit.js does: identity.anonymousKey(request).key (a hash of the address Cloudflare reports) files the session and the daily limit, and identity.trustedClientIp(request) goes to Turnstile. A request that cannot be shown to have come through our Cloudflare (no origin secret), or that carries the secret but no usable client address (which the helper would otherwise hash into one shared bucket, the same fault under a new name), is answered 403 FORBIDDEN before the store, Turnstile or Coder is used, and the log line says only that a request was refused. Only the helper's own refusal (its error now carries code: 'ORIGIN_NOT_VERIFIED') is answered that way; any other error, such as a TypeError from a method the helper lacks, is thrown on and shows as a server error. The status route describes only the caller's own session. The stop route ends a session only for the caller it belongs to, and answers any other caller 404 Session not found, the same words as for an id that never existed (or one that is already over), so holding an id is no longer enough. People behind one shared network address count as one visitor, as they do for every other anonymous limit here. A session already stored under unidentified can be neither read nor ended by anyone and is expired by the sweep when its hour is up. createVisitorSessionHandlers now refuses, when it is built, an identity without anonymousKey and trustedClientIp (assertClientIdentity, new in lib/auth/client-identity.js), where it used to carry on with no identity at all. So the same mistake cannot come back unseen: functions/src/lib/auth/client-identity-calls.test.js reads all of functions/src and fails any property read from an identity that the real helper does not have (the list of names is taken from the real object, not typed out), and any call of anonymousKey, trustedClientIp, viaCloudflare or rateLimitKey made with no request, naming the file and line; it also fails a test that writes its own identity = { ... } stand-in with a method the real helper lacks. The visitor tests, and the lab-start counter's tests in visitor-session.counts.test.js (which had copied the same stand-in), now build the real createClientIdentity() and send requests with the origin secret and an address, where they used to hand the code a stand-in that had the fromRequest method; new cases cover two callers (different keys, a daily limit each, Turnstile given each caller's address, neither able to read or end the other's session, nor to learn that a finished one existed), a request without the origin secret or without a client address, and a run of all three routes on an identity that throws on any property the real helper lacks, whatever the code calls it. Checked by putting the old lookup back (the identity guard and five visitor tests fail), taking the ownership check out of the stop route (four fail), letting an unverified request through as unidentified (four fail) and not giving Turnstile the address (one fails). .azure/api-surface.json and the visitor sessions section of docs/runbooks/labs-host.md describe the new answers. Not yet live: the owner deploys the functions. Keep LABS_VISITOR_SESSIONS_ENABLED off until this is deployed.
  • Ending a visitor lab session, and the sweep that expires old ones, now really save the session's record (#1383). functions/src/lib/labs/visitor-session.js saved a changed session with replaceDocIfMatch('lab_jobs', session.id, {...}, session._etag): an id, the document, then an ETag (the version stamp the database checks, so a write made from an out-of-date copy is refused). The real function in cosmos-client.js takes the whole document with its _etag inside it, then an options object, so both saves failed. The sweep (reapExpiredVisitorSessions) swallowed the error: an expired session stayed active and was never counted as reaped. The stop route (DELETE public/labs/visitor-session) deleted the Coder workspace and user first and then threw, so the visitor's "End session" ended in an error (the Functions host answers an uncaught error with HTTP 500) for a workspace that was already gone, and the record kept saying active, so the session still counted against the visitor limit (default 2) and starting again would hand back the dead workspace's address until the session's hour ran out. Not checked against a live deployment: this is from the code, the function's signature and tests that fail on the old calls. Both saves now go through one helper, finishSession, which calls replaceDocIfMatch('lab_jobs', { ...record, status, endedAt }, { partitionKey }) and, when the save loses a race (a second stop, or the sweep, wrote first: a 412, or a 404 because the record aged out), reads the record again: already ended or expired means someone else finished it and the loser has nothing to do; still active means something else changed it and the save is tried again on the new version, up to three times, then the stop answers 409 and leaves the record active. A second stop of a session that is already over now answers 200 { ok: true, status: 'ended' } without calling Coder again. The sweep now returns { reaped, failed }; a save that fails for any reason other than a lost race, a record that keeps changing, a sweep that cannot list the expired sessions, and a Coder clean-up call Coder did not accept are each logged (by kind only: no session id, no error text, no Coder id) where the request has a log, which is the start route; the status route's background sweep still has none, because it runs after its answer is sent. Not changed here, and now visible: a session is saved as over even when Coder refused a clean-up call, and then it is not retried (before this fix the record stayed active, so later requests re-ran the deletes by accident); making the final status depend on a confirmed clean-up is #1388. Also found by the review and filed separately, not fixed here: the code asks the client-identity helper for a fromRequest method that does not exist, so every visitor is treated as the same person (#1387); keep LABS_VISITOR_SESSIONS_ENABLED off until it is fixed. So the same mistake cannot come back unseen: functions/src/lib/cosmos-call-shape.js is the one definition of the shape replaceDocIfMatch(containerName, document, options) takes (a document object with a string id and the _etag it was read with; options only { partitionKey }); the real replaceDocIfMatch calls it first and refuses any other shape with a TypeError that says what to pass and names no value, before the SDK is touched, and the shared test store functions/test/memory-store.js calls the same check, where it used to accept any argument order (the visitor tests' own store did, which is why they passed). I read the 37 calls elsewhere in functions/src; each passes a stored document (so one with its _etag) and { partitionKey } or nothing, so none is affected. functions/src/lib/replace-doc-if-match-calls.test.js reads the source text and fails any replaceDocIfMatch call with more than three arguments, an id as its second argument or an ETag as its third, naming the file and line, so a call that no test reaches is still checked (it flags the two old calls, at lines 256 and 615 of the old file). The visitor tests now run on the shared store, with new cases for the save itself, the arguments the store receives, a second stop, the sweep winning or losing the race, a record that keeps changing, a record that aged out, two stops at the same moment, two sweeps at the same moment, and a store that cannot save. Checked by putting the old calls back (the visitor tests that save a session and the source-text test fail) and by taking the new check out of replaceDocIfMatch (the five tests that call it with a wrong shape fail). .azure/api-surface.json describes the stop route's answers. Not yet live: the owner deploys the functions.
  • Add-on follow-up to #1351: the status-route inventory names Cloud Crawler's setting, and the labs-host page stops saying the add-ons are not on the host. CodeRabbit's two findings on #1351 were real and the PR merged before they were answered. .azure/api-surface.json now lists ADDON_CLOUD_CRAWLER_URL among the settings the status route reads and says it is not provisioned while the row is coming. In docs/architecture/labs-host.md, the Tool add-ons row says that "not yet on the host" was the state at the 2026-10-08 review and that all three add-ons answered on the live status route on 2026-10-24, and the Applied state row for ADR 0035 now records the merge dates (#1043 on 2026-10-09, #1089 on 2026-10-12) instead of "not merged yet". No host run date or commit is written, because the repository does not hold them; the host's own record is still to be read (#1355). CodeRabbit's review of this pull request added two more corrections to the same page: a stale [REVIEW REQUIRED] marker on the ADR 0035 amendment is replaced by the owner's confirmation of 2026-10-10 (#1044, ADR 0035 decision 8), and two statements that said more than the evidence are reworded: the status route showing an add-on answering is not the host having applied the commit, and Cloud Crawler having no group_vars row is not a check of the host.

  • Interviews: the header link moved to the Azure landing page, interview pages keep the Azure menu, and a route check that failed now passes (follow-up to #1337). CodeRabbit's two findings on #1337 were real, and a browser run found a third problem beside them. (1) The top menu is a fixed row of nine 100-pixel slots whose divider must sit in the same place on every provider; the tenth link #1337 added for Azure overflowed it at normal laptop widths and moved the divider. It was invisible on the live site because no interview exists yet, so it would have appeared the day the first one was added. The link is now an Interviews section on the Azure landing page (/azure), shown only once an interview exists, and the menu stays at nine links. (2) The header finds the provider from the URL alone, and interviews was not a known page type, so on /azure/interviews the Azure menu was replaced by the generic provider list. It is now a page type in routeFactory.ts, as sandboxes is for Docker, and the Azure menu stays on every interview page. (3) npm run validate:routes failed on two hard-coded /azure/interviews/... paths in a test; they now come from interviewPath. That validator is not run by CI, which is why nothing flagged it. Checked in a real browser at 1280 and 1100 pixels: nine links on every Azure page, interview pages included, the divider at the same position on all of them, no overflow, no errors.

  • The Linkie Hub follows Linkie's rename of profiles to pages. An October 2026 notice from Linkie to API users says GET /api/v1/profiles is now GET /api/v1/pages, /profiles/{id}/posts is /pages/{id}/posts, the analytics parameter link_in_bio_id is page_id, and the key scopes profiles:read and profiles:write are pages:read and pages:write. Linkie's own API reference, read on 2026-10-23, shows the new endpoints, the data.pages list and the page_id parameter; the scope names come from the notice alone, because the reference does not list them. The hub's Links, Analytics and Settings tabs and the Connections test used the old names, and linkieProxy's allow-list only let the old paths through, so they would have stopped working as soon as Linkie retired them. The allow-list, the frontend paths, the list reader (data.pages) and the analytics parameter now use the new names; the old paths are no longer allowed. The allow-list moved into functions/src/lib/integrations/linkie-allowlist.js so a frontend test can run every path the hub builds through the real matcher, and a rename that moves one side and not the other fails there (ci.yml's frontend filter lists the two functions files that test reads, as the repository's CI-filter test requires). The API key may need creating again: a key that only has the old scopes answers 403 "Insufficient scope". Linkie's key form has three separate permissions, and the hub needs Linkie pages Read Only, Automated posts Read & Write and Analytics Read Only (a first draft of these docs asked for pages Read & Write alone, which would have passed the connection test and still refused posts and traffic; CodeRabbit caught it). docs/standards/required-inputs.md and the credential rotation schedule say how, with the exact Playground URL. Function names in the hub still say "profile"; a profile there is what Linkie now calls a page.

  • A healthy Functions deploy no longer goes red because the temporary firewall rule reached the origin one front end at a time. The deploy opens an allow rule for the CI runner's address, and App Service applies it to its front ends gradually. The workflow waits for /api/health to answer, but a later request can still land on a front end without the rule and be refused with the platform's own 403 ("Ip Forbidden"). That happened on 2026-10-23 (run 39997538990): health 200, the public content list 403, the detail read 200, and the deploy failed after everything had been deployed and registered. scripts/smoke-deployed.mjs now sends every request through originFetch, which waits 5 seconds and tries again when, and only when, the answer is a 403 carrying the platform's x-ms-forbidden-ip header, for at most 90 seconds across the whole run. An app-level 403 (the admin guards, the CORS refusal) is never retried, because those checks expect it, and a firewall refusal can no longer make them pass for the wrong reason. A run that had to wait says so on one line. The script needs no redeploy to take effect: the workflow runs it from the repository.

  • The weekly bump no longer lets kubectl run ahead of the Kubernetes schemas kubeconform checks against (#1323). The lab image pins kubectl (KUBECTL_VERSION) and a copy of the Kubernetes JSON schemas (KUBERNETES_JSON_SCHEMA_*) that are meant to sit on the same Kubernetes line, but the weekly bump only ever edited kubectl, so the first new line would have left kubeconform validating against the old API with a green build. The schema pins are a commit and a tree hash that no publisher serves a checksum for, so a bot cannot move them with kubectl. The bump now takes kubectl's patch releases from the pinned line's own pointer (stable-1.37.txt) and, when stable.txt names a newer line, adds a "Kubernetes release line" note to the weekly issue instead of moving kubectl: the line is moved by hand, kubectl and the schemas in one pull request. A test fails when KUBERNETES_JSON_SCHEMA_DIR and KUBECTL_VERSION are on different lines, so a hand bump that moves one fails in CI. Patch releases stay green because the check is on the line, not the patch number. Six new cases in lab-pin-bumps.test.mjs; breaking the planner or the pins five ways turned them red.

  • A Docker upgrade no longer leaves Portainer's local environment down until someone restarts the container (#1328). Portainer holds the Docker socket as a bind-mounted file, so when Docker restarts (any docker-ce upgrade does) a Portainer that was already running keeps the old, deleted socket: it still answers, but it can no longer see this host's Docker. The 2026-10-21 bootstrap (Docker 29.9.1 to 29.9.2, #1324) left it so, the 15-minute check failed three times in a row and raised the lab alert, and sudo docker restart portainer was the cure. The agent's socket proxy had the same fault in #1135. The portainer role now installs the labs_agent role's hcw-docker-proxy-stale helper (the same file, to the same path, so there is one copy to keep right) and asks it whether Portainer started before the daemon's current start. If it did, the container task restarts it onto the new socket and the run says so; if it did not, nothing is restarted. The weekly run proposes Docker upgrades, so without this it would have come back each time. portainer-stale-socket.test.sh holds the wiring (the restart tied to the answer and not to every run, asked before the container task, skipped in check mode, one helper file for both roles) and runs the helper against stand-ins for systemctl and docker with this role's own container name; putting each of eight breaks back in turn turned it red. CI runs it in the ansible-lint (lab-host) job. It does not prove a live restart, which shows on the first bootstrap that upgrades Docker. A Portainer that has only just started may still read its environment as down for a moment, and the role runs the check straight after the restart, so hcw-portainer-check now asks again every 2 seconds for up to portainer_check_environment_wait_seconds (30) while the environment reads down. A refused token, a missing environment or an odd answer still fails at once, so a real outage is reported about half a minute later at most. Three new checks in hcw-portainer-check.test.py cover reading up inside the wait, still reading down when it ends, and a missing environment not being waited for; breaking each of those three ways turned the tests red.
  • The weekly supply-chain check no longer skips the image scan and the weekly issue in a week when a pin is behind (#1308). The check job's first step runs lab-pins-upstream.mjs --summary, which exits 1 when a pin is behind and 2 when a source could not be read, and records that code for the steps after it: the scan of the published lab images, and the issue that names what is due. A run: step with no shell: is bash -e, which the step's own set -uo pipefail does not turn off, so node ...; code=$? ended the step at the first non-zero exit, before the code was recorded, and every later step was skipped. Two earlier runs (13 and 14 October) failed this way, and so did the manual run after #1320; the step can only fail like that when the checker exits non-zero, and the 20 October run, which passed, found every pin current. Since the weekly run now moves pins through pull requests that wait to be merged, a pin is routinely behind when the check runs. The step is now node ... || code=$?. The test file that runs the workflow's steps now runs them with bash -e, as the runner does; the three new cases (exit 0, 1 and 2) fail on the old step, pass on the new one, and the old step put back is caught by the same mutation check as the others. The same run showed a second thing that would have kept the weekly output unusable: the host pull request it opened (#1324, which moves Coder to v2.38.1) failed lab-pin-bumps.test.mjs, because one test typed v2.38.0 as the Coder pin it expected to find and so passed only while Coder stayed there. It now reads the pin from the real file, and passes with both the old pin and the new one.
  • The weekly lab supply-chain run now proposes tool and provider releases for the lab images, and no longer fails the whole week when cosign moves (#1308). Two gaps, both found by running the workflow's own steps against the live publishers. First, lab-supply-chain.yml ran only the host and image-base pin sets. The image-tools set (Terraform, kubeconform, Helm, kubectl, the Azure CLI and the Terraform providers in the mirror) was built in #1213 with its tests and its pull-request text, which says the weekly run found the newer versions, but no loop in the workflow named it, so those releases arrived only when someone ran the bumper by hand (azurerm 5.10.0 and Terraform 1.16.6 were waiting this week). It now runs. image-base and image-tools both edit lab-image/versions.env and the pull-request job applies one set's patch to a pristine checkout, so each of the two is planned from the pristine tree, held to its own paths, cut into its patch, and put back before the next one starts; the host set goes last because the Docker CLI fixture is recorded from its edits. Second, the cosign pin that #1240 added to lab-host/bootstrap.sh is edited by the host planner, but the host set did not list that file, and the workflow fails the run when the checkout changes outside every set's paths. Cosign had not released since, so nothing showed; cosign 3.1.4 is out now, and the next Tuesday run would have failed with no pull request for any set. The host set lists lab-host/bootstrap.sh, and its pull-request text says so. Because that is the script the owner runs as root on the lab host, the job that holds the App token now lets a patch change it only on its two cosign lines, and only to the shapes the bumper writes (digits and dots after COSIGN_VERSION=, exactly 64 lowercase hex characters after COSIGN_SHA256=, nothing after either, so a value cannot carry a command; CodeRabbit caught a first version that checked only how the line started): no other line, and no mode change, rename or binary hunk. An independent review of this change found the new per-set guard failed open if git status itself failed; it now writes to files with plain commands, as the existing guard does. Tests keep this from coming back. lab-supply-chain-steps.test.mjs takes the run: blocks out of the workflow and runs them in a temporary git repository, with stand-ins for the bump scripts, then checks the patches, the guards and the refusals (breaking the workflow eight different ways turned it red each time). The text test requires each loop over sets to list the sets it should, and every set the bumper plans except image-digests, which publish-lab-image.yml opens, to appear. A third reads the files the host planners edit from their source and requires each in the host set's paths; it is a text match, so it can miss an edit written another way. The weekly issue, the --bump documentation, the publish workflow's note and the tools set's pull-request text no longer describe the old two-set run, and that text no longer claims a checksum for the Azure CLI, whose pin holds a version only. The runbook, the lab host README and the lab image README now say the weekly run covers the tools.
  • The lab image's Terraform, Helm and azurerm pins move to their newest releases, and the tool bump's tests stop depending on today's pins (#1308). The weekly scan reports vulnerabilities in the Go code inside the tools and providers baked into the lab images. Terraform 1.16.5 to 1.16.6 and Helm 4.3.0 to 4.3.1 are routine moves (Terraform's repository still pins Go 1.26.8 at that tag, which the scan lists as vulnerable, so they should not be expected to clear findings). The azurerm provider 5.9.0 to 5.10.0 matters more: its repository pins Go 1.26.9 at that tag, the version the scan names as the fix for the standard-library findings in 5.9.0. Each pin moved with the checksum read from its publisher by node scripts/lab-pins-upstream.mjs --bump image-tools, the set the weekly workflow does not run, so these releases would not have arrived on their own. Merging publishes the images; the next weekly scan, not this change, shows how many findings are gone. What the scan still lists for the alz, modtm and azapi providers, kubeconform, kubectl and the Azure CLI's bundled Python packages is already at the newest release of each, so it waits on those publishers. Two cases in lab-pin-bumps.test.mjs also invented a "newer" release (azurerm 5.10.0, Terraform 1.17.0) on top of the real versions.env, so each stops being newer the day the pin reaches it, and azurerm 5.10.0 failed them on this very bump. They now work the newer version out from the pin, and pass with the old pins and the new ones.
  • An evidence row can name the one requirement it counts toward, and the monthly audience rows do, so two audience months no longer read as community work or hands-on material (#1313). Readiness matched a row to a requirement by its source type alone, and the audience timer's rows are manual, which Docker Captain's community and labs requirements and Microsoft MVP's reach also accept. The timer's list named reach as the requirement each row is filed under, but the stored row never carried it, and the evidence validator refuses a key it does not know. Now an evidence row has an optional requirementId: computeReadiness counts a row that names one toward that requirement only (the requirement must still accept the row's source, so naming can narrow where a row counts and never widen it), and a row that names none counts as it always did. The timer stores requirementId: 'reach' on every row and may update it on a re-run; a month already stored without it is read once more so the rows get it, and a row the owner removed is left alone. The checklist on an application offers the same rows readiness counts (evidenceCountsToward in ambassadorModel.js mirrors the API's, with the same test cases). Docker Captain's labs accepts manual again, as revision 4 of its seed (the owner files hands-on material by hand); a stored seed that was never edited is brought up on the next read, and one the owner edited is left alone. Nothing to run: it takes effect with the next Functions and frontend deploys.

  • The AVM versions bot regenerates the image's record of what each pinned version calls, so its pull request arrives green (#1306, #1309). AVM_CHILD_CALLS in lab-image/versions.env is keyed by each builder module's version, and what a new release calls is only known once the image's vendor stage resolves it, so every bump the bot opened carried the record of the old versions: #1306 (avm-ptn-alz 0.22.0, the connectivity module 0.17.7, the spoke module 0.22.3) arrived with scripts (operations) red on "records nothing for avm-ptn-alz@0.22.0 in AVM_CHILD_CALLS" and the image's Build and smoke red in vendor-avm.sh, which the pull request body called by design. update-avm-versions.mjs now does what that body told the reviewer to do: after writing the pins it runs docker build --target vendor --build-arg AVM_PRINT_PINS=1 --progress=plain lab-image (with --no-cache-filter vendor, since a cached stage prints nothing), echoing the build log to stderr, splices the AVM_CHILD_MODULES and AVM_CHILD_CALLS blocks it prints over the two in versions.env (no other line changes, and splicing twice is the same text), rewrites LAB_IMAGE_CHILD_AVM in labImage.js when the child list changed, and moves the exact pin on the module's block in the smoke payload (smoke/terraform-validate-payload/main.tf, whose module "alz" pins the builder's exact version on purpose and so broke on every bump of it). A build that fails, a print-out without the blocks, a line that is not the printed shape, or a record keyed by versions the file does not pin ends the run with exit code 1 and one sentence after the log, and no pull request opens. The workflow runs scripts/lab-image-avm-vendoring.test.mjs on the result before the upload, stages the two new paths by name (five, asserted by avm-versions-workflow.test.mjs), and the body says the record was regenerated. The vendoring test's own "goes red" cases carried the spoke module's pin as a literal (0.22.2), which would have failed the first bump of that module with a perfect record in hand; they read the pin now, as #1300 did for the frontend suites. Rehearsed against #1306's pins on a workstation with Docker: the vendor stage printed 17 children and 19 call lines, the splice changed exactly the 18 lines keyed by the three moved versions beside the six pin lines, and the vendoring test went from six failures to nine passed. --children <log> splices from a saved print-out for a run without Docker.

  • Study-guide refresher: an exam stored retired ahead of its day keeps its outline until the day. The refresher and the outlines test judged liveness by the stored status, so the first green updater run after #1289 deleted PL-400's outline eleven days before its last exam day (#1291, closed). Both now ask deriveStatus for today, as the badge does; eligibleCerts lives in scripts/study-guide-eligibility.mjs with a unit test.

  • Learn-catalogue updater: a refresh no longer deletes statusReadDate and availableDate (#1091's fields). The rebuild of every poster row carried an allowlist of fields that predated them, so each run erased them, PL-400 turned retired two weeks before its October 30 day and the data check refused the pull request (run 39218206063, after #1288). Both fields are carried now, with a unit test; a real run against Learn left only the sync date changed and the data tests green.

  • Learn-catalogue updater: two Node set-ups, because the two trees it installs pin two lines. The Monday job failed on 2026-10-19 at npm ci in frontend/ (EBADENGINE: the frontend has required Node 26.9 since the floors moved in #1161, functions/ is pinned to the Node 24 line, both engine-strict). It now installs functions/ under Node 24 and everything else under Node 26. The AVM updater's separate failure is #1287.

  • Cloudflare token roll: a child process that exits before reading its input no longer throws EPIPE (functions/src/lib/credentials/cloudflare-roll.js). The spawn helper installs an error handler on stdin and writes with end(input), so callers get the exit code and stderr through close; the tests model stdin as an emitter and add the EPIPE case. Found as an unlanded stash from #1247.

  • The lab host pre-pulls every template's workspace image, not hcw-lab's alone (#1268). lab_images pulled one image into the Coder sandbox daemon, the digest lab-host/coder/templates/hcw-lab/main.tf pins. The guided-lab templates (amo-guided-lab, na-guided-lab) pin another, and name the local default_image rather than image, so their first workspace after any bootstrap pulled about 2.75 GB while the visitor waited. files/lab-images.mjs now reads the image of every template under lab-host/coder/templates (either local name) and plans each distinct digest onto the sandbox daemon; a template whose image it cannot read fails the plan rather than going unplanned. scripts/lab-images.test.mjs and roles/lab_images/tests/lab-images-role.test.sh hold every template's pinned image to the sandbox plan, reading the pins without the helper. The templates' digests are unchanged; the host holds two images until the guided labs move to hcw-lab's.

  • Lab workspaces resolve names again, and the lab host proves it at every run and every 15 minutes (#1267). From the host firewall's first run on 2026-10-17, every Coder workspace started its container and then never connected. Coder showed "Agent is taking longer than expected to connect", because the container could not resolve coder.lab.hybridcloudworks.com to download its agent. The Terraform docker provider (kreuzwerker/docker 4.6.0) creates each workspace on the sandbox daemon's default bridge and then moves it to the workspace's own network. A container born on the default bridge gets systemd-resolved's upstream servers, not the stub, and keeps them. dockerd then asked them from the host's namespace as hcw-coder-docker, which HCW-SANDBOX-OUT refuses: at 23:20Z it had refused 576 packets since the 13:12Z boot, and its resolver rule had matched none. The firewall's README had said only a container on the default bridge, "which no template uses", would get those servers.
  • The fix: the sandbox daemon's daemon.json now sets dns: ["10.0.2.3"], slirp4netns's forwarder, for every container. Its lookups reach the host stub, the one resolver the firewall allows, whichever network a container was created on. Every template is covered, with no template edit. After each start, coder_sandbox reads rootlesskit's API and refuses a forwarder that is anything else. On the host, the provider's create sequence with that nameserver resolved, fetched the agent, and booted all nine hcw-lab labs.
  • The check that would have caught it: a new workspace_reach role. It runs from post_tasks after privilege_checks, whatever the tags, so it follows every role and the handlers they notified, and a failure stops only the commit record. At every run, and every 15 minutes from hcw-workspace-reach.timer, it creates one container the way the provider creates a workspace, as the sandbox user, on the sandbox daemon, from the newest hcw-lab image there. From inside, the container resolves Coder's name, fetches 100 bytes of the agent and runs git ls-remote on the lab repository. A failure fails the run, naming the step, what curl or git said, and where to look. From the timer, it reaches alert-lab-unit-failed through hcw-unit-failed@, and it is skipped while the sandbox user does not linger (Coder off). A plain docker run --network resolved on the broken host, so the check copies the provider's creation order, and its test holds that order.
  • The host_firewall and coder_sandbox READMEs now describe the path as measured. The runbook gains "Workspace reach", which covers reading each failure and restarting a workspace that was stuck before the fix.
  • The lab host records the commit it converged from again. site.yml's Record the commit this host converged from runs only on a full run (ansible_run_tags == ['all']). On ansible-core 2.21, ansible_run_tags is a tuple, which never equals a list, so no run has written /etc/hcw/applied-commit.json since the upgrade. On 2026-10-17 it still named 2026-10-14's commit after two full runs, and the agent's heartbeat reported that commit. The condition now reads ansible_run_tags | list == ['all']. Run on ansible-core 2.21.4, it is true on a full run and false with --tags or --skip-tags, and lab-host-site-tags.test.mjs refuses the old comparison.
  • Coder's first-account check waits for Coder to answer (#954). The first bootstrap after #1246 recreated the Coder container with its new sign-in settings. It then asked /api/v2/users/first two seconds after the container started, before Coder was serving. The published port reset the connection, and the run stopped there (2026-10-17). up --wait returns once the container runs, not once Coder serves. The question is now asked until Coder answers 200 or 404, every 2 seconds for a full minute (31 attempts). A Coder that never answers still fails the run before its route is written. Run against a stand-in server that started six seconds late, it answered 200 on the fourth try.
  • Data-subject request runbook: supported commands, real estate names, and reproducible exercise record (#1237). The runbook (docs/runbooks/data-subject-requests.md) was corrected to eliminate fictional Azure CLI commands (az cosmosdb sql query/document delete) and non-existent resource names (cosmos-hcw-prod-cus, sthcwprodbackupscus). It now establishes:
  • Supported operational access: Web UI via the Admin Ambassador Hub with optimistic ETag concurrency, and an operational CLI tool (scripts/data-subject-search.mjs) querying Cosmos DB account cosmos-site-prod-cus and database hcw via DefaultAzureCredential.
  • Thorough search: Recursive inspection of all text fields across ambassador records and full enumeration of referenced files and images, providing Azure Storage CLI commands to delete matched blobs from stsiteprodcus01 (container speakerevents).
  • Public submissions clarification: Confirms that public submissions in content (functions/src/lib/submissions.js) are completely anonymous and store no submitter or author email addresses.
  • Backup retention verification: Replaces fictional backup accounts with the real storage account stsiteprodcus01, container cosmos-export, and verifies lifecycle rule expire-cosmos-export (35-day deletion).

  • Verification exercise record: Replaces the unverified record with a verified, reproducible PASS using real commands and real estate assets.

  • Lab host: stale-bridge cleanup purges orphaned NAT rules for removed bridges (#1228). When rootless / userns-remap setups or dead Docker networks leave stale bridges, hcw-docker-stale-bridges.py removes the dead network bridge via ip link delete. However, Docker's default iptables rules in the nat table (such as -A POSTROUTING -s <subnet> ! -o br-<id> -j MASQUERADE and DOCKER chain jumps) remained behind in netfilter, matching unintended host-egress traffic and masquerading container-to-container connections as host gateway IPs. The script now inspects the IPv4 nat table (iptables -w -t nat -S) and removes all matching POSTROUTING and DOCKER rules referencing dead bridge interfaces by exact specification via iptables -w -t nat -D <chain> <rule_spec>, reporting deleted_rules alongside deleted bridges. The docker Ansible role triggers on either deleted bridges or deleted rules, and privilege_checks enforces the invariant that no IPv4 nat table rules reference nonexistent br-* interfaces.

Changed

  • Python 3.15 for the lab image, the lab host and four CI jobs, on ansible-core 2.22.0 (draft until PyYAML ships a wheel for 3.15; follows the 3.15 floor in #1458). The lab image's base moves to python:3.15.2-slim-trixie by its own index digest (the one the patch tag carries since its riscv64 build landed on 2026-11-02; the linux/amd64 image inside is unchanged), lab-host/bootstrap.sh to uv 0.13.6 (the first uv that can install CPython 3.15.2) and CPython 3.15.2, and the ansible-lint, harness, complexity and documentation jobs to python-version: '3.15'. ansible-core is 2.22.0 (released 2026-11-02; the branch carried 2.22.0rc2 until then), with its dependencies re-resolved for CPython 3.15, because 2.21 stops at start-up on 3.15, and the ansible-lint job installs the same 2.22.0. One publisher still blocks the image and the host: PyYAML 6.0.3, still its newest release, has no wheel for CPython 3.15, and since 2026-07-22 PyPI refuses a new file on a release older than 14 days, so only a new PyYAML release can clear it (yaml/pyyaml#951).

  • Docs: the delivery monitor's Telegram messages are explained, and the Linkie key's permission label is corrected. Owner request 2026-10-24 ("update Pages and readme"). docs/runbooks/alerting-and-support.md gains The delivery monitor speaks on Telegram: the four message titles (DELIVERY MONITOR RED, STILL RED, back to green, WATCH IS BLIND), what each means and what to do, the 8-hour threshold and the two limits; how-it-is-built.md links to it. The two pages that list the Linkie key's permissions now say the key screen labels Automated posts as Posts (seen 2026-10-24). The README was read and needed no change: it describes only what a visitor can use today, and the Interviews page has no interview yet. The Pages site itself was already current: its build succeeded on the merge of #1339.

  • Azure Architecture MCP Appliance on Azure MCP Server 3.0: the architecture analysis and design proposal is on the docs site (#1314). docs/architecture/azure-architecture-mcp-appliance.md is the analysis the owner asked for on 2026-10-21 before deciding whether to open a new App repository and a new AddOn repository for the appliance. It pins Azure MCP Server 3.0.0 (tag Azure.Mcp.Server-3.0.0, branch release/azure/3.x), verifies each capability the brief listed against the pinned source and Microsoft Learn, corrects four assumptions (no Resource Graph query tool, no Cost Management tool, local-only Terraform, aztfexport, conftest and azqr tools, elicitation-gated sensitive tools), recommends on-behalf-of authentication for labs and production, and lays out the evidence and ADR models, the detector and MCP tool catalogues, the promotion matrix, deployment, security, threat model, testing, roadmap, risks and probability estimates. The page is a proposal: nothing is provisioned and no repository was changed by it.
  • Ambassador hub: the seeded programmes carry the published rules (read 2026-10-20) (#1304). The Evidence tab counts against what each programme publishes, not against a first seed's guess.
  • Docker Captain takes its rules from docker.com/community/captains and the Advocu join form: open internationally at 18+, the three tracks (Advisor on a Docker member's recommendation, Thought Leader at 5,000 monthly views, Community Contributor at five years), the "sail the extra mile" exceptions, a one-year term renewed on continuous contribution with no numeric content quota, rolling applications with interviews a month or more out, and the perks as listed (discounts, not free seats or a pull-rate exemption). Its requirements become a measured audience figure, Docker content, community leadership and hands-on material, and the application link is the join form. A stored row still holding the guess updates itself on the next read, field by field (SUPERSEDED_SEED_VALUES, revision 2); a field the owner edited stays.
  • Docker-Sponsored Open Source, HashiCorp Ambassador and Coder Champion join the seeded catalogue, from their pages. DSOS: public Hub repositories under an OSI licence, updates within six months, no pathway to commercialization (donations allowed), the grants listed, a yearly renewal, and a paid subscription and DSOS mutually exclusive. HashiCorp: the three pillars, the three contribution categories, the Principles, specific examples from the preceding twelve months, self-nomination, the award expiring each year, and the 1 December to 5 January window with its notification form. Coder: five community chats, two people helped in #help, the starred repository and public templates, year-long then audited quarterly, self-nominated through the programme page.
  • GitKraken Ambassador (revision 3) words its rule as the page's tiers (Content Creator at 2+ pieces a quarter, Influencer, Community Ambassador, Feature Tester, Event Host at 30+ developers), the programme-wide asks (brand guidelines, the logo on the site and profiles, the quarterly check-in form) and the perks. The quarterly upkeep and the one counted requirement are unchanged; the logo and the check-in form are recommended activities, because the model counts evidence and neither is a count of evidence. A seed's superseded record now keeps its whole history (revisions, each step the values it replaced), and the backfill matches a stored row against every step above its own revision, so a row from the first seed that never saw revision 2 is corrected rather than stamped current with the guess kept (CodeRabbit on #1304).
  • Windows for the coming year: MIEE moves to 1 May to 31 July 2027 on the 2026 pattern, its description and its "apply before" line rolled with it so the seed does not contradict its own window, and AWS Community Builders' note records the 2026 close (2026-01-21) and the January 2027 cycle. Each moves through SUPERSEDED_SEED_VALUES, so a stored value the owner never edited follows the seed, and one they edited stays. MCT Regional Lead is not moved: its October 2026 window is open until 2026-10-31, and a stored row would show it a year away while it is live; it rolls to October 2027 in November.

  • CodeRabbit spends its one included review an hour on the head someone will read. The Advanced trial's ten reviews an hour became the open-source tier's one on 2026-10-20 (#1301 and #1304 sat on Review rate limited; #1304 asked eleven times and was refused ten), and .coderabbit.yaml now reviews no draft, pauses incremental reviews after two reviewed commits, skips Dependabot's pull requests (the owner approves those, #1299) and turns off the docstring pre-merge check, which has only ever reported zero functions. Pull requests from hcw-site-app[bot] stay reviewed, because CodeRabbit's approval is their required review (#1293). A review on this repository is asked for by hand with @coderabbitai review.

  • Foundry's first month is recorded on an estimate, and #849 closes on it. docs/architecture/cost-analysis.md gains Foundry — first measured month (October 2026, read 2026-10-20): USD 5.36 spent on the Foundry Models meter in rg-ai-site-prod-cus by day 20 of 31, a straight-line USD 8.31 and Azure's own USD 10.60 forecast against the USD 75 budget, so an October estimate of USD 8–11. Owner decision 2026-10-20: with spend at a seventh of the budget, the invoice closing in November is not worth holding the issue open for; the closed figure lands in the monthly re-read (Measurement plan, item 1). The budgets table also takes the live names site-monthly-budget and site-foundry-monthly-budget in place of the hcw- names it carried; the live name follows var.workload_name, which is site, and az consumption budget list on 2026-10-20 returned only the two site- names.

  • alert-lab-ssh-login is live and has fired; its plan declaration and StreamYard's are pruned (#953). The owner applied run-6uJYNGWLDxkdv5ZB, whose plan tfc-plan-check.yml read as the alert's create beside the permanent diff. verify-alert-state.yml then read the rule OK, and Invoke-AlertFireTest.ps1 -Test lab-ssh-login opened an alert at 00:01:44Z on 2026-10-17, six minutes after its line was written. The fire record carries the row. The alert's DECLARED entry and #1101's STREAMYARD_API_KEY reference, live as declared, are deleted. The probe flag's entry (#701) stays: the live setting still reads false. A test holds all three.
  • Credential register reads live dates the estate already knows (#1066). The credential register (functions/src/lib/credentials/) and Credentials tab view (frontend/src/components/admin/integrations/) now automatically resolve live dates from estate sources: Key Vault secrets resolve createdAt from metadata when lastWriteAt has not yet been set; lab host certificates (/etc/hcw/labs-agent.pem and /etc/vault.d/tls/vault.crt) resolve notBefore and notAfter from admin_config/lab_certificates (reported via vps-agent); and the Static Web App deployment token resolves its reset date from admin_config/swa_reset. Manual recording remains supported for owner overrides.
  • alert-kv-config-write's plan declarations are pruned, and the loop is proven gone (#1216). The owner applied #1217 in run-KZEnEksD78i8wtKr, whose plan the check read as the four declared provider additions, so the state had held []. verify-alert-state.yml then read the rule OK. The three alerts the cleanup itself had caused were closed at 15:10Z, and no new one opened in the ten minutes after. Five entries are deleted, and a test keeps them from coming back.
  • The plan check's applied declarations are pruned (#1207, #964, #970). Ten DECLARED entries in scripts/assert-expected-plan.mjs had met their own removal condition:
  • The eight for #1207's four stateful rules. The owner applied them in run-qrsPoBGgTrAZyqwa on 2026-10-16, and verify-alert-state.yml then read all four OK with autoMitigate true.
  • alert-timer-overdue's query, whose Resource Health now reads Available.
  • alert-export-window-latency's create, which the verifier reads OK.

They are deleted, along with the pinned timer rows only the timer_overdue entry used. The history comment records each one, and a test keeps them from coming back. - Both backup-missing rules are on the Fire record: fifteen of fifteen (PLAT-4, #964). Neither needed a staged test, because each fired on the real condition when it was created on 2026-10-14. That was before the host had uploaded anything. alert-lab-backup-missing-coder first opened at 06:23:14Z, with its actions run, and stopped after the first Coder dump landed at 03:37:18Z on 2026-10-15. alert-lab-backup-missing-vault first opened at 06:30:23Z and stopped after the first Vault snapshot at 14:35:10Z. Read from Alerts Management on 2026-10-16, so the two-night backup stop the rows used to call for was not made. alert-export-window-latency keeps its design test, since a degraded export window cannot be staged. - Every log rule whose condition can clear now resolves itself (ADR 0022 decision 6, amended). Four rules muted their repeats rather than resolving: alert-logs-capacity, alert-lab-disk and both alert-lab-backup-missing rules. A mute still opens an alert at every evaluation over the threshold and closes none, and on 2026-10-16 there were 43 such alerts open for conditions long gone. All four are now stateful: one alert per episode, resolved two clear evaluations later. The capacity rule's comment said its condition could not clear before the 08:00 UTC reset, and that the reset left nothing to auto-resolve. That was wrong: the reset clears it. Stateless by reason, and so recorded: - the three daily rules, which Azure will not make stateful; - alert-lab-unit-failed, whose condition is a log line, not a state; - the Activity Log alerts, stateless by platform. The 205 alert instances open in both subscriptions were closed the same day, each with its reason. 137 had already resolved and 43 were stateless repeats of cleared conditions. The other 25 were security events, each traced to a fire test, a Terraform apply or the owner's own role change. Live after the owner's hcw-azure apply. - alert-timer-overdue is seen to fire: thirteen of fifteen alert rules are now on the Fire record (PLAT-4, #964). After the owner's apply put #1195's corrected query live (20:42Z on 2026-10-15), Invoke-AlertFireTest.ps1 -Test timer-overdue disabled healthPulse at 20:48Z and the alert opened at 21:52:30Z naming it; the setting was put back as recorded. The runbook's Fire record row is dated. The two backup-missing rules remain, by the owner's call. - The Live Pages and Editor list truncation tests stand a bare element in for each row, so a loaded full run no longer times them out (#1194) (AP-F6, #930). Both proved the 500-item banner by rendering 500 full rows in jsdom: 0.8 to 1.1 s alone and 2.5 to 3.8 s in a full npx vitest run, so other suites on the same machine pushed them past the 5 s default. The banner reads only how many records came back, so LivePageRow and the Editor list's LiveItemRow move unchanged into pages/admin/livePages/LivePageRow.jsx and pages/admin/editor-list/LiveItemRow.jsx, where the tests at the limit mock them; every other test in the two files renders the real row. The read, the count check and the banner are still the page's own: a test fails if the banner goes or its >= becomes >, and a new test at 499 items in each file fails if it fires one short. They now take 65 to 117 ms in four full runs, two of them with the functions suite running alongside. No timeout was raised. - No raw error text reaches a response body, and a repository-wide guard keeps it that way: follow-up to #1198 for #945 (AP-B10). #1198 converted fifteen modules and checked those fifteen files for one spelling, (message|error): (error|err)?.message; error: error.message, any other variable name, any other file and text passed through a helper all went unseen: run against c9536e03 (as against b23a7c73), the new guard finds 197 sites, 37 of them raw text inside a response body (Cosmos, provider or socket text), where the old one passes. functions/src/lib/http/error-response.js is now the one door: an AuthoredError (and its subclasses: the CMS, settings, taxonomy, MCP, ElevenLabs, AI-router, GitHub-import, gallery-fetch, job payload-bound and rate-limit refusals) carries a sentence this code wrote, authoredMessage and failureFields show it and replace anything else with the route's safe sentence, and every failure answer carries a correlationId whose log line holds only the sentence, the error's content-free facts (class name, code, HTTP status, Cosmos substatus, the cause's code), up to three places in this repository's code where it was thrown (at=src/lib/jobs.js:657) and that id. The helper used to log the whole error; the message is not logged now, and of the stack only those locations. A caller's x-correlation-id is used only when it has the id shape, and the v4 worker's traceParent is read. The alerting runbook finds a reference's line in Log Analytics. Provider values that used to be relayed are now their status and machine code, each held to the door's code shape (codeShaped) before it goes into a sentence: Resend's error name, the OAuth error code and token_type, ElevenLabs's detail.code, the JSON-RPC code, a preview's content type, and the MCP sign-in metadata's addresses (scheme, host and a URL-character path only). A failed connection probe answers { ok, status, error } and no upstream body; the provider's sentence on a rejected key stays on the super_admin API-keys page (#463). no-raw-error-leaks.test.js replaces the fifteen-file regex with a parse of every module under functions/src: .message or .stack off anything, a caught error stringified (String(e), a template literal, +), a settled promise's reason, aliases and destructuring, scope-resolved by catch and .catch() parameter; a site is any object field, return, stored field or helper argument it lands in, a site inside a response builder's body (resolved by import, or an injected one called with a literal status) is marked, and an aliased AuthoredError subclass built from someone else's text is a site. A helper whose return holds a stored or log-only site carries its text to its callers, and that text is a site where it reaches a response body: the Podbean hub's reads and the Labs snapshot's lab checks were both that, and step 2 fixed them. Stored failures follow the same rule at the moment they are written: storedFailure and recordedFailure keep an authored sentence, or store the site's own sentence with the error's code and the reference of the content-free log line (The job failed (ECONNRESET; reference <id>).), still one string where a string was. That covers job.error, the Listen & Learn and podcast failure fields, the Podbean host record and the hub's reads of it, the cover, caption, inspection and Publer sync errors, the health pulse, the lab checks and the drift read, the Plaud token, the pricing and RSS job results, the Telegram activity log and the scheduled-publish records. The job workers' refusals and the study-guide, script, transcript, video, cover, grounding and Podbean failures are now AuthoredErrors whose sentences name a status and the provider's code, never its words, so those pages still say what went wrong. Documents already stored keep the text they hold: nothing is migrated. The 149 remaining sites (none of them raw text in a response body) are listed in 146 entries, each with a class and a reason (100 authored through the door, 42 that never reach a response, three message fields that are not an error's, and one stored diagnostic kept by decision: a provider's sentence on a rejected key, which #463 made the diagnosis and only the super_admin API-keys page reads), and the list fails on a new site, a stale entry and a wrong class. getJob now holds a job whose type is no longer registered at the enqueue floor (editor) instead of the viewer gate, and answers its 500 with a correlation id; the certification routes are pinned to refuse _createdAt, _updatedAt, _etag and _ts; a Forge Studio entry whose job write failed is said to be back in the queue only when the revert confirms it; and the admin client appends the correlationId to a failed call's message (… (reference <id>)) and keeps it on the thrown error.

  • Public reads return an allowlisted set of fields. The article detail (GET public/content/{slugOrId}), the signed preview, the Listen & Learn certification read, the podcast list and the news feed each answer with a named list of fields, projected in the Cosmos query where the read is a query and filtered in code on every path, instead of a stored document with a denylist removed. A field a writer adds to a document is no longer public until it is listed. The lists are derived from what the public pages read (PUBLIC_CONTENT_DETAIL_FIELDS, PUBLIC_PREVIEW_FIELDS, PUBLIC_LISTEN_AND_LEARN_EPISODE_FIELDS, PUBLIC_LISTEN_AND_LEARN_SET_FIELDS, PUBLIC_PODCAST_FIELDS, PUBLIC_FEED_CACHE_FIELDS in functions/src/lib/public-reads.js and public-preview.js), and public-reads.allowlist.test.js pins each one against the frontend components that read it and fails when a response carries a key its list does not name. The article list keeps its existing projection, less the framework builder's three generation prompts (no public page renders them, and useFrameworkData no longer maps them), and now also filters in code; the snapshot read keeps its publish-time sanitizers.

  • Lab host: the nightly reboot asks at the hour, and reboots only an idle host (LAB-13, #959). unattended-upgrades evaluated its logged-in check when the upgrade ran, about twenty-two hours before the 04:30 it then scheduled shutdown -r for with no further question, and nothing ever asked whether a visitor's lab job container was running; a job killed at the 04:30 reboot is the issue's evidence. The hardening role's override now sets Automatic-Reboot to false (the two settings read only while it is on are gone), and a new hcw-reboot-when-idle.timer (*-*-* 04:30:00 from hardening_unattended_reboot_time, Persistent=false, no jitter) runs /usr/local/sbin/hcw-reboot-when-idle, a root oneshot with the file system read-only to it, home hidden and the failure notifier in OnFailure=. It reboots only while /var/run/reboot-required exists, logind lists no login session (loginctl, class user or user- and a suffix; the manager, manager-early and background sessions of lingering service managers and cron do not count), the host daemon has no container with the agent's job label up (hardening_reboot_job_label, held to JOB_LABEL in vps-agent/lib/docker-runner.js by scripts/lab-job-limits.test.mjs and by the role's test; up is created, restarting, running or paused, so a job whose container exists and has not started counts), and the Coder sandbox daemon has no workspace container up (coder.workspace_id on /run/hcw-coder-docker/docker.sock, asked as root when the socket is there: a person working in a workspace has no logind session and the host daemon does not see it). It asks twice (CodeRabbit on #1191): when the first look finds the host idle it writes a maintenance hold, /run/hcw-maintenance/reboot-hold (hardening_reboot_hold_path, root's, 0644, in the service's RuntimeDirectory=), holding the epoch second it lapses (300 s on), waits for the labs agent to acknowledge it, and looks again. The agent reads the hold on every poll tick and claims nothing while it holds (vps-agent/lib/maintenance-hold.js; the claim loop moved from index.js to lib/poll.js so it has tests; LABS_AGENT_MAINTENANCE_HOLD_PATH from the new labs_agent_maintenance_hold_path, the same path, held together by scripts/lab-job-limits.test.mjs), caps any hold, unreadable ones included, at ten minutes from its write, and reads the time and the expiry from one descriptor, so a hold renamed over between two calls cannot pair one file's cap with the next file's expiry (CodeQL js/file-system-race). On a tick with no claim in flight and no job running it acknowledges the hold, writing its expiry exactly as read to /run/hcw-labs-agent/maintenance-ack (labs_agent_maintenance_ack_path, hardening_reboot_ack_path, LABS_AGENT_MAINTENANCE_ACK_PATH, one path), through a temporary file and a rename, in the agent unit's new RuntimeDirectory=hcw-labs-agent, which is not preserved, so a stopped agent leaves none behind; it withdraws it when a claim or a job appears, when the hold lifts, and when it stops. The script waits up to 45 s (hardening_reboot_hold_settle_seconds, at least a 20 s claim and one 15 s poll tick) for an acknowledgement naming its own hold, asking first whether systemd calls the agent inactive or failed, when no wait is needed, and defers the night without one (CodeRabbit's second round on #1191: a fixed wait could not see a claim whose container was slow to appear). An agent that cannot acknowledge, an older or a stuck one, so costs a night's reboot and never a job, and the first bootstrap.sh, which enables the timer before it deploys the new agent, defers rather than risks one. The second look still asks logind and both daemons; the hold is removed on every way out but the reboot, where it stays until the reboot clears /run. One syslog line per outcome under the tag hcw-reboot-when-idle: the reboot and the deferrals at warning, which the Arc rule ships, and a deferral of a reboot pending 7 days or more (hardening_reboot_overdue_days) at err, with what to do, never a forced reboot; the days count from the first night a run saw the marker, kept beside the hold in /run/hcw-maintenance/reboot-pending-since (cleared with the marker by a reboot), because every upgrade that wants a reboot rewrites the marker's own time, and the held report ages its pending line from the same record; nothing pending at notice; an apt-config, a loginctl or a docker daemon that cannot answer, docker missing included, or a hold or record that cannot be written, is an err line and a failed run, so the notifier pages rather than the host guessing. A stop exits 143, which the unit counts as success (SuccessExitStatus=143), so neither an owner's systemctl stop nor the reboot's own stop job pages. It does not ask users: on the host that is rust-coreutils, which reads utmp, and systemd no longer writes utmp, so users and who printed nothing with an SSH session open (measured 2026-10-15). unattended-upgrade's WithUsers check read users too, so #1172's "no reboot with a user logged in" never deferred anything there. At most one reboot authority at a time: the role first cancels a shutdown -r that unattended-upgrades scheduled before the run (one its log names; any other scheduled reboot stops the run with what to do, because a root shutdown -r for the same minute is the same record; the cancel's read-back is keyed on the cancel having run, since check mode registers rc 0 for a command it skips), then installs, enables and reads the timer back active, writes the override that turns Automatic-Reboot off, and reads apt's effective Automatic-Reboot back, stopping if a later file turns it on again. The script stands down with a warning line while apt has Automatic-Reboot on, so a partial run or a rollback never has both rebooting; a run that cancels unattended-upgrades' reboot and then stops before the override leaves one night with neither, until unattended-upgrades schedules again or bootstrap.sh is re-run, and the runbook has the line that withdraws the timer and the three limits kept on purpose. The daily held report names the timer and its hour in its pending-reboot line, the role refuses a reboot hour that is not HH:MM and a hold outside its bounds, and tests/hcw-reboot-when-idle.test.sh runs the rendered script against stub apt-config, loginctl (the host's measured session listing among its fixtures), docker (both daemons, filtering on container state as docker does), sleep (the wait, where the agent acknowledges on a chosen second, with the wrong hold, or never, and where a job, a workspace or a login appears), systemctl (the agent's state) and logger in the ansible-lint (lab-host) job; lib/poll.test.js and lib/maintenance-hold.test.js hold the agent's side (an acknowledgement only under a hold with nothing in flight or running, written atomically, withdrawn when busy or lifted), and scripts/lab-host-docker-proxy.test.mjs the sandbox socket. The backup and Coder timer comments and the Vault pages that named "the unattended-upgrades reboot at 04:30" now name this timer, and the two Vault passages that said a reboot needs three unseal keys say that holds under Shamir keys only: this host auto-unseals, so its idle reboot comes back unsealed. alert-lab-unit-failed in the alerting runbook lists the idle reboot and the held report among the units that page. The timers' schedules are unchanged. On the host after the next bootstrap.sh; the runbook's two checks after it are loginctl list-sessions listing the login as class user and shutdown --show printing No scheduled shutdown.. Ask 2 of #959; asks 1 and 3 are their own changes.

  • A late podcast publish hook clears only its own job's reservation, under the version it read (#942, follow-up to #1198). #1198 made the sweeper hand a poisoned or reaped job's onComplete the whole document, which made the podcast publish hook live on those paths; it already ran on runJob's timeout. Running late, it read the transcript and wrote unconditionally, so it could clear a newer Publish's host.podbean.pending marker or erase an episode a racing worker had just stored, and a second job could then put the same episode on the feed. recordHostJobFailure now takes the job's id, writes only while the marker is pending and names that job, and writes with ifMatch on the _etag it read; a 412 is read again at most once, and a marker that has moved on is the answer. Reading whole job documents is cheap only while their payloads are bounded, and that bound now holds at every door: newJobDoc refuses a payload over its type's maxPayloadBytes (JOB_PAYLOAD_TOO_LARGE, 413), the Cosmos export scheduler and the Telegram /approve command, which wrote object literals, now build their documents with it, a Forge Studio entry whose document cannot be built is put back to queued, and jobs.payload-bound.test.js reads every upsertDoc/createDoc to the jobs container (by JOBS_CONTAINER, a 'jobs' literal or a local name for it) and fails one whose document newJobDoc did not build. A payload over the bound is the caller's 413, not a 500: Listen & Learn's routes answer JOB_PAYLOAD_TOO_LARGE with a fixed sentence, and the recording upload measures its job payload before it stores the audio, since a title within 200 characters can JSON-escape past the 1024-byte bound. New jobs.signal.test.js keeps #1198's threading in place: it reads every module under src/ that registers a job type (aliased and namespace imports included), pins the eight modules and the sixteen types, and fails a worker that never hands its signal to a call. Only an argument that carries the signal counts (the signal under any name, ctx.signal, or the whole context, so a thin wrapper passes); one that destructures it unused, only checks it first, names it in a string, or passes Boolean(signal) or signal.aborted does not. noop was the one worker that ignored it; its delay now ends with the job. And #942's own acceptance, that a timed-out job's side effects stop, now holds for writes: under #1198 a worker checks its signal at points it chooses, but its Cosmos and Blob writes went through shared clients that never saw it, so one that started after the deadline still landed (a timed-out Podbean publish rewrote the transcript's host.podbean after the job was recorded timeout). New lib/job-context.js runs the worker in the job's context, and cosmos-client.js and blob-storage.js refuse a write that starts there once the signal has fired, and check again before a read-modify-write's replace and its retry. A write already dispatched when the signal fires runs to completion, SDK retries included, and the job's signal is never handed to the SDK: cancelling it on the client would not recall it from the service, so the write could land while the job saw it fail, and the records that follow it would never be written. Each refusal is a fresh error with the job's reason as its cause. Routes, timers and the hooks run outside the context, and so does work that other callers join: the MCP OAuth token refresh, a process-wide single-flight, is started with outsideJob. A write that records something already done is kept past the deadline: usage rows (ai_usage), rotated MCP tokens (mcp_servers), and the version snapshot and audit row that follow a content write that has landed (content_versions, admin_audit_logs, audits), so a forge save cut off by the deadline still leaves its version and its audit row. runJob disarms the deadline as soon as the worker settles, so a worker that finished just inside it does not see its signal fire during the terminal write, and the publish hook now warns, without ids, when every attempt met a 412 and pending was left set.
  • Admin routes are the menu labels' slugs and page titles are the registry's labels, the single-draft RPC file says what it registers, deleting a live page archives it with ten seconds of Undo and the Archive board carries Restore and Delete permanently, and Ctrl-K opens a command palette over every admin page and the content search (AP-F13, #937). frontend/src/config/adminNav.js is the one place a page's label, route and title come from: eight items routed to a slug that was not their label ("Publish" at /admin/published, "Review Queue" at /admin/queue, "Newsletter Hub" at /admin/mailing-list, "New Content" at /admin/submit, "Social Hub" at /admin/social, "Linkie Hub" at /admin/linkie, "Speaking" at /admin/speaking-events, "Platform Settings" at /admin/platform), and adminNav.test.js now holds every row to /admin/<slug of its label>. The old paths live on in frontend/src/config/adminRedirects.jsx, which also took over the five redirects App.jsx carried inline (#442, #569, #570) and, unlike them, carries the query, the hash and a nested segment through (/admin/queue/<id>?source=content still opens the review board); adminRedirects.test.jsx walks every entry and reads App.jsx for the route each lands on. Every link in frontend/src, the two e2e specs and the API's own admin links (functions/src/lib/decision-center/links.js, calendar/collectors.js, cms/drafts-handlers.js, timers/newsletter-autobuild.js, the ElevenLabs voice-picker URL) moved with them, since itemLinks.test.js holds the two sides equal. components/admin/shared/PageHeader.jsx (and ServicePageHeader.jsx, the Certifications page's) now render the registry label as the h1 and the document title; seven pages had written their own ("Content Queue", "Health Hub", "Integrations Hub", "Platform Settings Hub", "Speaking Events Hub", "Certifications Hub", "Ambassador Hub"), and PageHeader.test.jsx renders the header at every registry route and reads every admin page's source for a literal title=. functions/src/functions/draft-http.js, one letter from drafts-http.js and nothing to do with the Drafts stage, is generate-article-draft-http.js; the function name and route are unchanged. On Live Pages, Delete Live Page — one confirm, then softDeleteLivePage, which stamped a 24-hour soft delete nothing in the API ever swept — is Archive: published → archived through transitionContentStatus, which writes the audit row, and a toast with Undo for ten seconds (pages/admin/livePages/useLivePageArchive.jsx) that runs the new archived → published edge back. lib/cms/content-status.js gained that edge (publisher-only, like every edge into published; the port's "no resurrect-to-live" pin is reversed with the reason), and its archived and rejected appliers now also write the legacy Status word, because public-reads.js admits a document on Status: 'Live' as well as the flag — public-reads.test.js applies the archive patch to a document that says live every way it can and checks nothing serves it. Leaving the archive clears archivedAt; every non-rejected transition clears softDeleteExpiresAt beside softDeletedAt. The Editor's Archive board (pages/admin/editor-list/ArchiveBoard.jsx) carries Restore to live and Delete permanently (confirm, then deleteContentItem, the publisher RPC the Review Queue's delete uses; cmsDeleteContent is an editor-level hard delete and was left alone). MCP server removal keeps its confirm: the OAuth token lives only on the deleted document and is write-only, so nothing could restore it. components/admin/CommandPalette.jsx, mounted once in AdminLayout.jsx, opens on Ctrl-K, Cmd-K or its button as a Radix dialog — labelled, focus-trapped, Escape closes — listing the registry's pages filtered by what is typed and, from two characters, the content the list route's search matches (#1178; lib/api.js listContentItems), asked once per pause in typing and answered in order; Enter opens a page at its route or a content item in its editor. Every edge out of a page the public site serves is a publisher's: lib/cms/content-update.js gates on published or isPublicDocument, so a live editing page or a legacy Status: 'Live' one is covered, and a caller that names no role is refused rather than waved through (the Telegram bot's /reject names super_admin, its caller being the owner's chat). Archive and reject clear scheduledPublishDate, as the flows Archive replaced did, so timers/publishing-watchdog.js does not count an archived page overdue. The pre-render manifest's PUBLISHED_PREDICATE (lib/public-content-manifest.js) is now the public API's own SQL_PUBLIC_CLAUSE and SQL_NOT_SOFT_DELETED with archived, rejected and drafting excluded; its old three words admitted documents the API answers 404 for. test/cosmos-predicate.js evaluates it against documents. Live Pages offers Archive only on a published or rejected page, the statuses with an edge to archived, and says why on any other; the palette asks the content search only for 2 to 100 trimmed characters, the list route's bounds (#1184), and says so past 100. transitionContentStatus writes under the _etag of the read its gate and edge were decided from, and answers a lost race with a 409 ("The page changed since it was read; reload and try again.") and no audit row, so an editor's reject read before a publisher set a page live cannot land after it. Archiving a page also archives every linked legacy blogs document the site serves (publishedBlogId, sourceContentId, publishedContentId), stamped archivedWithContentId, and Undo or Restore to live brings back exactly those; content-update.test.js checks each with isPublicDocument. deleteContentItem deletes under the read's ETag (deleteDocIfMatch), refuses with a 409 a body's expectedStatus the document has left (the Archive board sends archived; the Review Queue and the framework review send none and keep their rules), and writes its own content_deleted row to audits, so the framework review's client audit row for the same delete is gone. The toast's Undo is sent once and disabled while in flight, and a restore on the Archive board re-reads the Editor's live list. The linked blogs are chosen from reads taken after the page's own write, each written under its read's _etag, and settled against the page as it then reads rather than the edge asked for: an archived page takes down every linked blog the site serves, a published one brings back the blogs it archived. Once a round's writes land the page is read again, and if it or a blog moved the round runs again (three at most), so a restore and an archive that cross leave the blogs agreeing with the page's final status; three interleavings are pinned in content-update.test.js. A blog deleted in between, or still contested after the third round, is named in the audit row (linkedBlogIdsSkipped, linkedBlogsSettled: false) and the transition still answers 200. patchDoc's read-modify-write path (lib/cosmos-client.js) answers a document gone before its read with code: 404, as the patch path already did. deleteContentItem's two failure lines are a fixed sentence and the error's code, with no document id. Rebased onto the role-cut menu (AP-F4, #928): Live Pages' Archive and the Archive board's Restore to live and Delete permanently sit behind the publisher RoleGate, and the command palette lists only the pages visibleNavFor gives the signed-in role and asks the content search only for a role the Editor is shown to.
  • alert-timer-overdue evaluates again, a KQL test refuses what silenced it, the weekly check reads every log rule's Resource Health, and the fire test matches the timer dimension (PLAT-4, #964). The rule never evaluated from its create on 2026-10-10: its query took each timer's last due time from bin_at(Now - Grace, period, anchor), Kusto requires bin_at()'s bin size and fixed point to be query constants, and period and anchor are columns of the rule's datatable. Every evaluation failed with SEM0040 bin_at(): failed to cast argument 2 to scalar constant, the rule's Resource Health read Unavailable / Semantic error from 2026-10-10 22:55Z, and a fire test on 2026-10-15 stopped healthPulse for 95 minutes with no alert. Azure accepted the rule at create, and nothing in CI ran the KQL. The line is now anchor + period * tolong((Now - Grace - anchor) / period), checked against appi-site-prod-cus-01 for every timer due in the last day. scripts/assert-expected-plan.mjs declares the in-place update of the rule's criteria[0].query, its before read back from the live rule character for character, and a new test holds overdueQuery() to the heredoc in infra/. scripts/kql-alert-columns.test.mjs fails any rule that passes a per-row value where Kusto needs a constant: bin_at()'s bin size and fixed point, measured against the component (a let, now() or toscalar() is accepted; bin() and floor() take a column and are not checked). It failed on timer_overdue before the fix. scripts/verify-alert-state.mjs reads each live log search rule's Resource Health and fails Unavailable or Degraded as UNHEALTHY, naming Azure's title, and a read it could not make as HEALTH UNREADABLE, never OK. Unknown passes, as does a reason-less Unavailable on a rule evaluated less often than every 15 minutes, for which Microsoft reports no health. The report gains a Health column. HCW Alert Rule Reader gains the two Microsoft.ResourceHealth/availabilityStatuses reads. The owner writes the role and registers Microsoft.ResourceHealth in sub-plat-mgmt-prod-cus, unregistered there on 2026-10-15 (docs/runbooks/alerting-and-support.md, "Resource Health"); until both are done the run fails, naming each step. A refusal is never reported as an unregistered provider: that needs ARM's MissingSubscriptionRegistration code or the measured 409 AuthError. The runbook's rollout record reads the role's actions, both its assignments and the provider in both subscriptions, works in Windows PowerShell 5.1 as well as 7, and restores the previous action set from 0bd89190 (#1074). Invoke-AlertFireTest.ps1 searched the whole serialized alert context for healthPulse, and every condition.allOf[] entry carries searchQuery, which names all 26 timers, so any timer's alert would have passed as healthPulse's. It now matches the values of condition.allOf[].dimensions[], read at the live nesting (context.context) with a fallback one level up, and the Pester fixture takes the live shape, with a case where searchQuery names every timer and the dimension another. The Fire record stays "not yet" until the test is re-run after the apply.
  • Lab host: ansible-core installed hash-locked from the lab image's own pins, and the weekly pin bump moves it (LAB-13, #959). The low-hanging-fruit entry below (#1178) says lab-host/bootstrap.sh hash-locked the ansible-core wheel. It did not: the script printed an ANSIBLE_CORE_SHA256 it carried as a copy and installed with uv tool install "ansible-core==2.21.4", which takes no hashes, so neither that wheel nor the eight it depends on was checked. The script now keeps no ansible-core pin. After it checks out the commit it will run, it reads ANSIBLE_CORE_VERSION, ANSIBLE_CORE_SHA256 and ANSIBLE_CORE_DEPS from that commit's lab-image/versions.env as data, never sourcing the file, so an edit to it cannot run anything as root, and refuses any requirement line that is not one pinned, hashed wheel, so nothing in it reaches uv as an option. It writes the requirements file with the lab image fetch stage's own two printf lines, and installs it with uv pip install --require-hashes --only-binary :all:, then uv pip check and the fetch stage's count check, into a new venv under /opt/uv/ansible-core-generations, a directory of its own for every build, built beside the one in use. Only when that passes, and its ansible-playbook --version reports the pinned release on the pinned Python, does the link /opt/uv/ansible-core move to it, so a failed download, hash or check leaves the host on the environment it was using, even when that is a build of the same pins that has stopped passing; a generation is removed only after the link has moved off it, and one of other pins is kept until the next move. ansible, ansible-playbook, ansible-galaxy and ansible-vault are linked into /usr/local/bin through that link, and hcw-vault-set runs its Python through it. The old uv tool under /opt/uv/tools is removed after the play of the first run that has installed that helper, so a check run, a run narrowed with --tags or a failed play never leaves the helper on a removed interpreter; a rollback with HCW_REPO_REF to a bootstrap.sh from before this change removes the ansible-playbook link before handing over, so that script reinstalls its uv tool. The install commands, run in an Ubuntu 26.04 container with uv 0.12.19 when first written, installed ansible-core 2.21.4 and its eight dependencies on CPython 3.14.7, refused a wrong hash and an unlisted dependency, and installed nothing on a re-run; hcw-vault-set's 81 checks passed with /opt/uv/ansible-core a link to a venv. scripts/lab-host-bootstrap-ansible-core.test.mjs runs the switch, the retirement and the handover against a stand-in uv, and holds the script to versions.env and to the Dockerfile's printf lines. The weekly chore/lab-pins-image-base pull request now also moves ansible-core to the newest release in its line that has a wheel PyPI has not yanked, with the wheel's SHA256 from PyPI's JSON API and every ANSIBLE_CORE_DEPS hash read back from PyPI, and moves a dependency only when the new release pins it exactly; a new line, a changed requirement or Requires-Python, or a hash PyPI does not serve is a note in the weekly issue instead (a dry run on 2026-10-15 proposes 2.21.4 to 2.21.5). On the host after the next bootstrap.sh.
  • A Health view inside 30 seconds of the last one is served from the cache instead of running the eleven-query fan-out again, and a Platform settings save made from a page that is out of date is refused and reloaded instead of overwriting the newer save (#946, the two acceptance items the merge of #1181 left open). The ops-health snapshot cache lived inside createOpsHealthHandlers, and functions/src/functions/ops-health-http.js builds a handler set per request, so every request found it empty; it is now held by functions/src/lib/ops-health.js for the worker's life (resetOpsHealthCache() for tests), keyed on the store and environment the snapshot is built from and not on the caller, since every admitted role gets the same snapshot and the guard still runs first. The HTTP file passes one module-level store. A build serves every view while it runs, however long Cosmos takes, and for 30 seconds after it settles; a failed build is dropped rather than served; and an alert update invalidates it as soon as the alert is patched, before the audit writes that can fail. The Health page's first read takes the cached snapshot, but its refresh after an action (Reviewer Digest, RSS Fetch, Run all, a probe, an alert answered) or a "Try again" sends { refresh: true } (useOpsSnapshot.js) and gets a build that started after it, so the action shows at once on any worker: one starts at once when none is running, and refreshes arriving while one runs share the single build queued behind it. On the settings side, the If-Match branch #1181 added could not be reached from a browser: lib/auth/cors.js now allows If-Match (the allowlist is exactly Content-Type, Authorization, If-Match) and exposes ETag beside WWW-Authenticate; GET cms/platform-settings/{setting} returns the document's etag in the body and as an ETag header; and the Platform Settings tabs (settingShared.jsx useSetting) and the Mailing List's NewsletterSettingsCard.jsx send it back as If-Match, hold the one each save returns, and on a 412 or 428 say the setting was saved elsewhere and reload it. The precondition is now required: a save that names no version may only create the document (createDoc), a stored setting is replaced only by replaceDocIfMatch under the version named (412 CONFLICT otherwise, also when another write lands between the read and the replace), and replacing one without If-Match, or with If-Match: *, which names no version, is a 428 PRECONDITION_REQUIRED. Those two pages were the only callers of the write; no script, Telegram command, MCP tool or timer calls it. The reminders sheet keeps its merge under its own read (review of #910), so its read offers no etag and a timer stamp landing mid-edit is still merged rather than refused.
  • Alert fire record: twelve of fifteen rules seen to fire, and the SSH burst test writes lines rsyslog cannot fold (PLAT-4, #964). The fire tests ran on 2026-10-15 from a detached runner (Invoke-AlertFireTest.ps1, one test at a time, every change put back and checked), and the Fire record in docs/runbooks/alerting-and-support.md carries a dated row for each rule that opened: the two Key Vault rules, both role-assignment and both role-definition rules, the Static Web App bandwidth, the poison queue, the lab vault unwrap, the lab agent offline and the SSH burst, with the 2026-10-11 Key Vault config write. The SSH burst test's first run failed for a reason in the test: it wrote a thousand identical lines with logger, and the host's rsyslog ($RepeatedMsgReduction on, Ubuntu's default) folded them into two Syslog rows against a rule that needs fifty; each line now carries its own port (Invoke-AlertFireTest.ps1, its Pester test and the runbook say why), and the re-run opened the alert in about fourteen minutes. Still open: alert-timer-overdue, which has never evaluated (fixed in its own pull request and re-run after the apply), and the two backup-missing rules, whose test stops a backup timer for two nights.
  • A lab job running when the host stops is interrupted and reported within seconds, not left claimed (LAB-13, #959). This replaces the 210 s drain #1178 added: that drain then exited with the job's container still running and nothing reported, so the job sat claimed until its 15-minute lease ran out. On SIGTERM the agent now aborts every running job: the runner kills the docker run child and starts removing the container by name through the agent's Docker proxy (docker rm -f, the one DELETE the proxy allows). Each job is reported at once, without waiting for that removal, which may take 10 s: failed, exit -1, with The lab host was restarting and this job was interrupted. Submit it again. The agent then waits up to LABS_AGENT_SHUTDOWN_TIMEOUT_MS for those reports (default 5000; anything but a positive whole number is the default), sends offline, and exits. Its stopping and offline heartbeats time out at 5 s each, and the timeout now covers the token request too. A claim already in flight is waited for within the same budget, and the job it brings back is reported without being started. The job loop moved from vps-agent/index.js into vps-agent/lib/shutdown.js, so the shutdown has tests. The labs_agent unit sets KillMode=mixed, so systemd signals the agent and never the docker CLI under it, and TimeoutStopSec=30 (was 240). The new labs_agent_shutdown_timeout_ms is written to the env file and listed in .env.example. scripts/lab-job-limits.test.mjs checks that 30 s covers the 5 s budget plus both heartbeats, and that the env template renders every limit. tests/hcw-labs-agent.service.test.sh renders the unit and env file with the role's values in CI. The containers are the unit's to clean up, not the process's: a removal can take 10 s, longer than the 5 s window, and a crash removes nothing. The unit therefore runs the new /usr/local/libexec/hcw-labjob-sweep as root after the agent exits and before it starts, removing every container labelled hcw.lab-job through the daemon's own socket. It judges by what is left running, since the agent's own removal may still be at work, and fails only when a container is still running at its 20 s deadline. The proxy allowlist is unchanged, and tests/hcw-labjob-sweep.test.sh runs the sweep in CI against a stand-in for docker. On the host after the next bootstrap.sh. Two LAB-13 asks stay open on #959: reboot only when no job runs, and a hash-locked ansible-core.
  • The admin menu is cut to the signed-in role, the role is printed under the brand, and a control above that role says so before the round trip (AP-F4, #928). Every item in frontend/src/config/adminNav.js carries a minRole, read from the role the server's guard asks for on the routes the page exists to call and recorded item by item in a table beside the registry (viewer: Dashboard and Health; publisher: Publish; editor: the other twenty-five); a row without one fails at module load, as the server's requireRole fails on a misspelt role. visibleNavFor(role) filters the groups and drops one left empty, and AdminLayout.jsx and the dashboard's Explore strip (DashboardPage.jsx) both read it, so a viewer no longer learns a page's role from its 403. AdminAuthGuard.jsx hands the role it verified down through hooks/useAdminRole.jsx, one value for the whole portal rather than a second status check per consumer, and the rail and the phone drawer print it as "Your role: …". components/admin/RoleGate.jsx wraps a control the signed-in role cannot use — aria-disabled, the click blocked, the reason as visible text the control names through aria-describedby, so it stays in the tab order and explains itself — and sits on the actions the server gates above the page's own role: Publish and Unpublish, Set slug, Re-host selected and Delete Live Page; Delete Rejected Now and Delete Permanently on the queue; Acknowledge, Resolve and Reopen on the Health alerts; the Recording Hub's transcript Approve, Return to draft and host Retry; the newsletter's Approve and schedule, Send test, Reschedule, Cancel send and Retry; Listen & Learn's Approve and publish; the calendar's Reschedule and Unschedule; and the dashboard's Recount (an editor write on the one page a viewer opens first). The server's guard is unchanged and still answers every request on its own; the gate is what the admin reads instead of the 403.
  • Polish defects that compound on a phone: no browser confirm dialog, toasts that stack and announce, a toaster that mounts with the admin shell, a loading state with a status role, one status request per user, and a catalogue failure that is shown (AP-F12, #936). The three window.confirm calls that sat beside the shared modal (frontend/src/components/admin/ai-engine/McpOAuth.jsx, components/admin/KeywordMatrixPanel.jsx, pages/admin/AIEnginePage.jsx) and the Drafts page's navigation guard (pages/admin/drafts/useUnsavedGuard.js) are state rendered with ConfirmModal (ADR 0033 §2); the guard keeps the cancelled click and replays it on confirm, since useBlocker needs a data router the app does not mount, and beforeunload stays the browser's. ConfirmModal gained a busy state: the confirm button disabled under a progress label and the dialog not dismissable while the confirmed request runs, used wherever the action awaits one. src/no-window-confirm.test.js reads every source file under src so the dialog cannot return. components/ui/use-toast.js holds three toasts instead of one, each expiring on its own clock, and toaster.jsx no longer makes the viewport one atomic live region (every arrival re-announced the list and the toast it replaced was never announced); Radix's per-toast role="status" region announces each. The toaster is a static import of pages/admin/AdminLayout.jsx rather than a lazy Suspense child of App.jsx, so a toast fired before its chunk arrived cannot expire unseen; no public page fires one. A shared components/ui/spinner.jsx (role="status", a label) replaces the bare animate-spin rings in AdminAuthGuard.jsx, PublishedPage.jsx and the two newsletter settings cards. hooks/useAdminAuth.js keeps one in-flight promise per user in its cache, shared by every mounted instance, dropped by clearAdminStatusCache, and never caches a failure; lib/api.js no longer forces a token refresh for getCurrentAdminStatus on every call — a page load takes MSAL's cached token, renewed when it has actually expired, and only an explicit re-check asks for a fresh one (freshToken), the case where a just-granted App Role matters. AIEnginePage.jsx shows a dismissible role="alert" notice when the model catalogue cannot be read, naming what still works and a Try again, where it used to log and show nothing.
  • AP-F11: the site's Content-Security-Policy drops 'unsafe-inline' and the scheme-wide image grant and gains base-uri, object-src and form-action, the admin token cache moves to sessionStorage, and no source map ships (AP-F11, #935). frontend/staticwebapp.config.json commits a policy with no 'unsafe-inline' anywhere and an img-src of the hosts the site was measured to load images from — the media route, the certification badge hosts, the show's artwork host and the three hosts published articles still hotlink; data: stays for the upload previews and two article bodies, blob: goes — beside base-uri 'self', object-src 'none' and form-action 'self' https://login.microsoftonline.com. frontend/scripts/stage-swa-config.mjs finishes the policy after every build instead of copying the file: one 'sha256-…' per distinct inline <style> body in style-src and one per distinct style attribute value in style-src-attr 'unsafe-hashes', computed from the pre-rendered pages, so a change to a component's style prop updates the header on the next build (2026-10-15: 2 elements with one body and 2,969 attributes with 80 values across 230 documents). frontend/scripts/check-image-hosts.mjs runs next and fails the build when a page, a stylesheet or an article body references an image host img-src does not name, tolerating the retired Firebase bucket only until the last page stops naming it; the measurement and how to repeat it are in docs/architecture/how-it-is-built.md. frontend/src/lib/msalConfig.js caches MSAL in sessionStorage, its own default for a single-page app, so a DOM XSS or a compromised dependency can no longer read a token pair that outlives the tab; the known risk is an in-app browser that loses the session across the redirect, the verification is the owner's sign-in on an iPhone after the deploy, and the rollback is that one line. deploy-azure-frontend.yml moves every .map out of dist after the build, keeps them for thirty days as the run's source-maps artifact so a stack trace can still be read, and refuses a payload with one left; frontend/scripts/deploy-source-maps.test.js holds the workflow and vite.config.js to each other.
  • The drift monitor's Terraform row reads the last apply from HCP Terraform, and reads ⚠️ rather than ✅ when it cannot (PLAT-3, #963). The row #1178 added to scripts/check-deploy-drift.mjs took the last successful tfc-plan-check.yml run's head as the deployed baseline. That workflow is a dispatch-only read of a plan and is green with TFC_TOKEN unset, on a plan nobody confirms and on a plan the owner never applies, so infra/ could keep merging while the row printed ✅, and an apply from the HCP Terraform UI with no dispatch in a day read ❌ (found by a read-only review, 2026-10-15). The row now resolves the hcw/hcw-azure workspace and reads its newest run in state applied (filter[status]=applied, page[size]=1, include=configuration_version.ingress_attributes), takes the commit-sha of that run's ingress attributes as the deployed baseline and its status-timestamps.applied-at as the date, and counts the infra/ commits on main after it as the other rows do; the walk through included is check-tfc-plan.mjs's, and the API call is tfc-workspace-busy.mjs's, both imported rather than copied. A baseline commit main does not contain is reported as such and fails the report instead of counting as zero drift; no token, or a read that fails, renders ⚠️ with the reason and fails the job, never ✅. monitor-deploy-drift.yml's check job sits in the automation environment and passes TFC_TOKEN to the step, the way tfc-plan-check.yml receives it; its permissions are unchanged. The entry's cosmos-containers.json path, which named a file that does not exist at the root (infra/cosmos-containers.json sits under infra), is gone, and the report's footer describes three rows rather than "both deploys". Tests cover the token-unset row, an applied baseline with later infra/ commits, a refused read, a baseline off main, a workspace that never applied, a listing whose newest run is not applied, every declared path existing on disk, and the workflow wiring.
  • Recording Hub saves again: the recordings create allowlist accepts the four fields both forms send, typed (AP-B6, #941). #1181 put an allowlist on POST cms/recordings (RECORDING_CREATE_FIELDS in functions/src/lib/admin-integrations.js) and left out source, plaudId, recordedAt and status, which the only two callers send on every save, so "route a Plaud recording" (recordingCards.jsx) and the pasted-transcript form (uploadForms.jsx) both answered 400 from that deploy on; the frontend tests mock the API and could not see it. The four keys are allowlisted and typed: source is one of plaud_mcp and manual_upload, plaudId a non-empty string of at most 200 characters, recordedAt an ISO string or epoch milliseconds stored as ISO, and status is accepted only as new, the value the handler stamps anyway, so the body still cannot choose a recording's state. Tests post each form's payload verbatim and expect 200, so a field that leaves the list again fails here.

  • The content list's search is answered from the index, bounded, and reads both summary casings (AP-F6, #930). The predicate #1178 added wrapped each field in LOWER(), which Cosmos DB documents as not using the index, so every search loaded all ~70k content documents across every partition. It is now CONTAINS(field, @search, true), the case-insensitive form the index serves (a full index scan of the path's distinct values, not a full scan of the container), and the parameter is no longer lowercased. The term is bounded as sort and limit already were: LIST_SEARCH_MIN_CHARS (2) to LIST_SEARCH_MAX_CHARS (100), measured after trimming, outside which the list answers the same { error } 400 an unknown sort field gets; a blank term is no search. Summary, the second casing the module header names, is searched beside summary, so a record whose summary lives only there can match. cms-content.test.js pins the clause, the parameter's casing, both bounds at their edges with a blank term and q, and a row whose summary lives only in Summary. No admin page sends search to the server yet: the Editor list and Live Pages still filter the 500 rows they fetch, so the bound has no UI mirror to add until the search moves server-side.

  • PLAT-9 remainder: the Well-Architected assessment's thirty-seven Planned controls read against the repository and the live estate, and the lab host page's Provisioning and Configuration rows read live (PLAT-9, #969). docs/architecture/well-architected-assessment.md carried Planned on thirty-seven of its forty controls since the baseline; each now carries what exists and the date it was read: six are Verified with their signal captured (service and API authentication, secret protection with three rules seen to fire, the budget's test notifications, observability's cap and billed ingestion, the migration log), fourteen Implemented with the missing measurement named (cold starts, cache-hit ratio, Core Web Vitals, RU by partition, a live anonymous fetch), seventeen Partial with the gap named (no rollback drill or retained Functions package, one HCP Terraform identity for plan and apply, no correlation ids on audit rows, no queue age or depth signal, no dashboards or SLOs), two of them by decision (Private Link, required approvals). The header says what each status means. On the lab host page the Provisioning row says what a session can and cannot read about the hcw/hcw-lab workspace and carries the VPS's Arc state, and the Configuration row's last converge reads from the host's applied-commit.json (2026-10-14 05:08 UTC, de0be17a) instead of a week-old date.

  • PLAT-10 remainder: the Static Web App bandwidth comment reads the live DNS, the runner address is checked before it reaches a firewall argument and the storage window's flip can be measured, Cloudflare's ranges are held to the published list by a weekly pull request, the content account drops its localhost origin, the registration monitor names its real identity and what it counts, and the export window's effect on API latency is an alert rule (PLAT-10, #970). infra/frontend.tf no longer says Cloudflare caching makes a bandwidth overage unlikely: the apex and www resolve straight to the Static Web App (read 2026-10-15), so every byte counts against the Free plan's 100 GB, and alert-swa-bandwidth is what watches the pace. deploy-functions.yml passes the address it learns from a third party through scripts/assert-ipv4.mjs (one dotted quad or the step fails naming what arrived) at all three places it is used, says what Microsoft documents about who writes the package on Flex Consumption, and leaves the host storage account's default action at Deny through the window when the repository variable FUNCTIONS_STORAGE_OPEN_DEFAULT_ACTION is the literal false, so one deploy measures whether the flip is still needed. scripts/cloudflare-ranges.mjs reads Cloudflare's published IPv4 list, refuses a short or malformed answer, and rewrites the default of cloudflare_ip_ranges; cloudflare-ranges.yml runs it on Tuesdays and opens the chore/cloudflare-ranges pull request through the App, with the same recorded-paths guard as the lab pin sets (the set lives in open-lab-pin-pr.mjs; the pull request body says apply before the next deploy, because assert-origin-lock.mjs holds every deploy to the list). infra/storage.tf drops http://localhost:5173 from the content account's CORS: nothing in the frontend fetches from that account by script, and the images and media it loads need no CORS. monitor-functions-registered.yml stops saying its identity can write site config (that split landed as #1098) and counts the host storage account's IP rules under that name, since storage IP rules carry no ci-* names. alert-export-window-latency (infra/observability.tf, severity 3, daily) compares the API's P95 inside the nightly Cosmos export window with the rest of the day and fires when the window is above 1 s and above twice the day's over at least 20 requests: ADR 0019's revisit trigger, measured, as a rule rather than the weekly readout the review asked for, because a readout from GitHub needs a workspace query grant the owner declined on 2026-10-11. The rule is a declared create in assert-expected-plan.mjs until the owner's apply, and has its rows in the alerting runbook and ADR 0019.
  • Completed low-hanging fruit remediations across Admin Portal, VPS agent, deployment drift, docs, and infra (AP-F6 #930, LAB-13 #959, PLAT-3 #963, PLAT-9 #969, PLAT-10 #970).
  • AP-F6 (#930): Added server-side search query parameter support in functions/src/lib/cms-content.js with Cosmos DB CONTAINS, added 500-item truncation notice and retry attempt state in LivePagesPage.jsx, and added unit tests in LivePagesPage.test.jsx.
  • LAB-13 (#959): Extended VPS agent shutdown timeout to 210s (LABS_AGENT_SHUTDOWN_TIMEOUT_MS) with active job terminal completion handling, added systemd TimeoutStopSec=240 to hcw-labs-agent.service.j2 and role defaults, and hash-locked ansible-core wheel SHA256 in lab-host/bootstrap.sh.
  • PLAT-3 (#963): Added Terraform Infrastructure (infra, cosmos-containers.json, tfc-plan-check.yml) to deploy drift monitor (scripts/check-deploy-drift.mjs) and test suite.
  • PLAT-9 (#969): Updated Azure Well-Architected assessment (well-architected-assessment.md) reflecting single Function App consolidation (ADR 0019) and verified recovery drill controls.

  • PLAT-10 (#970): Corrected apex DNS direct routing comment in infra/frontend.tf regarding SWA 100 GB Free bandwidth limits, and added strict IPv4 format validation for runner IP lookup in CI deployment workflows.

  • Lab host: no reboot with a user logged in, the upgrade origins rendered by the role, and job containers with a stated swap bound and an open-files ceiling (LAB-13, #959). The hardening role's unattended-upgrades override now clears and names the allowed origins itself (hardening_unattended_allowed_origins, Ubuntu's four) and sets Automatic-Reboot-WithUsers to false, with a test that renders it; the daily held-package report logs a reboot is pending since <date> (<n> days) at warning while the reboot waits, and the runbook names the owner's window (log out before 04:30, or reboot when convenient). The job runner writes --memory-swap as twice --memory (Docker's default, swap at most the memory limit, made explicit) and adds --ulimit nofile (LABS_AGENT_JOB_NOFILE, default 1024, also in the labs_agent role); both flags are sandbox-controlled, so a capability cannot set them. On the host after the next bootstrap.sh. The rest of LAB-13 (a terminal status for a job killed by a reboot, reboot only when no job runs, a hash-locked ansible-core) stays open on #959.

  • The Terraform busy guard fails closed (PLAT-3, #963). scripts/tfc-workspace-busy.mjs, which deploy-functions.yml runs before any firewall window opens, used to pass with a note when TFC_TOKEN was absent, so from the outside a deploy log read the same whether the guard was off or the workspace was idle. It now refuses the deploy and names the secret to seed, and prints "armed" when it can read the workspace. TFC_TOKEN has been a repository secret since 2026-10-08 (#980), so the deploy job already receives it. The runbook and the workflow comment say so.
  • functions_scm_lock_enabled defaults to true (PLAT-10, #970). The workspace set it true on 2026-08-25, when the SCM window was proven under Deny, and the Function App's SCM default action read Deny again on 2026-10-14, so the code default now says what the estate does and the plan for the change is empty. The comments in infra/variables.tf, deploy-functions.yml and scripts/assert-origin-lock.mjs, and the required-inputs row, say so instead of describing the 2026-08-25 posture.

  • Version floors: the weekly pull request moves the engines.node ranges with the floors, and arrives green. update-version-floors.yml used to move scripts/version-floors.json alone and open a pull request that was red by design, so every Node.js minor left five engines.node ranges for a person to move by hand (#1152). The updater now moves each range the new floors leave behind within its line, in its package.json and the root entry of the package-lock.json beside it, and re-reads the tree to list only what is still behind: an exact pin with a digest or checksum, a runtime_version, a runs-on, or anything on a new line. The commit job judges the patch before applying it, by a rule written in the workflow: the floors file, and otherwise only a "node": "<range>" line of a package.json or package-lock.json. The pull request body now follows .github/pull_request_template.md, and its first item says whether the floors check is the evidence or the work list. scripts/version-floors.test.mjs also fails a lockfile whose root engines.node differs from its package.json.

  • Lab host: a run can be narrowed to the roles that changed. Every role in lab-host/ansible/site.yml carries its own name as a tag, so bootstrap.sh --tags lab_images,coder after a digests merge, or --tags addons after an add-on's digest moved, re-applies one area of a converged host instead of the whole play; the pre-flight checks and the privilege checks run whatever the tags. A narrowed run is not a convergence: neither bootstrap.sh nor the play records it as a completed run, so the site keeps saying the host is behind until the next full one. ansible.posix.profile_tasks prints the twenty slowest tasks after the recap, so a long run says where its minutes went. lab-host/README.md, "Running part of the play", has the tags and the two things a narrowed run is not. Owner ask 2026-10-14, after a run that took most of an hour to move three pins.

  • Lab host: the proxy check reports in plain words. The privilege check used to print the three probes' raw answers, two of which the Docker CLI words as Error response from daemon: … refused … — the proxy doing its job, read as a fault. It now prints one sentence saying what passed; the raw answers appear only when the check fails.

Fixed

  • Lab host: a host firewall for the lab sandbox, the container bridges and Coder's internal network (#954, #958). A new host_firewall role, run after coder_sandbox, adds the host's own HCW-SANDBOX-OUT, HCW-BRIDGE-IN, HCW-CONTROL-OUT and HCW-REFUSE chains, jumped to first above ufw's.
  • The sandbox: the sandbox daemon's user (the daemon and every workspace) may reach DNS on systemd-resolved's stub, TCP 443 on the host's own non-loopback addresses, and TCP 80/443 on public addresses. Host-local, private, link-local, multicast and reserved destinations, and all other traffic, are refused at once, and flows already open to them are cut when the rules first load.
  • The bridges: containers on docker0 and br-* may open only TCP 80/443 to the host.
  • Coder's internal network: the host's own new connections to it are refused; Coder reaches it across its own bridge.
  • Loading: hcw-host-firewall loads each family in one iptables-restore --noflush transaction and keeps each jump first and single. Both families are dry-run before either commits, a jump to an undeclared HCW-* chain is refused first, and a lock in /run keeps one run at a time. The play fails on any read-back difference, and check mode reports drift.
  • Boot: hcw-host-firewall.service re-applies the rules at boot, after ufw and before Docker, and reports a failure to hcw-unit-failed@. The sandbox user's manager requires it.
  • Failing closed: if a run cannot load the firewall, the sandbox keeps running only while the live rules exactly equal the last policy the role verified live (/etc/hcw/firewall/verified); otherwise its manager is stopped and the play fails.
  • Removal: host_firewall_state: absent removes it all in a safe order.
  • Untouched: ufw's rules and the add-ons' DOCKER-USER rules.
  • Tests: roles/host_firewall/tests/hcw-host-firewall.test.sh (real iptables in a scratch network namespace, 106 checks, in CI with ufw required) and scripts/lab-host-host-firewall.test.mjs.
  • No auto-delete on the owner's Ambassador hub (follow-up to #1234, SEC-4 #973). #1234 set a 90-day container TTL on the ambassador Cosmos container, believing it held applicants' personal data. It holds the owner's own programs, applications and evidence, kept until deleted. A container TTL expires every document not written within the window, so it would have deleted the owner's records. The tfc-plan-check flagged it as undeclared before any run carrying it was confirmed, and it is taken off before any apply. generate-cosmos-container-spec.test.mjs now refuses a container TTL on the containers whose records are kept until deleted (ambassador, content, content_versions, blogs). The data-subject runbook keeps the hub in scope as a search of its free-text fields and files, redacting matches and keeping the records. Its content queries and its exercise record, which do not match the estate, are marked under review and unverified until #1237.
  • Lab host: Caddy's admin API moves to a Unix socket, and HTTP/3 is off (#958). The Caddyfile always renders a global block with admin unix//run/caddy/admin.sock instead of TCP 127.0.0.1:2019, and servers { protocols h1 h2 }, so nothing listens on UDP 443 (ufw already dropped it). caddy.service creates /run/caddy as RuntimeDirectory=caddy (0700), and its ExecReload passes the socket as --address, which only the caddy user and root can open. The first run restarts Caddy before any reload, through handler order across the caddy, coder and addons roles, which a test holds; and a running Caddy without its socket is restarted. caddy-adapt.test.sh reads the admin listener and protocols back from the pinned Caddy, and scripts/lab-host-caddy-admin.test.mjs holds the unit, the socket path and the handler order. Runbook: "Caddy: reloading by hand".
  • alert-kv-config-write no longer fires when one of its own alerts is closed (#1216). An alert's state change is logged against its target, as <vault id>/providers/Microsoft.AlertsManagement/alerts/<id>. That sits beneath the vault, and the rule's only filters were category and status. So closing one of its alerts opened another and paged again: on 2026-10-16 the alert cleanup closed three, and three new ones fired at 01:37Z. Activity Log conditions have no "not equals", so the rule now names the providers that make real vault changes: KeyVault, Authorization, Resources (tags, which the fire test writes) and Insights (diagnostic settings). Microsoft.AlertsManagement is not among them. scripts/activity-log-alert-filters.test.mjs refuses any Administrative Activity Log alert without a provider or type filter, or one that lets Microsoft.AlertsManagement through. Live after the owner's hcw-azure apply.
  • The lab backup account's blob diagnostic setting no longer plans a change on every run (#1209). It set log_analytics_destination_type = "Dedicated", which Azure keeps no value for on a storage target. Storage writes only to its own resource-specific tables, so the setting read back empty after its create on 2026-10-14, and every plan since proposed "" -> "Dedicated" again. The plan check named it as undeclared on #1207's merge run (run-qrsPoBGgTrAZyqwa). The attribute is gone. The rows land in StorageBlobLogs as before, as they always did for the site's four other storage settings, which never set it. scripts/storage-diagnostic-settings.test.mjs refuses the attribute on any diagnostic setting whose target is a storage account.
  • The /education page test computes its expectations on the day the page renders, not on the runner's real date (#495, #1092). Since #495, EducationIndexPage.test.jsx has pinned the clock to 2026-09-10 in beforeAll but read today with todayIso() in the file body. Vitest runs a file's body while collecting it, before any hook, so the page rendered on 2026-09-10 while the expectations used the runner's own date. The two agreed only while no row's status differed between those days. AB-400 was the first row that did: availableDate 2026-10-16 and statusReadDate 2026-10-12 make it upcoming on the frozen day and active from 2026-10-16. From midnight UTC that day the Linux runners expected 49 bookable Azure exams and rendered 48, which failed coverage (qlty) on run 38834846071. The frontend row passed only because its path filter skipped its tests on that docs-only pull request. A machine on Central time still read 2026-10-15 and passed. today is now todayIso(FROZEN_NOW), worked out from the frozen constant, and a new first test renders useToday(earliestDataAsOf()), the call the page makes, and requires it to return that same day. A clock mismatch now fails on the first day the two clocks differ, not on the day a status happens to change. The page needed no change: it reads one useToday and passes it everywhere. A local matrix of four collection-time clocks and six zones, UTC−12 to UTC+14, failed the old file in 19 of 24 cells; the new file passes all 24.
  • Monitor Functions Registered no longer fails on a storage window that a running deploy holds (#1205). On 2026-10-15 it read the host storage account at 22:45:38Z, while deploy run 38819173840 held its upload window (22:45:18Z to 22:47:35Z). It reported Allow and one IP rule as ❌, and the window then closed on time. Now an open window is believed only once no deploy-functions.yml run is in progress. While one is, the monitor reads the firewall again every 30 s for up to twenty minutes and passes as soon as the window closes. With none running, it reads again a minute later. An orphan stays ❌. A deploy still holding the window after the wait is ⚠️ and named, and so is one GitHub stops answering about for three polls. A Node crash is not taken for a deploy in progress. scripts/check-functions-deploy-in-progress.mjs answers the question, with a 15 s timeout and one retry. Its test runs the monitor's own step under bash -e, against a stubbed az, through that collision and the cases around it. The monitor does not join the deploy's concurrency group. There it would queue behind a deploy waiting on approval, and under the default queue a newer pending run cancels the older one.
  • Operator documentation drift (PLAT-9, #969). infra/README.md said not to pre-build hub networking while hub.tf existed; the cost analysis called the Management budget "declared, not applied" after the apply had landed (read live 2026-10-14); ADR 0028 section 7 said host storage stays LRS when ADR 0031 (2026-09-14) chose GRS and Terraform sets it; and the lab host page's summary and identity rows still read "planned" for a host that has run since 2026-09-26. Each line is corrected with the read that corrects it, and a docs-build gate, scripts/docs/check_stale_state.py, fails the build on the stale phrases the review found an operator acting on ("State: planned", "not yet armed", "delivery is unproven").
  • The route error boundary recovers on navigation, and a failed sign-in shows its code (AP-F7 #931, AP-F8 #932). One render error on any admin route kept the failure card on every admin route until a hard refresh, and always blamed stale build chunks; the boundary now resets when the path changes, tells a failed dynamic import from a runtime error in its copy, and shows the error message on admin routes. The sign-in callback page said "the browser console has the detail", which a phone cannot open; it now shows the Entra or MSAL error code with a Copy button that puts the code and the full message on the clipboard, while the message itself, which names the authority and client id, stays off the screen. App.boundary.test.jsx and AuthCallbackPage.test.jsx pin both.
  • Admin dashboard and Editor list say when a read failed (AP-F5 #929, AP-F6 #930). The dashboard rendered "You're all caught up" over zero counts when the snapshot call failed, with a small red line beneath; it now shows an error state with Try again, and a later failed re-read keeps the last numbers with a "Numbers may be stale" badge and a Retry button beside the reason. The Editor list ignored its three reads' errors, so a 401 or a 5xx rendered as "No items match your filters"; it now shows the failure with Try again, which re-reads all three boards, and a board that comes back exactly at the server's 500-item limit carries a banner saying older items are not in the list or its search. DashboardPage.test.jsx and a new EditorListPage.test.jsx pin both.
  • The Publer proxy forwards only the paths and methods the Social Hub sends, and every write through either proxy leaves an audit row (AP-B3, #938). publerProxy forwarded any verb to any Publer path with the server's key and no trail, so an editor session could reach accounts, members and workspace settings the Social Hub never touches. integrations-http.js now names its five endpoints and three methods (GET, POST, DELETE), and rest-proxy.js records every non-GET call, refused ones included, in admin_audit_logs with the actor, the path as sent and the outcome. rest-proxy.test.js sends every request the pages make and asserts an unlisted path is refused for every verb.
  • Landing zone HCL: the AWS and GCP emitters no longer hand state to generators that take none. Eight calls in each of frontend/src/lib/landingZone/hcl/aws/index.js and .../gcp/index.js passed the normalised state to functions declared without a parameter, which CodeQL raised as sixteen js/superfluous-trailing-arguments alerts after #1157. The arguments are dropped; the emitted files are unchanged.
  • Lab host: the add-on secrets are read with the vars lookup. The addons role built each add-on's secret environment with the vars dictionary, which ansible-core 2.21 warns about on every run and 2.24 removes; it uses lookup('vars', …) now, so the run prints no deprecation warning and does not break on the next ansible-core.

  • Lab image: the pins the weekly scan could move (#1133). The first scan the check job filed (2026-10-14) named fixable HIGH/CRITICAL findings in the images the host runs. Four of them have a publisher release to move to, and lab-image/versions.env moves them with the sums read as the file documents: terraform 1.16.4 → 1.16.5 (releases.hashicorp.com SHA256SUMS), the azurerm provider 5.7.0 → 5.9.0 and azapi 2.12.0 → 2.13.0 (the registry's download/linux/amd64 shasum), and the Azure CLI 2.90.0 → 2.91.0 (published in both the bookworm suite the full target installs from and the resolute suite the sandbox template uses), which is the bundle that carried PyJWT 2.13.0, cryptography 48.0.1, msgpack, setuptools 70.3.0 and urllib3 2.7.0. The rest of the table — azurerm 4.81.0, alz 0.22.0, random 3.9.1, modtm 0.4.0, time 0.14.2, kubeconform 0.8.0, helm 4.3.0, kubectl 1.37.1 — is already each publisher's newest release, so those Go-library findings have nothing to pin yet, and the Debian lines (openssl, libpcre2 ~deb13u3) arrive with the base digest the weekly bump moves. The README's mirror line follows.

  • The lab agent's Docker proxy follows a daemon restart. The proxy (hcw-labs-agent-docker-proxy, LAB-5) holds the daemon's socket as a bind-mounted file, so when a bootstrap run restarted the daemon under it — the 2026-10-14 run that took Docker Engine from 29.8.1 to 29.9.0 — it kept the old, unlinked socket inode: its own refusals still answered and every forwarded call got HAProxy's 503 Service Unavailable, and the run ended at the privilege check. The labs_agent role now installs /usr/local/libexec/hcw-docker-proxy-stale, which compares the proxy's start with docker.service's ActiveEnterTimestamp, restarts a proxy that predates the daemon's current start, and then requires GET /_ping through the proxy (ansible.builtin.uri on its Unix socket) to answer 200 before the play goes on. The privilege check's failure message names the symptom. Tested against stand-ins for systemctl and docker in ansible-lint (lab-host).

  • The lab supply-chain issue fits GitHub's body limit. The weekly check job scans the two published lab images with Trivy and files its findings as one issue; on 2026-10-14 the full tables ran past the 65,536- character body limit (Body is too long, run 38493246343) and the week's findings were filed nowhere. The scan now writes JSON once per image, the full table (trivy convert) goes to the run summary as before, and the issue gets a compact per-package table — totals by severity, up to forty packages with installed and fixed versions and their first CVEs, a pointer to the run for the rest — behind a hard cut at 60,000 characters that names the run summary. lab-pin-pr-workflows.test.mjs pins the two renderings and the cut.

Added

  • Lab host: an alert on an unexpected SSH login (#953, LAB-7). alert-lab-ssh-login (severity 1) pages when sshd accepts a login that is not hcwadmin by public key with the pinned key (lab_ssh_admin_key_fingerprints). Once the change that closes port 22 sets lab_ssh_loopback_only, it also pages on any login that did not arrive through Arc SSH. It parses OpenSSH 10.2's Accepted line, drops VERBOSE's Accepted key/Accepted certificate follow-ups, and pages on a line it cannot parse. The rule is stateless and unmuted (ADR 0022 decision 6): rows are chosen by ingestion time over 20 minutes and evaluated every 15, inside a one-day window, so no login is missed (even one ingested late after an agent backlog) and one may mail twice. A test keeps the fingerprint variable equal to the key pinned in group_vars, and Invoke-AlertFireTest.ps1 -Test lab-ssh-login fires it.
  • Scanner triage: a CodeQL section (SEC-5, #974). The triage standard covered Qlty, zizmor, checkov and Trivy but not CodeQL dismissals, of which 151 exist: 112 under the vendored harness, dismissed on 2026-08-21 without per-alert notes, and 39 first-party. The section states the rule (every dismissal carries a one-line evidence note, the owner dismisses), records the harness dismissals as a class closed by the codeql-config.yml exclusion, and writes the evidence for the two first-party dismissals that have no note (#398, #388).
  • Lab host: the way back from the off-host backups (LAB-9, #955, PR 3). hcw-backup-restore, installed by the backups role, lists the host-backups container as the Arc identity, fetches the newest or a named blob to /var/backups/restore/ by a .partial name until the listed length has arrived, and with --yes restores a dump into coder-postgres (stop, drop and recreate, load, start) or a snapshot into Vault (raft snapshot restore -force, the token on stdin). The reinstall runbook's step 7 gains the restore as its last act and the Last rehearsed / elapsed line, "not yet" until the owner rehearses on a fresh VPS. Two corrections from the first seed on 2026-10-14: Vault 2.0 made generate-root need a token as well as the recovery keys (CVE-2026-5807), so "The root token" now carries the temporary enable_unauthenticated_access switch and its removal; and the encoded token is read from what the third key prints, since the attempt's status is cleared on completion. Tested with stand-in curl, docker and vault.
  • Lab host: the backups leave the VPS (LAB-9, #955, host side). A new backups role, last in lab-host/ansible/site.yml, uploads the nightly Coder dump as it is written and a nightly Vault raft snapshot (hcw-vault-backup.timer, 03:45 UTC) to the host-backups container of stlabhybridprodcus01, as the Arc machine's identity: hcw-arc-token performs the agent's challenge, hcw-backup-upload puts one blob under coder/ or vault/ and logs one line tagged hcw-backup-upload, and no storage key is on the host. The snapshot token is the owner's one seed (hcw-vault-backup-seed: the owner's Vault token on stdin, a periodic orphan token under a snapshot-only policy, stored root-only), renewed by the unit itself; until it is seeded the unit is skipped, not failed. The role lists the container as the identity at every run and refuses the run when it cannot. The owner's line is in docs/runbooks/labs-host.md, "Off-host backups"; the restore and the rehearsal are the next pull request on #955. The seven creates #1147 declared in scripts/assert-expected-plan.mjs are removed, applied on 2026-10-14.

  • Lab host: where its backups go (LAB-9, #955; owner decision 2026-10-14). infra/lab-hybrid-backups.tf: a Standard GRS, Cool, Entra-only storage account in rg-lab-hybrid-prod-cus with one container, host-backups, under a 35-day time-based immutability policy — the host can write a backup and cannot delete or overwrite one inside the window — with versioning and 35-day soft delete behind it; the Arc machine identity's second grant, Storage Blob Data Contributor on that container only; blob read, write and delete logs to the Management workspace; and one alert per backup kind (coder/, vault/) when no blob was written in two days. The account name is an output for the host's group_vars. ADR 0032 gains the amendment of 2026-10-14, which also records the owner's choice over the VPS provider's object storage and local-only, and why uploads are not encrypted client-side. The host side (upload after the nightly dump, a nightly Vault raft snapshot, the restore helper) and the runbook's rehearsal line follow in their own pull requests once this has applied.

  • Lab pin bumps carry their dependents (#949). Three files are held to the pins the weekly bumps move, and a bump that moved the pin alone failed its own pull request's checks on 2026-10-13 (#1122 scripts (operations) and ansible-lint (lab-host), #1123 frontend). Now the host bump that moves Caddy also moves CADDY_SHA512 in lab-host/ansible/roles/caddy/tests/caddy-adapt.test.sh, from the release's caddy_<v>_checksums.txt; the one that moves Docker Engine re-records scripts/fixtures/docker-cli-agent-requests.json with the pinned docker-ce-cli, installed into a temporary directory from Docker's apt index — the signing key refused unless it hashes to docker_apt_key_checksum (the value the docker role holds the host's keyring to), the InRelease verified against that key with gpgv, the index by the InRelease, the .deb by the SHA256 the index lists (scripts/lab/record-docker-cli-fixture.mjs, a step of lab-supply-chain.yml that exits with nothing written when the fixture already matches); and the image-base bump rewrites the base tag and CPython release the Docker guide page names (frontend/src/pages/docker/BuildingImagesPage.jsx), and sets that page's CHECKED_ON to the day of the bump, since the pull request's frontend check is the read of the page against those files. The three files join their set's paths in open-lab-pin-pr.mjs, whose pull request body now carries the template's sections in order with only the true boxes ticked (CodeRabbit's Description check warned on #1122 without them). The tests that named those values by hand — lab-pin-bumps.test.mjs against the live group_vars, DockerGuides.test.jsx against python:3.14.7-slim-trixie — read them from the files instead, so a bump pull request is green on its own and a merged bump does not break main.

  • Recording Hub: a Stats tab with the show's Podbean statistics. Closes #1115 (the second of two slices; the first added the Distribution tab's live status and downloads). The show's figures lived only on Podbean's site; the hub now reads them through the same client (ADR 0029 §1c).

  • GET cms/podcast/host/stats?range=30d|90d|12m (editor; lib/podcast/host-stats.js): six independent Podbean reads over the range's window — downloads by day (30d, 90d, period: 'd') or by month (12m, period: 'm', from the first of the month eleven months ago), countries, platforms, sources, monthly listener retention and the audience counts (followers, likes, comments, totalEpisodeLengthSeconds). A failed sub-read is null in its place with its recorded error under errors.<name>; the others still answer and the response is 200. A range not offered is 400 before any read. Cached 60 minutes per range in admin_config/podbean_host_stats_<range>; ?refresh=1 reads live except while the range's cache is under a minute old, and the document carries the client's accountKey, both as the overview does. Not configured is 200 { configured: false, range, reason } with no call.
  • lib/podcast/host-cache.js: the read-through admin_config cache the overview and the statistics now share — served while younger than the TTL; on refresh, served while younger than the floor (read from the shared document, so every session and instance together make one live read a minute); a miss for any other Podbean account; written create-only first and etag-conditionally after; logged by error code and never a dependency when it cannot be read or written. host-reads.js moved onto it with no change in behaviour.
  • Stats tab (?tab=stats; statistics, downloads, analytics and listeners land there): a 30 days / 90 days / 12 months picker, the total downloads and a bar chart with one bar per day or month (zero where nothing was reported, so a quiet day is a bar and not a gap), top countries (named), platforms and sources as ranked lists with share % and an Other row past eight, retention as a small table with each point as a percentage, and the audience counts as tiles. Loading, not-configured, failed-read, per-section failure and "no data yet" states each say so. The chart is recharts, as AI Engine → Usage uses.
  • Two documented-but-unmeasured shapes are taken either way: a monthly download key of YYYY-MM is a month and YYYY-MM-DD folds into its month; a retention value at or under 1 is a fraction and above 1 a percentage. Both are stated in statsView.js and pinned by its tests, so the first live reading has a test to correct.

  • Contract and inventory: .azure/api-surface.json, functions/function-inventory.json (getPodcastHostStats).

  • Recording Hub: Podbean's live status and downloads per published episode. Part of #1115, re-scoped 2026-10-13 after Podbean's MCP server was measured unusable from a web app (redirect allowlist; a developer-app token gets 401 at /mcp). The Distribution tab showed only what the publish job recorded; it now reads Podbean through the client the publish already authenticates with (ADR 0029 §1c).

  • podbean.js gains the reads: getEpisode, getDownloadStats, getCountryStats, getPlatformStats, getSourceStats, getUserRetention and getAnalyticReports, every one a GET through the same call() with the token as access_token. Dates (YYYY-MM-DD, a real day, start on or before end) and periods (d|m, m|w) are refused as VALIDATION before any request. types[] is sent as a repeated parameter; the shapes are Podbean's documented ones and none is measured live yet — the client tests pin the documented requests.
  • GET cms/podcast/host/overview (editor; lib/podcast/host-reads.js): every transcript whose host.podbean.episodeId is set, joined by id to one page of the show's newest 100 episodes, plus the listed episodes no transcript names (transcriptId: null — the three published by hand). Per episode: Podbean's title, status, permalink, publish time, and the last 30 days of downloads ({ total, byDay }), newest first and at most 20 statistics reads per overview (downloadsCapped counts the rest). A Podbean failure lands where it happened — errors.show, errors.episodes (then on every transcript row, and no statistics are read), or error on one row — in the publish job's recorded-error shape, and the answer is still 200. Cached 15 minutes in admin_config/podbean_host_overview; ?refresh=1 reads live, except while the cache is under a minute old, so every editor session, tab and instance together make at most one live read a minute against the bucket publishing shares (the floor is read from the shared document). The cached document carries the client's accountKey (sixteen hex digits of SHA-256 over the client id, never the id), so settings pointed at another Podbean account make it a miss rather than serving the previous account's rows until expiry. A cache that cannot be read or written is logged content-free and the live overview is served. Not configured is 200 { configured: false, reason } with no call. Logs carry counts and codes, never a title or the token.
  • Distribution tab: under each "Published to Podbean" record, a Podbean line with the live status badge (Live / Draft on Podbean), the episode page, "N downloads, last 30 days" and a 30-bar sparkline; an "Also on Podbean" group for the episodes this hub never sent; a "Podbean read (cached)" status line with Refresh. Loading, not-configured and failed-read states each say so; a failed read shows no stale counts. The overview is read when the tab first mounts, not on page load.

  • Contract and inventory: .azure/api-surface.json, functions/function-inventory.json (getPodcastHostOverview).

  • MCP Disconnect revokes the grant at the provider (RFC 7009). Closes #1116, raised by CodeRabbit on #1114. Disconnect on an OAuth MCP server deleted the stored token pair and left the grant live at the provider until its refresh token expired or the owner revoked it there.

  • Discovery keeps the revocation_endpoint the authorization-server metadata names (RFC 8414), https only, on the pending sign-in and then on the stored connection (oauth.revocationEndpoint), beside the token endpoint — so, like the token endpoint, it is set only by the flow and never by a config write. A missing or non-https one is stored as none; it is not a reason to refuse the sign-in. Podbean advertises https://mcp.podbean.com/revoke (read 2026-10-13); Replicate and Hostinger did not when read.
  • Disconnect revokes first, then deletes. The refresh token, then the access token, each POSTed as token + token_type_hint with the client authentication the registration chose (client id in the form for a public client; Basic or in the body for a confidential one), through the same guarded request as every token call: https only, no redirects, one deadline. The local delete happens whatever the provider answers, so an outage cannot keep a token on this side. The response carries revoked (true, false with a short reason, or null when there is nowhere to revoke) and issuer; the audit row records both. No token reaches a log or the response.

  • The card says which. "Disconnected and revoked at mcp.podbean.com", or "Disconnected here; the grant may still be live at mcp.podbean.com — revoke it in your mcp.podbean.com account". A provider without a revocation endpoint gets the plain "disconnected" toast as before. The Podbean card's notes and the Connect runbook say the same.

  • Podbean's MCP server in the AI Engine (Admin → AI Engine → MCP Servers). The show moved to Podbean on 2026-10-13, and Podbean runs a hosted MCP server at https://mcp.podbean.com/mcp. It covers the show's episodes, its details and its listener statistics.

  • Sign-in: read that day, the server advertises RFC 9728 and RFC 8414 discovery, PKCE (S256), none client authentication and a registration_endpoint. So the existing Connect flow signs in with no key: press Connect, sign in at Podbean, approve. The token is kept server-side and renewed by the MCP token timer, like Hostinger's and Replicate's.
  • Ships switched off. Its tools can publish, edit and permanently delete episodes.

  • Where it is recorded: seed.js seeds it (the AI Engine page adds a missing default on load). The card treats it as built-in. The credentials register lists its OAuth token as mcp-podbean. The tests pin it among the Connect servers.

  • podcast.hybridcloudworks.com for the show's Podbean website. Hybrid Cloud Insights moved from RSS.com to Podbean on 2026-10-13.

  • The feed: it is now https://feed.podbean.com/hybridcloudinsights/feed.xml. RSS.com's feed and Podbean's first address (spatino) both answer 301 towards it, and all three episodes keep their GUIDs.
  • The record: cloudflare_dns_record.podcast_podbean points podcast at pbcdn2.podbean.com, DNS-only like docs, because Podbean issues the certificate itself.
  • Why a subdomain: Podbean's "Use a domain I own" page prefills www, which is the site. The comment in infra/frontend.tf and the edge DNS runbook say so, so the podcast never takes www or the apex.

  • The plan checker declares the one create.

  • The hourly functions monitor names a ci-* access window on the Function App whose run has ended (#1050). deploy-functions.yml and publish-content-manifest.yml open a one-host Allow rule for their runner and close it in an always() step, which does not run on runner loss or a force-cancel. assert-origin-lock.mjs accepts any single-host ci-* rule, because a concurrent run may hold one, so an orphan was invisible.

  • scripts/check-ci-access-windows.mjs matches each ci-* rule on the site and the SCM site to its run by the id its name ends in, and names any whose run has ended or does not exist. Each orphan comes with the exact az functionapp config access-restriction remove command.
  • monitor-functions-registered.yml runs it every hour, reading twice a minute apart so a run closing its own window is not named, and shows it as one more row: ✅, ❌ or ⚠️.

  • The 2026-08 review's T-713 "FIXED" counted only the storage account's rules. The Function App's ci-* rules it named had no probe until now, and the review carries a dated correction.

  • Labs & Architectures: per-lab VPS bootloaders, Linux container sandboxes, Defender WSL container lab, and shared mini landing pages (#1080, #1082, #1083, #1084).

  • Shared mini landing page (#1084): MiniLandingFrame (frontend/src/components/shared/MiniLandingFrame.jsx) extracts the mini-header bar, status pills, metric strip, and section tabs into a reusable shell. ArchitectureDetailTemplate wraps every architecture detail view in MiniLandingFrame, and frontend/src/data/architectures.js adds azureArchitectures (7 Azure architectures led by secure-2-tier-azure plus all 6 blueprints from /azure/architecture linked via detailSlug), complete awsArchitectures detail entries, and architecturesByProvider so /azure/architectures/:slug and /aws/architectures/:slug render full interactive detail pages with Mermaid topology, simulator controls, FinOps telemetry, and operational playbooks.
  • Per-lab VPS bootloaders (lab-image/bootloaders/): hcw-boot <lab-id> dispatches to isolated per-lab bootloader scripts (terraform-validate-walkthrough.sh, ansible-syntax-check-walkthrough.sh, landing-zone-builder-output.sh, wsl-containers-walkthrough.sh, defender-wsl-plugin-walkthrough.sh, plus the guided Add-On bundle bootloaders migration-assessment.sh, network-assessment.sh, and cloud-assessment.sh using python3 -m zipfile -e) baked into docker.io/hybridcloudworks/hcw-lab under /usr/local/libexec/hcw-boot/. lab-host/coder/templates/hcw-lab/main.tf adds the lab_id parameter and runs /usr/local/libexec/hcw-boot/hcw-boot "$LAB_ID" before starting the Coder agent, lab-host/coder/launcher/launcher.js passes rich_parameter_values (lab_id) when creating a workspace, and hcw-coder-template-push accepts an optional [template-name] argument (defaulting to hcw-lab).
  • Self-hosted Docker Sandbox CLI & Linux container lab (#1080, #1082): SandboxSection and /docker/sandboxes add a Run in Browser / Self-Hosted VPS toggle alongside Run on Your Machine (Docker Pro/Team local CLI), and wsl-containers-walkthrough in the lab catalogue (frontend/src/data/labs/catalogue.js) adds a live VPS terminal mode (liveEnabled: true) alongside its browser simulator.
  • Defender for Endpoint WSL 2 Linux plugin container lab (#1083): defender-wsl-plugin-walkthrough joins the lab catalogue as a Linux container lab (no Windows VMs) with DefenderWslSimulator (frontend/src/components/labs/DefenderWslSimulator.jsx) and the defender-wsl-plugin-walkthrough.sh VPS bootloader (mdatp health, mdatp connectivity test, and tvd-scan mocks).

  • AddOn lab-host pre-wiring: lab-host/ansible/group_vars/all.yml pre-wires network-assessment (127.0.0.1:18082, with always-on guided workspace handoff settings NA_PUBLIC_BASE_URL, CODER_URL, CODER_TEMPLATE_ID: na-guided-lab, NA_WORKSPACE_TTL_MINUTES: 240) and cloud-assessment (127.0.0.1:18083) with image_digest: "" (skipped until digest-pinned images are published).

  • Ambassador hub: AWS Community Builders, GitKraken's real quarterly rule, and what keeps a membership active (owner request 2026-10-11).

  • AWS Community Builders joins the seeded catalogue, from its program page:
    • the waitlist link;
    • open to every AWS builder, with a limited intake each year;
    • the yearly January cycle, noted as "edit to match".
  • GitKraken Ambassador takes its rule from the program's quarterly check-in of 2026-10-08:
    • Two or more content pieces a quarter, each logged on the program's contribution tracker, on that quarter's tab.
    • Its criteria are the check-in's focus: review sites, Reddit, articles that use data, shared GitKraken content, and videos on Merge Conflicts, Dev AI Use and Commit Graph.
    • The stored program updates itself on the next read, field by field, wherever it still holds the first seed's guess (SUPERSEDED_SEED_VALUES). A field the owner edited stays as it is.
  • Upkeep (upkeep): a program can say what keeps a held membership active, as a minimum of evidence filed under it in every calendar quarter or year. A card whose membership is Active shows this period's count and its due date, and Details shows this period and the last. Only evidence filed under the program counts; general evidence says nothing about it.

  • Contribution tracker (contributionTrackerUrl): the link to where a program wants contributions logged. It is set on Settings and stored with the program only. No seed carries one, and a test holds the seed file free of shared-document links: such a link is private to a program's members, and this repository is public.

  • Alerting: one script runs each PLAT-4 fire test and proves the alert fired from Azure Monitor, not the inbox (#964). "Making each one fire once" in docs/runbooks/alerting-and-support.md was ten long manual blocks whose proof was a mail arriving. scripts/alerts/Invoke-AlertFireTest.ps1 -Test <name> runs one block as written and puts it back.

  • Same stop conditions, refusing and changing nothing when one fails; the same record files in the home folder (plat4-rbac-test.json, plat4-roledef-test.json, plat4-poison-test.json, plat4-swa-test.json, plat4-timer-test.json), written before the change they describe, with Out-File -NoClobber where the runbook makes the record the lock. Every change is put back in a finally, so a failure or Ctrl+C part-way still restores, and a record is removed only once the runbook's own read shows nothing left; otherwise it is kept and the run exits 3. -Recover does a test's recovery from its record; -WhatIf runs the reads and prints every change as the one line it would run.
  • The proof is Alerts - Get All (Microsoft.AlertsManagement/alerts, api-version 2019-03-01, the version Microsoft's reference documents), filtered to the rule's id over the last hour and read once a minute. PASS <rule> fired at <time> needs an alert that started after the test did; alert-timer-overdue's must also name healthPulse in its context. Anything else is FAIL <rule>: no alert within <n> minutes and exit 2. The run ends with the Fire record rows to paste.
  • Checks the runbook leaves implicit, each a read: az's default subscription is sub-app-site-prod-cus; every rule exists before anything changes; swa-bandwidth saves nothing unless the threshold reads 2666666666; lab-agent-offline refuses unless FEATURE_FLAG_CHECK_AGENT_HEALTH is true.

  • scripts/alerts/Invoke-AlertFireTest.Tests.ps1 (Pester 5) tests the parser, every stop condition, the lock, -WhatIf, and each put-back when the action fails, against a mocked az, and runs the whole script against a stand-in az that refuses every write. Repository Policy's Pester step now runs scripts/alerts. The runbook's section opens with a table of the ten run lines and the seven recovery lines.

  • The migration add-on moves to v0.3.1. The migration row in lab-host/ansible/group_vars/all.yml pins tag 0.3.1 by its image index, sha256:88b8d088…, published by the add-on's v0.3.1 release. The release adopts the upstream explorer 0.3.1 from the first live run with a portal export: the questionnaire keeps focus while typing, the destination region is a select over the Azure regions, and the portal export's RESOURCE LINK column yields resource ids, so no re-export is asked for. Runbook and the pair's architecture page carry the new tag and digest.

  • The migration add-on is turned on for the lab host (ADR 0035, #1064). The migration row in lab-host/ansible/group_vars/all.yml now carries:

  • image_digest set to v0.3.0's image index, sha256:175622db…. Tag 0.3.0 points to it, the add-on's publish run recorded it for pinning, and its build-provenance attestation verifies against the add-on's repository.
  • The hcw-addon-migration verification widget's site key, in place of the provider's test key, which the role refuses to deploy.

The image was run read-only, with every capability dropped and the row's environment, and it answered /api/health 200 ("ok": true, version 0.3.0, verification required). The lab-host README, the labs-host runbook and the app-to-addon pages record the widget and the digest, and drop their [VERIFY] and [REVIEW REQUIRED] markers for this row. The next bootstrap deploys it, once the widget's secret key is in the vault as vault_addon_migration_turnstile_secret.

  • Hybrid Lab: an hourly canary runs one real lab job end to end, and a refused Coder status token is raised (#1009, "PR 2: prevent recurrence", items 2 and 4). The admin Labs probe enqueues and cancels, so it passed with no agent at all while no lab job ran for days; and Coder refused the status token for about 32 hours with only a card saying "unknown".
  • labCanary (functions/src/lib/labs/canary.js), hourly at 20 past, behind FEATURE_FLAG_LAB_CANARY (LAB_CANARY in enabled_timers, off until the owner arms it): one shell-echo job with the payload hcw-canary <job id>, enqueued as enqueueLabJob writes one, waited for up to 150 s, passed only when the agent echoes the payload exactly, then deleted under its ETag. Nothing is enqueued when no active agent registered for shell-echo is online. A job still queued at the deadline is cancelled and deleted; one still claimed is left to finish, and the next run deletes it (or withdraws it past the claim lease) and enqueues nothing new meanwhile. Each run is recorded with its claim and run times in admin_config/lab_canary, where the job's id is reserved under the record's ETag before the job is created, so a run that dies or cannot write its record never loses a job to the next. The record also names its owning run (activeRun), so an overlapping run touches nothing and a run's last write never clears another's reservation.
  • Every read of Coder with CODER-STATUS-TOKEN records Coder's answer in admin_config/coder_status_token (recordTokenAnswer, ETag-guarded, an unchanged answer at most hourly, one outage keeping its start), kept per operation: the labs status read and the Integrations card's read of the token's own record, so a refusal is cleared only by the same read succeeding. A 401 writes a warning starting coder-status: Coder refused CODER_STATUS_TOKEN (401) for a log alert to match; a 403 now says it lacks a read scope rather than "expired or revoked". checkAgentHealth, which is armed, runs checkTokenIfDue and so asks Coder directly once an hour, never through the anonymous minute cache, whether or not the canary is armed, so warm-cache traffic cannot hide a refusal; visitors' live reads record their answers too.
  • Two more Health Hub probes, recorded by the pulse and shown by the Labs snapshot: Lab job canary (lab-canary; offline when nothing ran the job, critical when it ran and failed, unknown once the canary stops) and Coder status token (coder-token; critical from the first refusal until Coder accepts it again, unknown after a day without a read).

  • The timer is in local.timer_catalogue, the enabled_timers validation, function-inventory.json and alert-timer-overdue's schedule map; the plan check declares the new flag, its arming, and the rule's query with the extra row (rendered with terraform console and checked against the map by a test). No alert rule is added; the alerting runbook lists the two queries a follow-up rule would use.

  • Hybrid Lab: the site raises a host that lags main, and a Coder template the host did not publish (#1009, "PR 2: prevent recurrence"; the heartbeat half of LAB-4, #950). On 2026-10-08 the host was six merged pull requests behind and nothing said so. It raises; nothing here re-runs the playbook.

  • The host records what it converged from. site.yml's last task, after every role, handler and privilege check, writes /etc/hcw/applied-commit.json (root:root 0644; the commit, its committer date, the time the run finished; lab_host_applied_commit_file). A failed run leaves the previous record and a --check run writes nothing.
  • The agent reads it on every heartbeat (vps-agent/lib/applied-commit.js) and sends it as applied; heartbeatAgent checks it (a full sha, two ISO dates, none more than ten minutes ahead of the site's clock) and stores it on the agent's registry document only when it changed.
  • checkAgentHealth reads main's commits under lab-host/ and vps-agent/ from GitHub's public REST API once an hour, from the oldest commit an active host reports, into admin_config/lab_drift (functions/src/lib/labs/drift.js), through the Drafts import's pinned, redirect-refusing, size-capped fetch. No token and no new secret: the repository is public. The Functions app's own build commit was not used, because deploys are by hand and say nothing about main. A failed read keeps the last list and records why (RATE_LIMITED, TIMEOUT, …), and the run's marks and messages come first and never wait on it.
  • Two Health Hub probes, recorded by the health pulse every five minutes and shown by the Labs snapshot, judged by the same functions (functions/src/lib/labs/lab-checks.js): Lab host runs main (lab-drift) is critical once a host-path commit has waited more than 24 hours unapplied, or when an active host has never reported a commit; Coder template published (coder-template) is critical when the template Coder serves is not the converged commit's, from two digests the host's daily Coder upkeep report now carries (templateDigest, what push-template last published; templateSourceDigest, the checkout now), and when the automation never published one. A stale read of main, or a report without the digests, is unknown, never healthy.

  • The Labs Agents tab shows each host's commit <8 hex> · converged <time> and the drift verdict, in its colour.

  • Eleven alert rules for what the estate collected and nothing paged on (PLAT-4, #964). All route to ag-plat-prod-cus-01, and all read data that already arrives, so none adds ingestion.

  • alert-timer-overdue: one rule over all 25 timers, one alert per timer through a timer dimension, firing when a timer due at least 45 minutes ago has no successful request row since. The schedules live in local.timer_schedules (period and anchor). The new functions/src/functions/timer-alert-schedules.test.js holds that map to the real app.timer() registrations, to each NCRONTAB and to function-inventory.json. More than five overdue at once collapse into one alert, and the rule is quiet while no request telemetry arrives at all, so a capped workspace or a dead host does not send 25 mails and 25 texts.
  • alert-lab-agent-offline: checkAgentHealth's decision to tell the owner an agent has been offline five minutes, so the owner hears by mail and SMS when Telegram is what failed. The nightly reboot does not fire it.
  • alert-jobs-poison: a successful PutMessage to platform-jobs-poison, from the storage queue logs already shipped.
  • alert-swa-bandwidth: a metric alert on the Static Web App's BytesSent over a day, at a pace that would use 80% of the Free plan's 100 GB month.
  • alert-kv-errors and alert-kv-data-write: Key Vault requests answering 400 or above (past the routine 401 challenge and unseeded-secret 404s), and data-plane secret or key writes by anyone but the Function App, on both vaults.
  • alert-kv-config-write, alert-rbac-write-app and alert-rbac-write-mgmt: Activity Log alerts, which are free and need no export, for Resource Manager writes to either vault and role assignment or definition changes in both subscriptions.
  • alert-lab-vault-unwrap: the seal key wrapped or unwrapped from an address the lab host has not heartbeated from, read from the host's own Heartbeat rows rather than copied from the hcw-lab workspace.
  • alert-lab-ssh-burst: failed SSH lines in 15 minutes at least 50 and four times the day's quarter-hour average.

  • Each create is declared in scripts/assert-expected-plan.mjs. kql-alert-columns.test.mjs now reads every .tf file, not observability.tf alone. terraform-identity-grants.test.mjs classifies the Activity Log alert and admits a whole subscription in its scopes only (read, not written; ADR 0005's step-two table records it). Every query was parsed and type-checked with Microsoft's Kusto.Language parser. Estimated cost is about USD 4.70 a month. The alerting runbook's new The PLAT-4 rules section has the operator table, three pre-apply checks, a fire test for each rule and a fire record. A daily digest of denials needs code and is left as a follow-up.

  • ADR 0035 accepted (2026-10-10, #1044). The owner confirmed decision 8 (AddOn containers run beside Coder on the lab host, amending ADR 0032) and decision 9 (allow-downloads on AddOn panes): the migration catalogue row now grants navigate and downloads, so its sandbox is allow-scripts allow-same-origin allow-forms allow-downloads and the report bundle saves from inside the frame; the accepted-risk row in TODO.md applies. The standard, the program package and the decisions index record the confirmation; [REVIEW REQUIRED] remains only on the per-AddOn human-verification widgets, which the owner creates.

  • AddOn panes: independently built tools in sandboxed panes on their own lab names, starting with the Migration Hub (ADR 0035; the Phase 6 website pull request of the App-to-AddOn program). The site gains a catalogue of tool add-ons (frontend/src/data/addons/catalogue.js: the migration row available, the network and cloud assessments coming since 2026-10-10), a generic pane page at /tools/<id> (pages/tools/AddOnPanePage.jsx: the status gate, the sandbox and allow lists derived from the row's capabilities, the hcw-addon message filter with its four states, an allow-listed navigate, a 30-second watchdog, and one sentence for every way the tool can be missing), routes, pre-render entries and the Tools menu derived from it, and one exact origin per available row in the CSP's frame-src. /tools/migration goes from a placeholder to the pane. The Function App gains GET /api/public/addons/{id}/status (lib/addons/): ids closed in a server-side registry, a 5-second https-only read of the add-on's /api/health, a one-minute cache per id with failures cached, and a six-field projection that never carries the URL; the address is the plain app setting ADDON_MIGRATION_URL in Terraform. Admin gains the Tool add-ons group with the Migration add-on card, probe and directory row, and the Credentials register lists the add-on's vault key. The lab host gains the generic addons Ansible role (one hardened container per group_vars row by digest on the loopback, a Caddy route per name with the site's sentence at 503, a per-name direct-visit redirect whose directive ordering caddy adapt checks in CI, the end-of-run privilege checks extended to each container) and the migration row, with the digest and the widget's site key marked [VERIFY] for the owner. The AddOn integration standard lands in docs/standards/, the program package in docs/architecture/app-to-addon/, and ADR 0035 records the model with two items for the owner's review. Review round 1 (Copilot): the add-on containers run on their own bridge network with inter-container communication off and egress limited in DOCKER-USER to the verification endpoint's published ranges on 443 (also written to ufw's after.rules for the next boot); no volume or mount, compared strictly; a row with no digest answers 503 directly and any container left from an earlier digest is removed; the privilege checks assert the network, the one loopback binding and no mounts, null-safe; the status proxy validates the whole version string, shares one in-flight read per id, and logs by setting name, class and code only (the minute cache's warnings too); the pane focuses its frame on load; the migration row ships without downloads until ADR 0035 decision 9 is confirmed.

  • Credentials: one register of every credential, with its age and expiry, on Admin → Integrations → Credentials, and a Telegram reminder for each one only the owner can renew (#1026). functions/src/lib/credentials/ holds a static register of every credential by name and metadata only — where each lives, what uses it, its issuer, whether it renews itself (self, automation, hand, or none for an identifier) and, where a rule exists, its lifetime: Key Vault (built from secret-catalog.js so the two cannot drift, which register.test.js holds), the three MCP OAuth connections in mcp_servers, the GitHub Actions and Agents secrets, the OIDC federations, the HCP Terraform variables of hcw/hcw-azure and hcw/hcw-lab, the Static Web Apps deployment token, the Entra and Docker identities, and the lab host's vault keys, files and the owner's password-manager entries. GET /api/cms/credentials (super_admin) merges in the live dates — the Keys tab's lastWriteAt, the lab host's Coder token report, each MCP connection's state, read as a Cosmos projection so no token field is ever fetched — and answers each credential's age, expiry (reported, or the last rotation plus its rule for a hand-renewed one), and a state: ok, due soon, overdue (red), or unknown. A source that cannot be read, or is present but malformed, is never read as empty: the credentials that depend on it show unknown, with no expiry or reminder. PUT /api/cms/credentials records, under the document's ETag, when a hand-renewed credential the site cannot read was last rotated (admin_config/credential_register, a date per credential). Every hand credential with a rotation rule gets the reminder row credential-<id> on the existing Reminders sheet, said by the existing sendReminders timer: the Anthropic key, the Telegram bot token, both GitHub App private keys and the Coder GitHub OAuth secret (every 180 days, the owner's rule of 2026-10-09), the Static Web Apps token's 90-day reset and the two 730-day lab host certificates. With a known date it is "Rotate …" at the due date; without one it is "Record when … was last rotated", due the day it was added, until the date is recorded. The sync owns only those rows: the owner's own reminders pass through untouched, the timer's stamps are kept while a row's due date and title stand and reset when either changes, and every attempt re-reads the sheet and then the dates and writes under the sheet's ETag, so a rotation recorded mid-sync is never overwritten. Recording the rotation is the only way a register row's cycle ends: the sync reopens one found marked done, the Platform Settings save (editor) carries every credential- row through exactly as stored and cannot add, change or remove one, and the Reminders sheet shows them read-only, "Managed by Integrations → Credentials". It runs when a rotation is recorded, from the tab's Update reminders (POST /api/cms/credentials/reminders), and at the start of every sendReminders run. No value is read, stored or returned anywhere. sendReminders also binds each Telegram stamp to the cycle it was said for, the row's due date and title: after an ETag conflict it re-applies a stamp only to a row still in that cycle, so a reminder re-dated or renamed meanwhile is said afresh. This applies to every reminder, and nothing changes when there is no conflict. A Platform Settings save of the sheet is written under the ETag of the read it merged from (412 or 409 re-reads, three attempts, then 409), so a sync or a stamp that lands between the read and the write is never written over. A Keys-tab write syncs the credential reminders at once, so a rotation moves its reminder without waiting for the timer, and the register copies only lastWriteAt and rotatedOn out of the records it reads. An editor's save of the sheet may hold at most 180 of the owner's own reminders: 20 rows stay free for the register, so its warnings can always be installed.

  • Lab host: Coder automation. The host renews the site's Coder status token and publishes the hcw-lab template on its own, with one rotation credential the owner seeds once (owner approval 2026-10-08). hcw-coder-automation-seed takes the owner's short-lived token on stdin and creates the rotation credential: an unscoped, year-long token of hcw-status, which must be a Template Admin and must not be an Owner, so it cannot manage users or act as anyone else. It is a root-only file, /etc/hcw/coder/automation/rotation-token, not a vault secret, because nothing on the host can read HashiCorp Vault unattended (no auth method, sealed after a reboot under Shamir keys) and the Ansible vault's password is a root-only file beside it. The daily hcw-coder-automation.timer renews the site's status token when the one the site holds expires within 30 days or is over 60 days old: a 90-day token with exactly template:read, workspace:read, api_key:read and user:read (the last so the token can read its own expiry, which the site checks), checked against Coder, then handed to the site through the lab agent's CLI (vps-agent/bin/report-coder-automation.js, run with systemd-run as the agent's user with its EnvironmentFile), live only once the CLI answers stored: true. It deletes hcw-status's older status tokens, keeping the newest two, the one before the live one and any younger than 48 hours, renews the rotation credential under 60 days, and reports checkedAt, both expiries, the rotation and the template's publish, and a content-free lastError, every run; a failure fails the unit, which the hcw-unit-failed@ notifier turns into the lab alert. Every bootstrap.sh run publishes the template with the credential when a file hcw-coder-template-push copies, or the default autostop, changed since its last publish, before lab_images removes the old version's images; with no credential it says how to seed one and carries on. Coder v2.38's token API was read from its docs and source: lifetime is a Go duration in nanoseconds, no scope means coder:all, and a Template Admin's tokens may live a year where an Owner's stop at 168 hours. The owner's lines, what success looks like and how to rotate by hand are in docs/runbooks/labs-host.md, "Automatic renewal"; the design is in lab-host/ansible/roles/coder/README.md, "Coder automation". Tests: lab-host/ansible/roles/coder/tests/hcw-coder-automation.test.py, run in the ansible-lint (lab-host) job against a fake Coder API, a fake systemd-run and a fake site CLI, with no token in any output, state or report. The seed keeps a stored credential only while hcw-status is still a Template Admin and not an Owner, and every run refuses one that is not a Template Admin; a new credential that fails before it is stored, and that Coder will not delete, is recorded by id and deleted by every later run until it is gone (review of #1035). A replaced credential is listed the same way, before the new one is written, so an interruption cannot leave it usable and forgotten; hcw-coder-automation revoke turns the automation off by deleting the credential in Coder before the file, and comes before any revert of this code; a suspected leak is contained on the account, by deleting hcw-status, which deletes every API key it has, successors included (CodeRabbit, #1035).

  • Coder automation, the site's half: the lab host can hand the site a renewed Coder status token through the lab agent, and the site stores it only after Coder vouches for it as a working, read-only key of the status user; the Hybrid Lab card shows what the host last reported (owner approval, 2026-10-08). Until now CODER-STATUS-TOKEN was renewed by hand once a year, minted in a pane and pasted on the Keys tab, because the host cannot reach Key Vault and must not. A fourth agent route, POST /api/agent/reportCoderAutomation, guarded by requireAgent exactly as the heartbeat is, takes { agentId, statusToken?, report { checkedAt, statusTokenExpiresAt?, statusTokenRotatedAt?, rotationTokenExpiresAt?, templatePushedAt?, templateVersion?, lastError? } } and refuses anything else with a 400 and a sentence before any read (functions/src/lib/labs/coder-automation.js, composed in lib/lab-agent.js). A token must have Coder's key shape, measured against Coder v2.38's source (a 10-character id, a dash, a 22-character secret), and must pass three reads made with it through coder-status.js's guarded GET, each answering exactly 200 (verifyStatusToken): the running-workspaces read gives an integer count, GET /api/v2/users/me names hcw-status (CODER_STATUS_USER may rename it), and the token's own key record lists template:read, workspace:read and api_key:read, with user:read the one more it may carry. An unscoped key (coder:all, or all in the deprecated scope), a key with any other scope, and another user's key are refused, so a working Template Admin key cannot become the read-only status token. Only then is it written by writeCatalogSecret, the Keys tab's own writer factored out of admin-secrets.js, so the Keys row goes amber and then green as for a paste and there is no second vault client. The writer never throws past the vault write and says vaultWritten, so a failure after it is answered 200 { ok: true, stored: true } with the failure as lastError, and the host does not mint again. The secret_state document the Keys tab, the agent and every key verdict share is now written one secret's record at a time under its ETag (updateSecretRecord), so a verdict landing during a renewal is no longer lost. One coder_status_token_rotated audit row, never the value. A token that fails a check is 422 with a sentence naming the check, and a vault refusal 502; the report is recorded either way in one admin_config/coder_automation document, merged under its ETag (event dates kept when a report omits them, lastError cleared by a clean check, anything shaped like a Coder key in it removed). A checkedAt, statusTokenRotatedAt or templatePushedAt more than ten minutes ahead of the site's clock is a 400, and the card measures silence from the site's own reportedAt, so a host clock in the future cannot hide a host that has stopped. Warning and error lines name a status, never the token, the agent or a document id. GET /api/cms/labs/coder-automation (editor) serves the document, and the Hybrid Lab card on the Services tab now groups its keys: Coder's two values together with Automatic renewal (status token renewed on, rotation credential expires on, template last published, last check, and lastError amber or red by lib/status.js), Turnstile's apart. Before the first report it reads Automatic renewal not set up yet, and the manual paste on the Keys tab is unchanged. On the host side, vps-agent/bin/report-coder-automation.js reads one JSON object on stdin, adds agentId, posts it with the agent's API client, prints {"ok":true,"stored":<bool>}, and on failure prints the error's class alone; the agent's configuration moved to vps-agent/lib/config.js so the agent and the one-shot read it one way. lab-host/README.md, "The status token for the site", gains a paragraph naming the three checks, saying the host should mint with --scope user:read as well (from Coder v2.38's source: both identity reads load the caller's user through its authorizing store; not yet measured on the lab's Coder), and pointing to the host's setup. Tests: functions/src/lib/labs/coder-automation.test.js, coder-status.test.js, admin-secrets.test.js, key-verdict.test.js, lab-agent.test.js; frontend/src/components/admin/integrations/coderAutomationView.test.js, IntegrationsServices.test.jsx, serviceRegistry.test.js; vps-agent/lib/report-coder-automation.test.js, config.test.js, api.test.js.

  • CodeRabbit is configured for this repository, for its free open-source plan. It reviews nothing until the CodeRabbit GitHub App is installed on the repository, an owner step. .coderabbit.yaml at the root (owner request, 2026-10-08) enables no paid add-on, and every key and value was checked against CodeRabbit's schema.v2.json. It reviews drafts too, with the chill profile and no request-changes gate, and it skips lockfiles, build output, CHANGELOG.md and docs/history/. It reads the repository's own review checklist in .github/skills/code-review/ as its code guidelines, with short path instructions for the functions, infra, lab-host, public-page and docs rules. On the open-source plan a public repository with few stars is reviewed when asked: comment @coderabbitai review on the pull request. scripts/validate-repository-structure.ps1 allows the file at the root.

  • Publer MCP: PUBLER_API_KEY usable as an MCP key, bound to mcp.publer.com only, a server-side MCP tool allowlist that keeps Publer's write tools out of reach (#995), a disabled Publer MCP seed entry, and a Publer runbook with the 2026-10-07 readiness check (nothing published). functions/src/lib/ai/mcp-policy.js refused PUBLER_API_KEY as an MCP key name, so Publer's MCP server could not be registered with the key the Social Hub already holds. The name is now on the allowlist with mcp.publer.com as its one host (app.publer.com, the REST host, stays publerProxy's and is refused for an MCP server), and a URL on that host may carry no query string, because Publer's settings page hands out its server URL with the key as ?api_key=, which would otherwise sit in Cosmos, in every read of the server list and in the audit row. No header code: Publer's server accepts the resolver's plain Bearer, and a Sync through the repository's own path reached its 47 tools. The AI Engine seed gains publer-mcp, switched off, written the first time a super_admin opens the MCP Servers tab, with a setup page. docs/runbooks/publer.md records the transport, the auth shape, which tools read and which write, and the owner's read-only verification (get_publer_user). Tool allowlist (#995): an MCP server may carry allowedTools, and callMcpTool (so mcpProxy and the platform jobs) refuses any other tool with 403, naming the tool and the server, before any upstream request and whatever the caller's role. A server using PUBLER_API_KEY must carry a non-empty list: saving one without it is refused and a stored one without it refuses every call. The seed's list is Publer's 27 read and 7 session tools, never the 13 that create, schedule, publish, edit, delete or upload. Writing allowedTools, and switching a PUBLER_API_KEY server on or off, need super_admin; a PUT that omits the list keeps the stored one; Sync never touches it; the ai_config_updated audit row records it before and after. Firecrawl, Replicate, the VPS token and keyless servers carry no list and still accept any tool name, as before. The MCP card reads "34 of 47 tools allowed". Tests: functions/src/lib/ai/mcp.test.js, functions/src/lib/admin-integrations.test.js, and frontend/src/lib/aiEngine.test.js (the seed against the API policy).

  • Platform Settings → Reminders shows the Telegram bot's QR code, for pairing a phone with the bot again (owner, 2026-10-07). The owner's QR artwork for @agenticarchitectbot now sits at the top of the Reminders card, at a fixed 220 px wide on a laptop and 60 % of the screen width on a phone, on a white panel so the dark artwork reads in both themes, with the handle linked to https://t.me/agenticarchitectbot beneath it and an Open full size icon on its top-right corner. The artwork is AI-generated (its C2PA manifest names the image generator) and no decoder read its embossed modules, so it is not verified to scan; the handle link is the fallback. A second, plain black-on-white code that #994 placed beneath it was removed the same day at the owner's request, and on a laptop the steps beside the artwork now start level with its top and end, with the Test Telegram frame, on the handle's line. Three steps cover a phone that loses the bot (scan the code or search the handle, press Start or Unblock, press Test Telegram and watch for the message on that phone), with the existing Test Telegram button inside the same panel. The note under them says what Sent proves: Telegram accepted a message for the chat id in TELEGRAM_CHAT_ID, not that this phone holds that chat. When Sent shows and nothing arrives, check which account the phone is signed in to first (the same account keeps the same chat id on any phone), and change the stored id on Integrations → Keys only if the account itself changed. The artwork is imported from frontend/src/assets/admin/ as delivered (1,907,187 bytes) and fingerprinted into /assets; it loads only when that page is open. TelegramReconnectPanel.jsx, with tests beside it.

  • Governance remainders of the 2026-10-06 estate review: the Terraform run identity's Contributor scoped to its resource groups, code owners on the auth and lab paths, a 100-day restore-drill check, and Arc guest configuration off (findings SEC-1, SEC-3, PLAT-1, LAB-6; #971, #972, #961, #952). SEC-1: scripts/bootstrap-terraform-oidc.ps1 now grants id-plat-terraform-prod-cus-01 Contributor on each of the nine resource groups infra/ declares and a custom role, HCW Terraform Subscription Scope, for the only things infra/ does at subscription scope (create and update those groups, the two subscription budgets, provider registration), with no resource-group delete, since at subscription scope that verb deletes any group and its contents. Both lists live in the new scripts/terraform-identity-grants.json; the custom role is written there with its assignable scopes filled at run time, because the Management and Connectivity subscription ids are not published. Step one of two: subscription Contributor stays until a separate later change removes it, after a plan and apply have run on the narrow grants, and the file comments say why the two are apart. The RBAC Administrator condition (Set-TerraformRbacCondition.ps1) now also refuses Contributor, so the scoping cannot be undone by one role assignment, and its recovery path keys on the custom role as well as subscription Contributor. New scripts/terraform-identity-grants.test.mjs keeps the JSON's groups equal to the azurerm_resource_group blocks, its register actions equal to var.azure_resource_providers, and fails if infra/ ever assigns a role the condition refuses. ADR 0005 gains a dated amendment: the condition stays an operator script (held in Terraform it would be maintained by the identity it constrains), the two-step scoping, and the read-scoped team token recorded as an owner decision (HCP Terraform Teams is not entitled); TFC_AZURE_PLAN_CLIENT_ID is noted as the separate, Teams-free route to plan-only Azure credentials, not built. SEC-3: .github/CODEOWNERS adds /infra-lab/, the identity scripts, functions/src/lib/auth/, admin-identity.js and its HTTP route, useAdminAuth.js, msalConfig.js, /lab-host/ and /vps-agent/; the ruleset rule and 2FA stay the owner's, and the file says that requiring code-owner review blocks every pull request until a second reviewer exists. PLAT-1: new scripts/check-drill-age.mjs reads the Drills table in docs/runbooks/cosmos-restore.md (row format now defined there: date, RTO measured, RPO measured, who, applied runs, documents, notes), exits 1 when the newest dated row is more than 100 days old or there is none and 2 when a Date cell is not a real past date or a dated row leaves any cell but Notes empty (a date alone is not a drill), and new .github/workflows/check-drill-age.yml runs it every Monday and opens or comments on one issue. The table is empty, so its first run fails on purpose. LAB-6: the arc role sets guestconfiguration.enabled false with the same read-compare-set as the allowlist and incoming connections (owner decision 2026-10-07: Microsoft's locked-down-machine guidance, and no machine-configuration charge); ADR 0032 records it, and that the audit-only Linux baseline assignment then has no agent to evaluate it, which is left to the owner on #952. Owner steps: re-run the bootstrap script, then a plan-and-apply in hcw-azure showing only the permanent diff; the three GitHub settings pages; bootstrap.sh on the lab host and azcmagent config list; the drill.

  • The admin journey, signed in, in a browser on every pull request, with no tenant and no secret (estate review 2026-10-06, findings QA-1, AP-F1, AP-F2). e2e/fixtures/entra.js is a credential-free identity: the production bundle, built with placeholder settings (GUIDs no directory issued, an api:// scope that does not exist, the same-origin /api base; one list in e2e/fixtures/stub-build-env.js, set only by the e2e job's Build step and the local Playwright web server), runs MSAL's real authorization-code redirect, and page.route() answers login.microsoftonline.com (metadata, /authorize echoing MSAL's state, /token carrying its nonce) and /api. Nothing in the application knows it is under test, and no test switch was added to it. It fails closed: a request to any third host is aborted and fails the test (the icon font is answered locally), a tenant or client other than the placeholder is refused, and the local web server is never reused, so a bundle built with real settings, from a local .env say, fails the run instead of sending the stub's bearer to a real API. e2e/admin-authenticated.spec.js asserts that the callback lands on the route the person asked for, that a registry refusal shows "Access Denied" and nothing of the admin, that an insufficient_scope 401 shows the configuration card with the API's description and does not retry, that an invalid_token 401 leads to exactly one re-authentication (no account picker) and then the honest card, still once after a reload, and on a Pixel 5 at /admin, /admin/queue, /admin/platform and /admin/health that the content column is at least 340 px with the drawer closed, the drawer opens from the visible menu button and closes on Escape, and nothing scrolls sideways. Each test names the product change that makes it fail; eleven such changes were made, rebuilt and run, and each failed its test. The twelfth, dropping the callback page's pathname check before its /admin fallback, did not: MSAL 5 returns to the start page with a full navigation, so that line never runs in this flow, and the spec says so rather than claiming it. The e2e job runs it on desktop Chrome and the Pixel 5, still not a required check; the three specs together took 100 s on one worker locally. The signed-out spec's /admin/platform-settings was not a route and is now /admin/platform. Owner step: promoting the job to a required check stays the two-weeks green decision, about 2026-10-20.

  • The collapsed admin rail shows each item's label on hover and on focus (AP-F2). Collapsed, the rail was icons with the label in a title attribute, which touch never shows and a keyboard never triggers. Each item, Back to Site and Sign Out now show a visible label beside the icon on hover and on focus, carrying the text the link's name starts with; Escape dismisses it. It is fixed-positioned, because the nav is a scroll container and would clip anything placed against the 64 px rail; the browser test checks it is painted outside the rail.

  • Lab pins move themselves by pull request, each with its publisher's checksum; the lab images rebuild weekly; the host reports held packages with an upgrade waiting; runtime advisories have stated response times (estate review 2026-10-06, finding LAB-3, the remainder; #949). The weekly check from #899 reported and bumped nothing. Now scripts/lab-pins-upstream.mjs --bump <set> makes the edits, and only with the checksum or digest read from the publisher beside each version (new scripts/lib/lab-pin-bumps.mjs): a Docker or NodeSource package's SHA256 from the Packages index whose own hash InRelease lists, an image's index digest from the registry (bytes hashed, Docker-Content-Digest agreed) and the Docker Hub API, HashiCorp's SHA256SUMS, node_exporter's sha256sums.txt, and the Go checksum database for the Caddy module. A pin the check calls current is re-read too when its publisher can rebuild a release under the same name (an image tag re-pushed on a patched base, such as postgres:18.6 that day, or a new Debian revision), and its digest or revision moves. A pin with no verifiable checksum is not edited; it is listed in the weekly issue. --dry-run prints the planned edits. lab-supply-chain.yml gains a propose job (read-only) and a pull-request job that keeps one pull request per pin set open on a fixed branch (chore/lab-pins-host, chore/lab-pins-image-base), replacing its commit and body while it is open and never overwriting a commit someone else added (scripts/open-lab-pin-pr.mjs), with the GitHub App token the other scheduled pull requests use, so the required checks run and no workflow holds contents: write. The lab image base moves within its release line (python:3.14-slim-trixie today, so 3.14.7 to 3.14.8) with BASE_PYTHON_VERSION and every FROM. publish-lab-image.yml gains a Tuesday 05:20 UTC rebuild that publishes only when the full image's Debian packages moved (tagged <sha>-rebuilt-<date>), and after every publish a chore/lab-pins-image-digests pull request pins the new digests in vps-agent/lib/capabilities.js and the Coder template, each read again from GHCR first; capabilities.js leaves the push filter so that merge does not republish. Its header says what a rebuild changes on its own (the Debian packages of full) and what it cannot (the base, its Python, every tool in versions.env), and the Building images guide on the site says the same. The hardening role installs hcw-held-upgradable.timer, daily at 07:15: one daemon.warning line tagged hcw-held-upgradable per held package apt could upgrade, which the Arc data collection rule ships, and a notice line when none, with its own test in the ansible-lint (lab-host) job. docs/runbooks/labs-host.md "Runtime advisories" states the response times for a runc, containerd or Docker Engine fix (48 hours for a critical or an exploited escape, 7 days for high, the next weekly bump otherwise), what runs on its own and the owner's commands. The checker also watches buildx and compose. An alert on the new syslog line is PLAT-4's work.

  • Admin Labs → Coder tab: Coder's own dashboard, framed (owner, 2026-10-07). Coder's pages open only inside the site (the lab host's panes-only rule), and since #925 the lab panes never show one, so the operator had no way to Coder's Tokens page, where the status-token renewal starts. The tab frames the dashboard, signed in as the operator, with Workspaces, Templates, Your tokens, Users and the audit log one click away, a Reload, and the lab pane's GitHub sign-in link for a signed-out frame. Same sandbox and permissions as the lab pane; nothing here calls Coder's API, the frame does, as Coder's own front end.

  • The lab launcher starts a stopped workspace itself (owner, 2026-10-07; #911). "The lab was enabled but I still needed to start it." A workspace whose last build stopped or was cancelled is now started with the call Coder's own Start button makes, with the CSRF token read from Coder's own page the way Coder's front end does, and the visit waits for the build as for any other start; an outdated workspace is started on its template's active version, so a template change reaches every lab on its next open. A failed build is not started blind, since its page carries the reason and Retry. When Coder refuses, or its page carries no token, the launcher frames the workspace page where Start is, as before: a Coder change here costs a click, never the lab. One attempt per visit.

  • Integrations directory: a configuration-driven tab on the admin Integrations page (owner brief 2026-10-06, slice 3; #919). One card per external service — its Powered-by badge, category, a one-line summary, what it powers here, where it is used, connection status from the same key lights and test verdicts the Overview reads, Docs and Official site links, and Set up, which opens the service on the Services tab. Search, category and connection filters, alphabetical. The rows are frontend/src/config/integrationsDirectory.js over the service registry and the brand registry; tests hold the three to each other, so a service cannot be missing from the directory. Admin-only, as every vendor-naming surface is.

  • One "Powered by" badge for every third-party service in the admin (owner brief 2026-10-06, slice 2; #918). IntegrationBadge renders "Powered by " in the platform's colour with a medallion, from one registry of the platforms the site actually uses, keyed by the Integrations page's service ids; an unknown id renders nothing, so the badge can sit beside any service. Placed on every service card on Integrations (the two composite cards, cloud pricing and the Hybrid Lab, have none by decision), every AI provider card, the podcast voice (ElevenLabs), social autoposting (Publer), newsletter settings (Resend) and the Reminders card (Telegram, which also says "Delivered by Telegram" in the form). Logos are a licensing decision per platform and are not shipped; the medallion is the initial in the brand colour until one is recorded. The label is theme foreground, since brand colours fail WCAG AA at badge sizes. Admin surfaces only: the 2026-09-28 rule keeps vendors off public pages.

  • Reminders: edit in place, and a Test Telegram button (owner brief 2026-10-06, slice 1; #917). Every reminder in the pane has an Edit action that opens it into the same form the New reminder uses — title, date, days before, link, notes, and done as the status — saving through the same PUT and updating the pane in place; one delivery method exists, so the form says "Delivered by Telegram" rather than offering a choice of one. At the top of the card, Test Telegram sends one message through the production notifier (new editor route POST cms/platform-settings/reminders/test) and shows sent, cooldown, or Telegram's own refusal with its status — 403 is a blocked bot, which is what silenced every notification the day this was asked for.

  • A refused Telegram delivery now pages the owner by SMS and mail (2026-10-06). The notifier is best-effort by design, so when the owner's account had the bot blocked, Telegram answered 403 to every send for the rest of the day and the only record was a trace line nobody reads; the lab-agent-offline message was among the losses. alert-telegram-delivery is a stateful log alert on the Application Insights component: one [notify] Telegram API error or notifyTelegram failed line in an hour fires it through the action group, which is not Telegram. Owner step: confirm the hcw-azure run (1 to add beside the permanent diff; declared for the plan check).

  • Reminders on Telegram: a sheet under Platform Settings and a daily check (owner, 2026-10-06). Dated things not to forget — a token that expires, a re-verification, a renewal — had been living in session notes. Platform Settings → Reminders now holds them as one admin_config document (reminders, the seventh platform setting, with Change history like the rest), and the new sendReminders timer (13:00 UTC, flag SEND_REMINDERS) says each one on Telegram once within its days-before window, once on the day, and weekly while overdue, until it is marked done. A stage is stamped only when the message was sent, so an unconfigured bot means a retry tomorrow, not a reminder counted as said. Same evening, on the owner's first look: the tab is a form on top that adds one reminder and clears when the save took, and a pane below listing every reminder with its details, a Done box and a red circle X that cancels it after one inline confirmation; nothing is edited in place. Owner steps: add SEND_REMINDERS to enabled_timers on the hcw-azure workspace and confirm the run. The plan reads 3 to add, 1 to change, 3 to destroy: the three are the permanent azapi replacements every plan of this root carries, and the one change is the function app's FEATURE_FLAG_SEND_REMINDERS, declared for assert-expected-plan.mjs, which is the check to read rather than the counts. Then add the first rows at https://hybridcloudworks.com/admin/platform?tab=reminders.

  • The owner hears on Telegram when a lab agent stops cleanly (LAB-2 follow-up, 2026-10-06). systemctl stop sends SIGTERM and the agent heartbeats offline itself, so the health timer never had a stale agent to mark and its message never fired; the owner's first test produced no message for exactly this reason. The heartbeat now sends the same message on that transition, with the same source, so a stop and the timer's own mark are one message within the cooldown.

  • Forge Studio Queue: one document per entry, so a favourites export fits, and a remove on every row (owner, 2026-10-06). The first cut kept the whole queue in one admin_config document capped at 200 entries, and the owner's first real import was a browser favourites export of several hundred links. Entries are now documents of their own (docType forge_queue_item, the same partition), the caps are 2,000 URLs per add and 5,000 in the queue, and every edit, Save and job outcome is an independent ETag-guarded write — no byte budget, no bulk-edit race with a finishing job. An entry's id is its URL's hash, so the URL is the identity and two adds (or an add and the migration) meeting on one URL is a 409 read as skipped; a Save claims the entry before writing its job, so two Saves on one entry start one job. Mutations answer only the entries they changed and the page merges them. The old document is migrated into entries and emptied on the first read after deploy. On the Queue tab every row has its own ×, the header gains Remove selected (n), and the detected list under From a URL shows its first 200 rows and adds every one.

  • Forge Studio: From a URL takes many URLs and .html files, and a Forge Studio Queue completes them singly or in bulk (owner request 2026-10-06). The Source URL box accepts one URL or many (one per line, or a block of text with links in it) and an Import .html button reads one or more saved pages for the URLs they link to; every URL found becomes a row with its own Remove, "Add all to the queue" (green) sends the rows to the queue and "Remove all" (red) clears them first. A single URL continues to the Brief as before. The queue is one admin_config/forge_queue document behind GET|POST cms/forge/queue, POST cms/forge/queue/update and POST cms/forge/queue/forge (editor; ETag writes, retried, because the job's onComplete writes it too; a byte budget half the Cosmos item limit; the list reconciles forging entries against their job documents, so a lost onComplete never leaves one forging; removing keeps a forging entry). On the new Queue tab the owner selects one entry to complete its "From a URL" fields, or several to set the shared fields once (a field shows a value when every selected entry agrees; blank leaves each as it is); Save applies the values and starts one forge-from-url job per entry with the brief on the payload, so the brief lands on the document the job creates whether or not a browser waits, and the entry shows forged (with an Open link) or failed when the job reports back. "Save for later" applies the fields and stops. The Start grid gains a Queue card with the count.

  • Blog run: the October 2026 certification calendar for Azure, AWS and Google Cloud (owner request 2026-10-05). Three drafts in docs/content for Admin → Drafts "Import from docs/content", each built only from vendor sentences read during the 2026-10-05 catalogue re-read: Azure (AI-500 out of beta, PL-400 → AB-400 on October 16, MS-102's last eight weeks, the DP-420 rename, retired exams and their successors), AWS (the November version bumps and AWS's 2026-10-02 correction of the SAP/DVA last days, the two open betas, ANS-C01, the language retirements) and Google Cloud (Agentic Architect GA registration on November 2 and why that is not an exam day, the PMLE move away from Vertex AI). drafts-handlers.test.js now expects fourteen importable files.

  • Learn catalogues: the 2026-10-05 re-read of Azure, AWS and GCP against their vendor pages (owner request 2026-10-05). Every row of the three catalogues was read on the day and DATA_AS_OF moves to 2026-10-05 on all three. Azure: AI-500 is out of beta (active); PL-400 is expiring with expiryDate 2026-10-30 and hands over to the new AB-400 row (upcoming, availableDate 2026-10-16, Microsoft's study-guide outline included); MS-102 and the retired AZ-204, AZ-500, PL-200 and MB-280 name the successors Microsoft recommends; the DP-420 rename (2026-10-06) and PL-400 → AB-400 (2026-10-16) join the timeline; the MB-330 retirement Microsoft expects in December is noted without a date, as it has none; twenty-two learnUrls that 404'd or pointed at the browse root carry the live credential page. The study-guide outlines were regenerated (AB-400 in, AZ-800/AZ-801 out as retired, MS-721 and SC-300 changed on Learn). GCP: PAA stays upcoming without a date and says that GA registration opens November 2, 2026 with the exam expected mid-November; PMLE no longer names Vertex AI, following Google's note on the exam's transition to Gemini Enterprise Agent Platform. AWS: nothing stored moved; AWS corrected its SAP-C02 and DVA-C02 exam pages on 2026-10-02 to the dates the file kept. education-catalogues.test.js pins the re-read.

  • Ambassador hub: a program can be additional to another, and MCT Regional Lead now is (owner request 2026-10-05). A program may carry parentProgramId; the seeded MCT Regional Lead names Microsoft Certified Trainer, and a stored Regional Lead from before takes it on the next read (seedBackfillFor). The child is shown, and its application may start, only while the parent's membership is Active: the Programs tab lists it as its own card right after the parent once that is so (and until then the parent's card says what opens and when), the Settings row disables its Application state with the reason, POST cms/ambassador/applications answers 409 PARENT_NOT_ACTIVE otherwise, and GET readiness/{id} reports the gate and counts evidence filed under the parent for the child too. One level only: a parent must exist, be another program and have no parent of its own (400). The Regional Lead seed gains the role's requirements and expectations as application questions (nomination, consecutive years, MCT-eligible certifications, Lounge and Regional Hub activity, the term's commitments), from Microsoft's published role description; the October form's own questions are not published and the seed says to edit against them when the invitation arrives.

  • The Coder status token's expiry on the Integrations card, so no calendar is needed (#763). CODER_STATUS_TOKEN is made with a one-year lifetime and nothing renews it. readTokenExpiry in lib/labs/coder-status.js reads the token's own record from Coder (GET /api/v2/users/me/keys/{id}; the id is the token's part before the dash) and GET cms/labs/coder-token (editor) answers the expiry, the days left and whether that is inside the 30-day warning window. The Hybrid Lab card's test ends Status token expires on <day> (<n> days), goes red inside the window or when Coder refuses the token, and — for a token made before the api_key:read scope, which is what lets a token read its own record — says expiry unknown: add the api_key:read scope at the next renewal. The README's create line carries the scope. Tests: the key id off the token, the record read and both outcomes of the window, the five reasons, the guard, and the card's sentences.

  • Ambassador hub: membership status and a direct application-state select on Settings, documents archived on an application (owner request 2026-10-05). Each program carries a membershipStatus — Not a member / Working / Active / Denied — separate from the application funnel, set on the Settings row or in the editor and shown as a badge on the Programs card. The Settings row also sets the program's latest application to any state directly: a super_admin statusOverride the API accepts outside the transition table and records in the history row (an editor is refused), starting an application at that state when none exists. The application's upload takes documents as well as images (PDF, Word, Excel, PowerPoint, CSV, text): the upload route accepts those types into private containers only, never a publicly served one. That exposed a standing gap (review): a private upload answered no URL and the application cleaner dropped the entry, so every file ever uploaded to an application was orphaned behind a success toast — a file now keeps a { container, path } reference and a new editor-guarded GET cms/ambassador/files/{container}/ {*blobPath} serves it as a download (private containers, ambassador/ prefix only, nosniff, no-store). The application patch writes with the row's ETag and answers 409 on a conflict; statusOverride is refused on a create; the Programs card always shows the membership badge; the Settings selects wait for the applications read. "ageing" on the Dashboard reads "aging". Tests: the override (403 for an editor, history override for a super_admin, the plain table still enforced, refused on create), the 409, the file reference and the download route's four refusals, the private-only document types, the membership vocabulary through the form, and latestApplicationFor.
  • The Ambassador hub seeds the programs from the owner's documents, walks the official application questions, and groups and guides the evidence (owner request 2026-10-05). seedPrograms ran only on an empty container, so a program added to DEFAULT_PROGRAMS never reached the live one; ensureSeededPrograms now runs on every programs read and inserts any seed whose id is absent after the highest stored order, never overwriting a stored program and never bringing back one the owner disabled or soft-deleted; a stored seed that has no applicationQuestions or no scoring of its own takes the seed's in the same read (one patch on the row's ETag, nothing else on the document touched), so the live MVP gets its question list without losing an edit. Four programs join the seeds from the program rules as published in 2026 — Microsoft Elevate Educator – Expert (MIEE, with the official 2026-2027 question list and its May to 31 July 2026 window), GitKraken Ambassador, the Microsoft Management Community (scored in credits: a scoring field with the badge tiers, metrics.credits on evidence, and readiness that sums credits against each threshold instead of counting items) and MCT Regional Lead — and three are updated: AWS Ambassador carries the Charter and Established levels with their contribution counts and the thought-leadership rules, Microsoft MVP carries applicationQuestions mirroring the official form's five sections (profile, network links, the application questions, award category and technology areas, the eight expertise gates with up to 24 tagged activities), and Microsoft Certified Trainer names the annual renewal, the MCT Lounge and the Metrics That Matter classes export, which the Evidence tab now imports as "MCT classes (Metrics That Matter CSV)" through a CSV reader on the existing import route, one manual row per class id (a row without one is skipped and counted by reason; a text over 1,000,000 characters is a 413 naming the limit, and the dialog refuses it first). An imported row is stored under an id derived from its source pair with the atomic create, so two imports racing for one source document leave one row; a missing seed is inserted the same way, and a 409 from an overlapping read keeps what is stored. A program with questions turns the workspace's Responses into a guided form, section by section, each question rendered by its kind (text with the live count, choice, yes/no, URL, a frequency grid, network + URL rows, activities picked from the attached evidence) with Copy on every answer and Copy all for a plain-text packet in the form's order; answers stay in responses[] as { questionId, text } with structured kinds JSON-encoded, and the print packet lists the sections. The Evidence tab gains Group by (Source, Program, Year, Verification) with collapsible groups remembered per browser, and a guide above the list, "What each program still needs", with have / need per requirement from the readiness arithmetic, a hint naming the evidence types that would count, and Add opening the editor on that source and program. No answer of the owner's is seeded; the questions are the programs' own. ambassador.test.js pins the seeding on an existing container, the credits arithmetic and the CSV import; validate.test.js the question and scoring rules; applicationQuestions.test.js, GuidedResponses.test.jsx and evidenceView.test.js the guided form, the grouping, the stored collapsed state and the guide rows.
  • Certification badges are normalised on upload: PNG or SVG in, a 512 × 512 PNG out (owner request 2026-10-05). The Settings tab promised "PNG / JPG / SVG" while the upload route refused SVG into certifications (a publicly served container, where an SVG is a scriptable document), and a PNG was stored at whatever size the issuer shipped. lib/badge-image.js decodes every badge with sharp and re-encodes a transparent 512-pixel square (contained, EXIF rotation applied, metadata dropped); an SVG is rasterised at 300 dpi and the PNG is what is stored, which is why the route now accepts the type there. The stored path takes the .png extension and the response's path/url name it; bytes that do not decode are a 415 "Not a decodable image" and nothing is written. Other containers are unchanged (SVG still refused into blogs/covers, content stored as sent). admin-uploads.test.js pins the SVG, JPEG, undecodable and as-sent cases with real decoding; sharp joins the functions dependencies (prebuilt linux-x64 in the lockfile for the deploy runner).
  • Legacy badge references are re-pointed at publish, so a fifth of the certification registry gets its badge back (#868). 22 of the 109 published certifications still named the decommissioned Firebase bucket, which the About page skips rather than renders broken; the migration had copied the blobs into the certifications container under the same paths (20 of the 21 distinct ones answer from public/media). New lib/legacy-badge-url.js turns either Firebase URL shape for a certifications/… object into the media delivery URL; publishSnapshot (the button, and the weekly reVerifyCertifications timer) asks blob storage whether each such blob exists, rewrites imageUrl, credentialImage and image[0].downloadURL in the published item, and persists the same rewrite to the document so the editor's card agrees. A missing blob is reported by name in the response's legacyBadges and the document is left as stored; the Publish toast shows the counts. The persist is one conditional write per row on the _etag it was queried with; a 412 re-reads the row, decides again from what is stored now and writes once more, and a second 412 publishes the fresh row as stored. A row whose imageUrl is a live selection is never rewritten (its old upload metadata would otherwise come back in front of the new badge), and the sanitizer now publishes the editor's imageUrl ahead of a legacy credentialImage, which a re-uploaded badge over a migrated row needed. The nightly cleanup's URL parser recognises the media route, so a re-pointed badge is a referenced blob, never a deletion candidate. legacy-badge-url.test.js pins the URL shapes, the alias rewrite, the edited-row rule and repointAll; snapshots-publish.test.js pins the exists / missing / no-storage / ETag / 412 / edited-row / cleanup cases; the About page's test pins that a dead legacy upload falls past to the published URL.
  • AI Engine: audio and image tasks join the Tasks table (ADR 0034 slice 5, #860). The last slice of the ADR: the non-text paths are tasks of the same registry and selection document as the text ones, and their settings pages lost the model fields they carried. Registry: listenAndLearnSpeech and podcastVoice (tts), coverArt and manualImages (image), each recommending the model that was its default the day before (Gemini's Economy voice, ElevenLabs's eleven_v3, Replicate's google/imagen-4-fast) with the reason and asOf: 2026-10-05, and recordingTranscript (stt), pageOcr (ocr) and embeddings (embedding) registered as planned — no provider carries them yet, so the Tasks tab shows "no eligible model" rather than omitting them, and they have no feature switch and no call site. MODALITIES and CAPABILITIES gain the five media kinds. Media providers in the catalogue: ElevenLabs (GET /v1/models, ids from model_id) and Replicate (a static list of the image models the repository calls) are catalogue providers keyed by ELEVENLABS_API_KEY and REPLICATE_API_KEY — no card, a key is what switches them on, and the drawer says so; *-tts and eleven_* ids carry tts, Replicate's and imagen-* carry image; a price in another unit says so (unit: ElevenLabs per 1M characters, Replicate per image at the figure CONTENTFORGE_IMAGE_COST_USD is set to). The selectable-models rule takes the task's needs on both sides (selectableModelsFor). Resolver and selection: a media task defaults to mode Recommended, because the Priority list is a chat list a media provider never joins; a custom chain may name only providers that can carry the task's needs (a 400 on save, dropped on read); each media task carries only, a product rule in code no document lifts — the podcast voice is ElevenLabs-only, Listen & Learn speech Gemini-only, the images Replicate-only (ADR 0029 §2b) — and its recommendation stands while the catalogue has not yet confirmed it, so the week between a deploy and the next refresh is not a week of silence. Migration: a stored Listen & Learn model (Platform settings) or a LISTEN_AND_LEARN_ELEVENLABS_MODEL setting that differs from the recommendation becomes that task's custom chain (thenGlobal: false), on a v1 document and on a v2 document saved before the task existed, so nothing changes for the owner on merge. Call sites: the Listen & Learn jobs, the podcast pipeline and its live check, and the Replicate client read their model through one router door, modelForTask({ task }) — the feature switch, the key check, the pin and the locks all apply — and the per-run ttsModel, the per-book voice.model and the stored default are ignored on write and dropped on read; Azure AI Speech stays the Listen & Learn adapter's own fallback. Settings pages: the Platform settings Listen & Learn voice card, the Generate tab's voice model field, the book dialog's Gemini model select and the Settings tab's model list are gone; each says "The model is chosen under AI Engine → Tasks" with the link, and cms/platform-settings/listen-and-learn-speech left the allowlist. Usage by task: the audio and image rows carry ai:<task> as their source, their old slug (listen-and-learn:audio, podcast:audio, images:cover, images:manual) as product, so Usage & Cost groups by task ("Breakdown by Task"). Tasks tab: the new rows render with their modality badge, the chain editor and the exclusions offer a media task the providers that carry its needs under its product rule (ElevenLabs and Replicate among them, named without a card), and a media task's Test is a dry run — the candidate it would use and the resolver's reason, no audio or image made, said on the line. Both suites, ai-call-sites.test.js reading modelForTask({ task }) beside feature:, and a new lib/ai/media-tasks.test.js hold it; the ADR records the slice.
  • AI Engine: the Priority list and the Tasks tab (ADR 0034 slice 4, #859). The three overlapping controls the ADR's context describes are gone from the page and one list and one table stand in their place, both reading and writing the version 2 selection document (slice 3). AI Services gains the Priority list above the cards: the providers a call may reach, in global.priority order, P1 badged as the default, each row with a model dropdown fed by the catalogue ("Provider default" or a selectable model) and a one-line "will use" for text and vision, moved with keyboard-reachable Up and Down; providers that are switched off, hold no key or are not named yet are listed below with the reason. Tasks (the Routing tab renamed; its ?tab=routing address and tasks both open it) is one row per task from the registry: label, description, modality, a public badge, the mode as radios — Recommended with the registry's reason and asOf, Global (P1), Custom — the effective model ("gpt-5-mini via Foundry", or the provider default with the model it was judged on) with the first turned-away candidate's sentence under it, badges for a recommendation that did not lead, a custom chain that fell through, an unpriced model or nothing eligible, an exclude control per provider, and for Custom a chain editor of up to four provider → model rows whose model dropdown is filtered by the task's needs through the catalogue's capabilities, with a "then the Priority list" switch. A row is a draft until Save; every save PUTs the whole document with the updatedAt it read, a 409 reloads and says "changed elsewhere" rather than overwriting, and a document the API refuses ("this task would have no model") is said on the row with the API's sentence. The effective model is the router's own answer, never a copy of the resolver on the page: GET cms/ai-routing/effective runs selectChain for every task over the configuration the router reads (router.js resolveEffectiveSelection: the same loader as chainDetails, cache dropped first) and answers per task { mode, chain, rejected, flags } beside the registry entry and the document's, per Priority row what a null model resolves to (defaults per purpose from DEFAULT_MODEL_TABLE with environment overrides, the modality recommendation it is judged on), and availability; effective-selection.contract.test.js holds the route's answer for every task equal to selectChain's on the same documents under three document sets and two key sets. A per-task Test, POST cms/ai-routing/test/{task}, walks the task's effective chain with the card Test's prompt and caps (proxy.js testTaskCandidate: 16 tokens, 45 s, usage source ai-engine:task-test, nothing written onto the cards) until a candidate answers and reports who did, how fast, and why each one above it did not; the chain is the resolver's, so the public explain route is never tested against the trial tier (ai-tasks.test.js pins it with the real router over a document that puts the trial tier first). A Model catalogue drawer opens from both tabs: every provider's models with capabilities, price or "unpriced", status, first and last seen, Hide and Show, each provider's refresh age with "Refresh now", and a banner naming retired models the Priority list or a custom chain still names, read from the resolver's rejections. A null model keeps meaning the provider's default per purpose (ADR 0034, amendment of 2026-10-05 under "Consequences and accepted risks": §2's modality reading would have moved every drafting task to nano against the owner's 2026-10-04 decision; the Tasks tab's Recommended mode is how a task gets the registry's recommendation). Removed: the per-provider placement selects under "Where AI is used" (the switches stay) and the API's placement field (PUT cms/ai-features takes only { features }; a stored placement stays on the document as the migration's input until the first v2 save), the card's model pin (setProviderModel) and the order-of-preference card (setProviderOrder; the cards now follow the Priority list), the v1 Routing tab (RoutingTab.jsx, RouteEditor.jsx, routingModel.js, useRoutingTable.js, setAiRoute, setAiPlacement), and the v1 { routes } body on PUT cms/ai-routing with the routes and maxFallbacks fields of its GET, which nothing sends or reads any more. features-catalogue.js's placement functions and routing-table.js stay, marked deprecated: migrate-selection.js imports placementFor, PROVIDER_PLACEMENT_DEFAULTS and normalizeRouting to read a stored v1 document until its first v2 save. Pinned by TasksTab.test.jsx, PriorityList.test.jsx, selectionModel.test.js, AIEnginePage.shell.test.jsx (the five tabs with Tasks, the Priority list from the document, the drawer), AIEnginePage.test.jsx (the switches without a placement), tabs.test.js, aiEngine.test.js (the new usage source has a label), admin-integrations.test.js (the placement body refused, the v1 routes body refused, the v2 PUT and 409 unchanged), ai-tasks.test.js, proxy.test.js, the route inventory and the API contract (.azure/api-surface.json), with public-copy.test.js and education-a11y.test.js green beside them; the page names a provider only through the card's name or the registry, never in a sentence of its own. No app setting or Terraform run: the first save from either tab writes the v2 document, and the deploy carries the routes.
  • Docker article series, part 3: "Publishing to Docker Hub from GitHub Actions without a token" (#787). docs/content/blog-docker-03-docker-hub-oidc.md, a how-to taken from the live setup (#790, the publish-dockerhub job and the Docker Hub publishing runbook): why a stored Docker token is the liability, how the OIDC exchange works (GitHub's ID token for the audience https://identity.docker.com, the RFC 8693 exchange at Docker, a 300 s token), finding the subject the repository presents (gh api .../actions/oidc/customization/sub and the immutable-form string built from the owner and repository IDs, plus a step that prints a real token's claims), the connection form in Docker Home with every field value, the ID and switch as repository variables, the job itself (login with no password, copy by digest with --prefer-index=false and a read-back, one attestation per registry), verification from the outside (a pull by the GHCR digest, gh attestation verify --bundle-from-oci --source-ref), deleting the old secret, and the failure table (400 access_denied and the explain step, insufficient_scope, the expired token, the digest mismatch, the plan gate), with the one-off local copy for an organisation that cannot create a connection yet. Parts 1 and 2 now read of 3. The owner imports it from /admin/drafts and sends it to In Review; the article's names are the template's generic ones, and the verification runs against this site's real image so the output is real.
  • Replicate image spend is recorded and bounded (owner request 2026-10-05). Replicate bills on its own account, where no Azure budget sees it, and until now a generated image carried a per-image price for the editor's caption and nothing else. Every image the change feed or the Images pages generate now writes an ai_usage row (sources images:cover and images:manual) priced at CONTENTFORGE_IMAGE_COST_USD, which Terraform sets to 0.02 (the google/imagen-4-fast price on replicate.com that day), so Usage & Cost shows Replicate beside the token providers. Before every generation the client sums the month's rows and refuses with IMAGE_BUDGET_EXHAUSTED (429) at CONTENTFORGE_IMAGE_MONTHLY_BUDGET_USD (Terraform: 10) or at CONTENTFORGE_IMAGE_MONTHLY_MAX images (200), counting the image it is about to make, so an unpriced month still has a ceiling. The guard fails closed: a month that cannot be read refuses with 503, because no account-level stop is configured (the owner declined a Replicate-side spend limit on 2026-10-05; the application guard is the only stop). Concurrent generations can overshoot by their number times the per-image price, a few cents, which is accepted and documented. The manual image routes answer the guard's 429 or 503 with its code rather than a generic
  • ai-cover.test.js pins the row, the unpriced row, both ceilings, the pending-image rule and the fail-closed rule; manual-images.test.js pins the route's 429.
  • The AI task registry, with a recommended model and the reason per task (ADR 0034 slice 1, #856). functions/src/lib/ai/tasks.js names every piece of work the site sends to a model: label, description and route as before, plus the kind of answer it needs (modality), the capabilities a model must carry (needs), whether anonymous visitors can trigger it (public), and recommended: { provider, model, reason, asOf }. AI_FEATURES is derived from it, so the feature switches, the call-site test and the portal's toggles are unchanged. provider-recommendations.js carries each provider's best model per modality for the null entries of the global priority list to come. tasks.test.js refuses a recommendation that names an unpriced model, a public task that recommends the trial tier, or a grounding need outside Gemini. Nothing reads the new fields yet; slices 2–5 do.
  • The model catalogue, refreshed weekly from the providers' own lists (ADR 0034 slice 2, #857). Each AI Services card's model list was typed into the frontend seed, so a new deployment, a retired id or a price change was a pull request, and a card could offer a model its provider had stopped serving. functions/src/lib/ai/model-catalog.js now keeps admin_settings/ai-model-catalog (beside ai-features and ai-routing; ADR 0034 §1 names admin_config, which holds nothing the AI code reads): per provider refresh: { lastOk, lastAttempt, lastError } and per model status: live | retired | unknown, firstSeen, lastSeen, missed, hidden, capabilities and context from a code table keyed by id pattern, and pricing from COST_TABLE or unpriced: true. The weekly probeAiProviders timer lists every provider with a key after its Tests (OpenAI and NVIDIA /v1/models, Anthropic /v1/models paged, Gemini models.list filtered to generateContent, Foundry /openai/v1/models with the app identity's token); a catalogue failure is logged and reported and never costs the Tests already written. A 200 without the provider's collection is a malformed list, not an empty one. A failed list changes no model — the stored models are written back byte for byte — and records only lastAttempt and lastError (ADR 0034 review finding 3); retired takes two consecutive successful lists without the model; a refresh never touches hidden. Both writes are ETag-conditioned (replaceDocIfMatch, re-read and re-apply on 412), so a Hide and a refresh landing together both survive. Three editor routes: GET cms/ai-model-catalog (every provider, stale when lastOk is missing or older than 8 days, the router's defaults seeded in memory under whatever is stored until the first successful list), PATCH cms/ai-model-catalog/{provider}/{model} with { hidden } (the id URL-encoded, since NVIDIA's carry a slash) and POST cms/ai-model-catalog/refresh; every write drops the router's cache. A card's selectable list is live or unknown, not hidden, and carrying the text capability; the speech, transcription, embedding and image ids the lists also return stay in the disclosure only. The AI Services page reads the catalogue once and after every hide or refresh, merges each card's models from it in one place (frontend/src/lib/aiEngine/catalog.js), shows a "Models (N)" disclosure per card with retired, unpriced and hidden badges, a Hide / Show per model and the "List refreshed" line, and a "Refresh model lists" button in the section header. The seed's models arrays and providerModelPatches are gone; a pin the catalogue no longer lists reads "(not listed)" on the select instead of being released. model-catalog.test.js pins the adapters, the enrichment contract (every router default and every recommendation priced; every capability from the registry's vocabulary), the retire rule, the failed-refresh rule and the read rule; model-catalog-handlers.test.js the routes and the encoded id; ai-provider-probe.test.js the order and the isolation of the catalogue step; aiEngine.test.js holds the page's visibility rule equal to the API's and refuses a model list in the seed. No app setting or Terraform run: the first list lands on the next Monday probe, or on the button.
  • AI model selection: the version 2 document, the resolver and the migration (ADR 0034 slice 3, #858). Three mechanisms decided which provider served a call — the cards' order, the per-feature placement and the v1 routing table — and they overlapped. admin_settings/ai-routing is now version 2 (functions/src/lib/ai/selection.js): global.priority, the Priority 1, 2, 3 … list of { provider, model|null }, and per task { mode: recommended | global | custom, chain?, thenGlobal?, exclude? }. One pure resolver, select.js, decides the chain for every call and nothing else does — explicit call-site model, then the task's recommended model if it is live, priced, keyed, enabled and carries the task's needs, then a custom chain and the Priority list after it, then the Priority list — with every candidate filtered by key, switch, exclude, catalogue status (retired out; unknown or not yet listed allowed with a flag, so a fresh deployment never goes dark) and needs, and the policy locks no document lifts: the trial tier never serves a public task or a call that names no task, and grounding is Gemini-only. A turned-away candidate is in rejected with its sentence ("not eligible: trial tier on a public route"), never silently dropped. The migration (migrate-selection.js, pure, idempotent on a v2 document) keeps every choice: card order and pin → the Priority list; a v1 route → a custom chain with thenGlobal: true; a placement of first → that provider after the route's steps (the route ran after placement) or alone as the chain; off → exclude; everything else global. The config loader reads the catalogue beside the three documents (same cache, a catalogue that cannot be read is null) and migrates a v1 document in memory on every read, writing nothing; the router's chainDetails calls the resolver, applyFeaturePlacement and applyFeatureRoute are no longer called and their modules are marked deprecated, an empty chain for want of a capability fails with AI_NOT_CONFIGURED naming it, the CONTENTFORGE_AI_PROVIDER pin keeps its meaning (one provider the resolver found eligible, no fall-through), and every usage row gains selection: explicit | recommended | custom | global. GET cms/ai-routing answers the v2 document (migrated: true when derived from a stored v1) beside the v1 routes view the Routing tab still renders; PUT takes a v2 document (validated whole, 409 on an updatedAt that is not the one read) or the v1 { routes } body the tab sends, and stores version 2 from the first save either way — after which the "Where AI is used" placements and the card pins no longer reach the router, having been migrated into the document; slice 4 (#859) removes those controls. A null model in a chain or the list stays the provider's default for the call's purpose (DEFAULT_MODEL_TABLE), judged for eligibility on the provider's modality recommendation and reported as modalityModel; making it that recommendation outright (ADR 0034 §2) is one marked line for slice 4, when the Tasks tab lets the owner choose Recommended for the drafting tasks the owner's 2026-10-04 decision keeps on GPT-5 mini. select.contract.test.js holds the resolver, on a representative fixture of the 2026-10-04 documents (fixtures/selection-2026-10-04.js; the production export was not readable from the session) migrated to v2, to the chain the pre-slice router produced for every task under five document variants and four key sets, before and after the first catalogue refresh; the two differences are documented there (the grounded task's chain is Gemini alone, as the grounded call always was; a call naming no task may use Foundry, never the trial tier). select.test.js pins the precedence and the locks in both directions (the public explain task cannot be given NVIDIA by the list, a chain or a recommendation; a document cannot enable a provider with no key); selection.test.js the read and save rules of §6; migrate-selection.test.js the migration; admin-integrations.test.js both PUT shapes, the 409 and the "would have no model" refusal. No app setting or Terraform run: the first admin save writes the v2 document.

  • OpenAI's gpt-5-mini and gpt-5-nano are priced ($0.25 / $2.00 and $0.05 / $0.40 per 1M tokens, OpenAI's published rates, the same figures Azure lists for the same models; owner confirmation 2026-10-05). They had been unpriced since ADR 0033 pending that confirmation; rows written before stay at $0, flagged. The OpenAI card's note says so.

  • Microsoft Foundry as a paid AI provider (#849, owner decision 2026-10-04). Of a budget of about USD 75 a month, the first call was Foundry with the cheapest text models: gpt-5-nano ($0.05 in / $0.40 out per 1M tokens) for short calls and gpt-5-mini ($0.25 / $2.00) for drafts and analysis. A fifth provider foundry in the AI router on the account's OpenAI v1 chat endpoint; placed per feature like NVIDIA, first for the content features and locked off the anonymous public explain route and the grounded call; priced at the Global Standard rates in the cost table. No key: infra/foundry.tf creates the AIServices account with local authentication off, the two deployments (Global Standard, centralus), the Cognitive Services OpenAI User role for the Function App's identity, and a USD 75 monthly budget on the returning ai resource group with alerts at 50, 90 and 100 %; the router sends an Entra token through @azure/identity, cached until five minutes before expiry, and FOUNDRY_ENDPOINT (set by Terraform) is what makes the provider available. The AI Services card, its setup instructions and the Required-Inputs row follow. foundry-provider.test.js pins the request, the token cache, the placement and the pricing. Live after the owner's TFC run.

  • Setup instructions for OpenAI and NVIDIA on the AI Engine. The book icon on each provider card opens /admin/ai-engine/docs/<provider>, and for OpenAI and NVIDIA that page said "Documentation not found": the docs data (frontend/src/data/serviceDocsData.js) had entries for Gemini and Anthropic only, written before the other two were wired. Both now have the same shape as the others (requirements, where the site uses them, configure / use / disconnect / advanced, references). NVIDIA's carries the day's findings: GLM-5.3 as the only default, the pacing guard, and the wait before judging a freshly seeded key (#701).

  • ContentForge alignment: one content operating system (ADR 0033, owner request 2026-10-03). Twenty-six admin pages inventoried, then rebuilt in eight slices so a first-time user can read the menu, every stage of the pipeline agrees with every other, and nothing a feature creates is lost when the page is left. The brief, findings and acceptance criteria are in docs/decisions/0033-contentforge-alignment.md; a Figma file holds the redesigned Home, Forge Studio, Image Gallery, Calendar, Audio Library and Ambassador screens.

  • Approve works again. Every Approve button — queue, review boards, editor save bar — sent approved_blog, a status the backend normaliser never mapped, so the transition table answered 400 "Invalid transition" from every surface. The normaliser maps approved_blog and published_blog; the pages send the canonical names. The Queue "Approved" chip, the Editor list and the Frameworks / Coder Corner filters queried statuses never stored and always came back empty; they now query the stored ones, and forge_ready and needs_rework — counted in the Editor badge but shown on no screen — have a chip.
  • Home. One pipeline graphic (New Content → Drafts → Review Queue → Editor → Publish → Live Pages) replaces the two identical strips; an Explore section introduces every menu group in a sentence. Sidebar and dashboard read one counts hook that refetches on navigation, so the badges follow a Recount.
  • Sidebar reads one registry (frontend/src/config/adminNav.js) with a description per item shown on hover and repeated under each page title. Live Pages joins Pipeline; the Content group becomes Enhanced and takes Listen & Learn and Labs; Spotlight gains Ambassador; Prompts is Image Prompts.
  • Shared primitives for every hub: PageHeader (purpose line, collapsible "How this works"), StatusBadge over one status vocabulary (lib/status.js: healthy / degraded / misconfigured / unavailable / unknown; content statuses with their help sentences), EmptyState, TaxonomyPicker and TaxonomyChips, useGuardedLoad and useWriteGuard.
  • Content taxonomy. Every content record can carry kind (what it will become) and ideaOrigin (how it became an idea), two separate lists edited on Platform Settings → Content types & origins (admin_config/content_taxonomy; built-ins can be disabled, never removed). Older records derive both from type and source at read time, so filters and chips agree with no backfill. Pickers on New Content, Drafts, Forge-from-URL, Forge Studio and the Review page; chips and filters on every pipeline list.
  • Pipeline pages. One useContentTransitions hook replaces three hand-rolled approve/reject/restore copies; Frameworks and Coder Corner share one typed list; Architecture and Framework boards share one shell; Coder Corner items get their own board on the Review page. The Review page shows save and publish errors and never navigates on a failed save. Live Pages asks the server for live pages ordered by publish date instead of filtering 500 arbitrary documents. The editor's Metadata tab gains History: content_versions was written on every save and read by nothing; it now lists versions and restores one as a new version. The purge timer updates the dashboard counters it used to drift. Duplicate client audit rows removed where the server already writes one.
  • Forge Studio is a workspace, not a settings form: Start (idea, template, existing content, URL, blank) → Brief (objective, audience, kind, origin, tone, length, must and must-not cover, sources, channel, SEO) → Draft (the document is created on the Drafts page first, the forge runs against it, ten AI actions apply only on an explicit button, every AI edit recorded in activity[]) → Finish (send to review, open in editor, publish, social post, images, audio — each with a destination). The old configuration form is the fifth tab, with its zero-value, chip-save and ETag bugs fixed. The manual forge job's store lacked incrementIf, so the budget claim threw; fixed with a test.
  • AI Engine gains per-task routing (admin_settings/ai-routing: primary provider and model plus fallbacks per feature, read ahead of the global order; Simple and Advanced views), an "Auto (per task)" model option, usage recorded once per call for every call site (nine of fourteen recorded nothing), unpriced models shown as unpriced rather than at gpt-4o rates, and config invalidation after every write. A custom MCP server can no longer name an arbitrary app setting as its bearer key and must be https.
  • Image Prompts and Image Gallery share lineage. A prompt set is an image set: purpose, theme, style rules, negative prompt, version history, duplicate, rename, archive, a Generate sample, and every image it produced. The AI cover uses the assigned set's prompt and style rules (the hardcoded "Lego minifigure" prompt is the last fallback) and generated rows record set, prompt, template version, provider, model, size and path; regeneration writes a new stamped blob instead of overwriting. The gallery is a visual library: search, filters, sort, paging, bulk tag/move/archive/trash with a confirm, a details dialog with alt text, caption, licence, dimensions, "used by" and a link to the set, import from URL, persisted folders. Thirteen listed gallery bugs fixed, among them "Update selected" updating one image and tag toggles erasing tags.
  • Calendar aggregates every scheduled kind through one read (GET cms/calendar): content publishes and failures, newsletter issues, social posts, talks and CFP deadlines, certification expiry and renewal, Ambassador deadlines, audio releases. Month, week and agenda views; filters; drag-and-drop rescheduling with a keyboard alternative; unschedule; quick create; conflict marks; a refetch after every change (the old page never refetched). The Newsletter Hub embeds the same calendar filtered to its issues.
  • Newsletter. Issues reach sent: nothing moved a scheduled issue past scheduled, so metrics never appeared. Reconciliation reads the Resend broadcast on read and on demand; cancel, reschedule, retry and duplicate from the hub; per-issue send time; version history; manual content blocks (a live article or a custom block with a gallery image) beside the automatic sections; the From address is a setting; audience CSV export, add a subscriber, whole-list search. Social Hub's header reflects the real Publer accounts call, and a scheduled post can be edited and rescheduled. Recording Hub's "Send to pipeline" called a function it never imported and crashed after creating the content; fixed, with the editor links.
  • Listen & Learn is an Audio Library. Book or course → chapter or lesson → audio version, on the two existing containers. The hub opens on a Library of books with covers and durations; a book lists its ordered chapters with status, player (position remembered), Approve, Regenerate, Versions, Rename, Archive and Delete; new books from pasted text or a content item, with a cost estimate before generating. A failed regeneration no longer unpublishes a chapter (it keeps the current take and offers Retry); regeneration writes a new stamped file instead of overwriting one served under a one-year immutable header; re-running a guide keeps approvals and order; areas dropped from a guide are flagged. Text-to-speech defaults to the cheapest Gemini model with the estimate stated, chunks long dialogue on sentence boundaries, and takes per-book voice, language and rate. Publishing is publisher-gated.
  • Labs live under each provider's Learn section (/:provider/education/labs), with objectives, prerequisites, steps, resources and a panel that explains Lab, Desktop and Agent in visitor words; /education/labs stays as the index grouped by provider. The catalogue carries providers, difficulty, steps and validation, and a new lab family is one row and one template entry. The admin hub gains a Catalogue tab with Validate, and a Concepts panel with live status. Job listing and claiming are ordered; the dead running status and the stale .env instructions are gone.
  • Speaking and Certifications. Status, CFP deadline, sessions and evidence on talks; renewal date, requirements, evidence and validation on certifications; dates stored as plain days so nothing shows a day early; editors are the shared dialog; "Show on site" off now hides a Sessionize event publicly; "Feature in Spotlight" has a public effect; Publish snapshot has an effect before the next deploy (the live snapshot wins when newer); the speaker id is no longer hardcoded on About.
  • Ambassador, new under Spotlight: programs (seven seeded, all editable, disable never deletes), applications with a status machine, deadlines, renewals, files and history, an evidence library that imports from Speaking, Certifications and published content without re-entry, readiness per program that explains its arithmetic and never promises acceptance, an application workspace and packet view, and Settings as the last tab. Private by default; soft delete; audited.
  • Health probes every hub and dependency through a registry of 36 checks with one vocabulary, last-checked times (every time said "Not available": the formatter expected Firestore timestamps and Cosmos stores ISO strings), impact, fix and a deep link; Test all runs the safe ones in parallel. Integrations gains service cards for every AI key and the lab host, with persisted last-worked / last-failed status. Platform Settings opens on an index of every setting document and where it is edited.
  • Infra. One new Cosmos container, ambassador (/id), declared in the migration manifest and infra/cosmos-containers.json. It needs a terraform apply with owner review; until then the Ambassador hub shows a not-provisioned state and names the plan command, and nothing else breaks.
  • Links to the non-existent /admin/platform-settings fixed; dead exports removed from config/admin.js.

  • Code structure. Qlty measured 180 code smells the first cut introduced and 57 pre-existing ones it made worse (handler factories every slice added routes to, files past their complexity budget). All were restructured away rather than suppressed: handler bodies moved out of their factories into module-level functions over a context, request validation into tables, large components into reducers, hooks and per-section components, large modules into sibling modules behind a re-export shim. Every extracted helper has its own test (about 350 new tests); existing tests pass unchanged; four pre-existing eslint-disable complexity comments were removed because the code no longer needs them.

  • Drafts: a ContentForge stage before the Content Queue, at /admin/drafts (#840, owner request 2026-10-03). Articles drafted as docs/content/*.md files could only be read on the site by importing them straight into review. They now have a stage of their own, saved in the site: create, edit, delete and push forward from any device, with no GitHub credential and no pull request per save.

  • The page. A list of drafts on the left; on the right an editor for the seven front-matter fields the repository drafts carry (title, subtitle, date, track, part, tags, reading) and the markdown body, with a live preview through the article renderer. New draft, Save, Delete (confirmed), Send to In Review and Back to Drafts; the save state (saved, unsaved changes, saving, error) beside Save; a warning before leaving with unsaved changes, whether by switching drafts, following a link or closing the tab. Its menu entry, Drafts, sits immediately above Review Queue.
  • The data model needs no infrastructure change. A draft is an ordinary content document at the new contentStatus: 'drafting', Live: false, Status: 'Draft' — same container, same shape as the repository import's documents, no Terraform. drafting is deliberately not draft, which forgeScheduled picks up to rewrite. The state machine gains two edges, drafting → in_review (Send to In Review) and in_review → drafting (Back to Drafts, only for an article that came from Drafts and is not approved or live); both are walked only by the Drafts routes, and transitionContentStatus refuses them (DRAFTS_STAGE). A draft is kept out of the queue filters and the dashboard counters.
  • Never public. isPublicDocument refuses drafting outright, and a draft's fields fail the public list query and the manifest/pre-render query; drafts.test.js evaluates both SQL predicates, as written, against every draft shape, so a predicate change that let one through fails there.
  • Send to In Review moves the same document to in_review in the end state the old import produced — body normalised, dedup title, quality and image reports, provider from the tags when none is set — so it lands on the review board and goes on to the Publish Queue as before. Delete removes a draft or an article from Drafts still In Review (one document, so both go) and refuses anything live with a message saying so.
  • Two tabs cannot overwrite each other. Save, delete and both transitions carry the version's ETag and are written conditioned on it (patchDoc ifMatch, and a new deleteDocIfMatch); a stale one is a 412 CONFLICT, and the page keeps the owner's text and offers Reload latest.
  • Migration. "Import from docs/content" on the page reuses the existing parser, allow-list and pinned GitHub fetch to bring the ten blog-*.md articles in once, as drafts (blog-template.md and blog-machine.md are contracts and skipped). It is idempotent: a path already claimed by repoPath is skipped before any fetch, and the create is create-only under the id derived from the path. The two Docker drafts the old import put In Review show on the page as In Review, with Back to Drafts.
  • Retired: the Content Queue's "Import drafts from the repository" panel and its two routes (POST cms/content/import-repo, GET cms/content/import-repo/candidates), which wrote straight to review and refreshed in-review articles from the repository — which would now overwrite edits made on the Drafts page. The queue links to /admin/drafts instead. Eight admin-only routes under cms/drafts replace them, behind requireRole at editor, in the route inventory and the API contract. The docs/content files stay as an archive; blog-template.md and blog-machine.md say the Drafts page is the source of truth after import.
  • Docker guides: building images, and the Docker Desktop app (#772, #773). The Docker hub's first two focus areas said "Coming soon". Both now link to a page of their own, the way the sandbox recipe does (#774) and the way Terraform's sections link to theirs, and all three areas read "Available now".
  • /docker/building-images walks through lab-image/Dockerfile and publish-lab-image.yml: the five stages and the runner and full targets, the base image pinned by digest and the workflow check that holds it to versions.env, pinned sums on every direct download, the provider mirror and the vendored Azure Verified Modules, the non-root user, smoke.sh, the provenance attestation, and the push to GHCR. Each command is printed for PowerShell and bash.
  • /docker/desktop installs Docker Desktop on Windows and macOS and covers the dashboard, the WSL 2 backend, resource limits (.wslconfig under WSL 2), the labs' "Run it locally" line, and how Desktop relates to Docker Engine on the lab host. The run line is imported from the lab catalogue, not copied. The release (4.93.0 of 2026-09-28), the subscription terms and the requirements were read from docs.docker.com on 2026-10-02, and the page carries that date.
  • Both pages are static routes. They are in STANDALONE_ROUTES, so they pre-render, and they are page types in routeFactory, so the Docker hub header stays on them. DockerGuides.test.jsx checks each page's sections, commands and dates, and holds the building-images page to the Dockerfile, versions.env, smoke.sh and the workflow, so a change that falsifies a sentence fails a test. The pages pass public-copy.test.js with no new allowance: the runner image's registry name and written-out digests stay off the page.
  • Three weeks' warning before a catalogue row goes stale (#818). The catalogue alarm runs on the real clock. On the day a stored status is overtaken by its own dates, CI fails on every branch until someone re-verifies the row. #770 is that day for three rows, and until now the only warning was a hand-filed issue.
  • A new findStatusesDueWithin in certStatus.js asks the alarm's own findStaleStatuses about each day ahead, so the warning and the alarm cannot disagree.
  • frontend/scripts/catalogue-due-soon.mjs lists the rows due within 21 days, across certifications and Azure's applied skills.
  • The new Monday workflow warn-catalogue-due.yml keeps one issue, labelled catalogue-due, in step with that list: it opens the issue, refreshes it, or closes it when nothing is due.
  • The catalogue list now lives in one module, src/data/catalogues.js, shared by the alarm and the warning.

AWS's SAP-C02, DVA-C02 and MLA-C02 pages were re-read on 2026-09-29. The one-day conflict between the exam pages and the blog's Key dates is unchanged, and the file records that and keeps the blog's dates. The first rows the warning will list are SAP-C02 and SAP-C03, in the run of 2026-11-02. - Tests for the untested edges (#817). The sweep of 2026-09-29 found modules where an untested line is a security, money or data risk. Each now has a test: - telegram-http.test.js. A missing, empty, raw-token or wrong webhook secret gets 401 before the bot, the job store or the queue is touched. An unset token, or an unresolved Key Vault reference, gets 404. Once the secret is valid, bad JSON and a throwing bot still get 200, so Telegram does not retry into a storm. The enqueue writes a queued job as telegram-bot@system. - secret-vault.test.js. One slash in the vault URL, the name encoded, and a refusal naming the secret and status but never the value or body. The reference refresh reports every failure instead of throwing. - flag-gate.test.js. Only an exact "true" arms a timer; only an exact "false" master stops them all. The first skip warns, later ones log at Information. - default-agent-guard.test.js. Nothing is built at import, and it is built once. Lookups read lab_agents, and denials write agent-auth-denial to admin_audit_logs. - vps-agent/lib/api.test.js. Every POST carries the bearer token, and the agent's own agentId cannot be displaced. A refusal carries the server's status and message, and claimJob answers null when there is no job. api.js gains an injectable credential, next to its fetchImpl. - route-inventory.test.js. Every change feed has startFromBeginning: false; the content feed takes 8 documents a batch and the others 50; an empty batch does nothing. - scripts/generate-cosmos-container-spec.test.mjs. Runs the generator's --check, documented for CI but never run by any workflow, and pins content_versions on /contentId and cp_sortDate on content and blogs. - Vault on the lab host: auto-unseal with a lab-only Key Vault through the Arc identity, built and off (#726). First, the question #726 asked: can Vault 2.1.1's azurekeyvault seal sign in through the Arc agent's local endpoint with no secret on the host? Yes, from the pinned release's source: with no client_id the seal (go-kms-wrapping azurekeyvault v2.0.14) uses azidentity v1.13.1's default chain, whose managed-identity credential hands over to MSAL for Go v1.6.0, which detects an Arc machine and runs the agent's challenge-file flow; go version -m on the signature-checked binary lists those versions. HashiCorp documents managed identity only for Vault "hosted on Azure", so this rests on the code. infra/lab-hybrid.tf adds a Standard Key Vault kv-labhybrid-prod-cus-01 (RBAC, purge protection, prevent_destroy, AuditEvent to the Management workspace; the pattern's kv-lab-hybrid-prod-cus-01 is one character over Key Vault's 24), one RSA 3072 key vault-seal allowing only wrapKey and unwrapKey, created through Resource Manager so the run identity needs no data-plane role, and the Arc identity's only grant, Key Vault Crypto Service Encryption User on that key, for the principal read from the Arc machine at plan time (nothing is planned while it is absent). About $0.13 a month, from Vault's 10-minute seal health check at $0.15 per 10,000 advanced-key operations. The vault role gains an opt-in seal "azurekeyvault" stanza with no client id and no secret, switched on by the host fact /etc/ansible/facts.d/hcw_vault_seal.fact as Arc's is; it reads the key as the Arc identity before writing the stanza, refuses to drop the stanza while Vault's data is under it, pins the credential chain to managed identity, adds the himds group to the unit only, and lets systemd retry every 30 seconds while Key Vault is unreachable. With the seal off the unit and configuration render byte for byte as before, so merging restarts nothing. bootstrap.sh --tags vault_seal_check runs the checks alone. The migration from Shamir (vault operator unseal -migrate, each key at Vault's own hidden prompt) is a runbook section of owner lines, rehearsed on 2026-09-29 against the real 2.1.1 binary in an Ubuntu 26.04 container with stand-ins for the agent and Key Vault: the refusals, the migration, Sealed false after systemctl restart vault, a Key Vault outage across a restart, both ways back, and a rebuilt host. The rehearsal found that a restart before Vault logs seal migration complete leaves it migrating again, so the runbook waits for vault status to drop the migration line. ADR 0032's amendment of 2026-09-29 records the decision as proposed and the trade as the owner's to accept: with auto-unseal, root on the host together with the Arc identity can unseal Vault, and the five keys become recovery keys that cannot. scripts/lab-host-vault-seal.test.mjs holds the vault, key and machine names in infra/ and the role together and pins the grant's shape; assert-expected-plan.test.mjs now compares its permanent-replacement list with azapi's patch resources only, since an azapi_resource is an ordinary managed resource.

  • Docker news feed, podcast page and home hub (#777).
  • News feed. PROVIDER_FEEDS.docker in functions/src/lib/rss/feeds.js reads Docker's blog at https://www.docker.com/feed/, the address https://www.docker.com/blog/feed/ redirects to. On 2026-09-29 the ingest's own parser and User-Agent read it as RSS 2.0 with 10 items, the newest from 2026-09-24. The feed's display name is Docker. /docker/news and /docker/rss are now the shared NewsPage, which already had Docker's NEWS_META row, and rssFeeds in ProviderContext names the feed. scripts/verify-timer-witness.mjs queries Docker's feed route too; its test already held its list to the feed catalogue's keys.
  • Home page aggregate. Nothing on the site reads the homepage_feeds aggregate today. With a ninth provider in it, it now carries nine first-pass items and one second-pass item, where there were eight and two.
  • Podcast.
    • PodcastPage has a Docker row in PROVIDER_META.
    • ProviderAudioDispatcher routes Docker there. It is now one membership test against VALID_PROVIDERS instead of a branch per provider; Docker would have been the tenth identical branch. provider-coverage.test.js reads that form as "every provider".
    • PODCAST_PROVIDERS names Docker, on the server and in the admin copy.
    • /docker/audio and /docker/audio-architecture render and pre-render instead of answering 404.
  • Home page. "Quick Access Hubs" ends with a Docker hub linking to /docker/rss, marked "NEW HUB" like VMware's and Ansible's. The hubs were eight copies of one <Link> block, and a ninth tripped Qlty's similar-code check. They are now data, QUICK_ACCESS_HUBS, rendered by one block. Each hub keeps its own classes as whole literal strings, since Tailwind only generates classes it finds written out, and the built page carries the same nine links with the same classes.

  • Newsletter. Nothing changed: newsletter/issue.js has nothing keyed by provider, so Docker items reach the weekly issue as they are published.

  • Docker in the CMS: an article can be written, classified, voiced, illustrated and previewed as Docker (#775).

  • Classification. The analysis prompt in content/inspect.js lists Docker among the providers an article must be filed under, as a service provider. It files Dockerfiles, image builds, Docker Desktop, Compose, Docker Hub, Scout and Sandboxes under Docker, and containers on another provider under that provider (Docker images on Azure Container Apps are Azure, a workflow that builds an image is GitHub, a module that deploys containers is Terraform), the same rule as docker being last in lib/providers.js.
  • Voice. content/voice.js has a docker voice (a container practitioner quoting real Dockerfile stages, compose services and docker commands).
  • Cover art.
    • ai-cover.js has a Docker theme for AI covers.
    • cover-svg.js has a Docker palette (#1D63ED) for template covers.
    • HERO_PROVIDERS names Docker on the server and in the admin copy, and a new test holds the two lists to each other.
    • generate-brand-art.mjs gains Docker in PALETTES, COVER_PROVIDERS and HERO_PROVIDERS, and npm run art:generate produced public/images/default-heroes/docker.png and public/images/docker-hero/1–5.png. Every existing PNG regenerated byte-identical.
    • The Docker landing page rotates those five images, as Terraform's does.
  • Image prompts. The server allowlist in cms/image-prompts.js and ImagePromptsPage both name the seven Docker pages (hub, news, blog, code, sandboxes, tools, education), and a test holds the admin page's Docker group to the allowlist.
  • Submissions and admin.
    • Docker is offered on the blog and Coder Corner submission forms, but not on architecture or frameworks, since Docker has no such pages.
    • Docker is in the editor list's filter and the image gallery's provider tags.
    • pageMeta.js infers Docker from a URL, last, so a URL about Docker on Azure, GitHub or Terraform stays with that provider.
  • Staging preview. PreviewPage reads VALID_PROVIDERS instead of its own copy, which lacked Docker, so a Docker draft now previews as Docker, not AWS.

  • CI. ci.yml's functions row also runs when either of the two admin files the new parity test reads changes.

  • Docker education catalogue, checked on 2026-09-29; /docker/education is a real Learning page and Docker is on the Learn index (#778).

  • Docker runs no certification exam of its own. Docker's own list is docker.com/trainings, and it offers self-guided modules, a YouTube series, and one credential that LinkedIn Learning issues. src/data/docker/education.js says so as data (DOCKER_RUNS_OWN_EXAM = false). The page says it first, in visitor words (CREDENTIALS_NOTE).
  • Two credentials, each naming its issuer, which the card shows as "Issued by …":
    • the Docker Foundations Professional Certificate (LinkedIn Learning: three courses of about four hours, then a final exam);
    • the Docker Certified Associate (Mirantis: 55 questions in 90 minutes, USD 199 or EUR 200, valid two years). docker.com does not list it.

  • Seven learning paths. Six are the modules docker.com links:

    • "Build and share a containerized application", 15 minutes. The link on docker.com now redirects here.
    • "Building images", Beginner, 25 minutes, five modules.
    • Compose.
    • Docker Scout.
    • Build Cloud.
    • The administrator set-up guide.

    The seventh is the LinkedIn path. Each title, module list, level and time is the page's own, and nothing is estimated. - Four resources: Docker Training, the documentation, the guides and the Docker Concepts playlist. - EducationTracks takes three optional fields. A path may carry linkLabel and duration, and a certification may carry issuer. Every other catalogue renders as before, which a pathMeta test pins. - Learn index. EducationIndexPage lists Docker as a ninth catalogue, and the copy, meta description and table caption say "nine". The table's minimum width is 72rem, up from 64. A test holds PROVIDER_CATALOGUES to VALID_PROVIDERS. - No Docker architecture blueprints. #778 asks for them only "if Docker architecture designs are wanted". The Docker hub's three focus areas do not include them, so /docker/architecture-designs and /docker/frameworks still answer 404, as Terraform's do.

  • Docker blog, news and code detail pages; Docker in the content manifest and the public API (#776). /docker/blog is now the shared ProviderBlogPage ("Docker Containers Blog"), listing published articles filed under Docker. /docker/code is the shared code listing ("Docker Code Patterns"), like Terraform's and Ansible's. Both replace the placeholders, and with nothing published each says so the way every provider's does. App.jsx routes /docker/blog/:slug, /docker/news/:slug and /docker/code/:slug to BlogDetailTemplate; before this they were the 404 page. docker joins PROVIDERS in scripts/build-content-manifest.mjs, so a Docker article pre-renders at /docker/blog/<slug> and enters the sitemap. That list is now exported and held to VALID_PROVIDERS by a test. ProviderBlogDetailDispatcher and ProviderNewsDetailDispatcher are now one membership test against VALID_PROVIDERS instead of a branch per provider (ten returns each once Docker was added), and a route test renders every provider's blog and news detail path, so no provider's pre-rendered article can be a 404. The four code pages (GitHub, Terraform, Ansible, Docker) were copies of one body and now share components/shared/ProviderCodePage.jsx, each passing its provider, title and description. docker: ['Docker', 'docker'] joins PROVIDER_ALIASES in functions/src/lib/public-reads.js: ?provider=docker now matches documents filed as "Docker", and listCertEvents answers platform=docker with an empty list instead of a 400. The section counts derive their providers from that table, so once Functions is deployed and the manifest next published, the sitemap leaves out Docker sections with nothing in them, as it does for every other provider.

  • Docker article series: two how-to drafts for review (#772, #773). docs/content/blog-docker-01-building-images.md, "Building a lab image you can trust", reads lab-image/Dockerfile as the worked example: base images pinned by digest and the workflow step that fails an unpinned FROM, every download checked against versions.env, the five stages and the runner and full targets, the provider mirror and the Azure Verified Modules vendored under /opt/avm, uid 65534 with every write under /tmp, smoke.sh, provenance from actions/attest-build-provenance, and the Docker Hub copy by digest through Docker's OIDC connection with no stored token. It ends with gh attestation verify oci://docker.io/hybridcloudworks/hcw-lab:latest --bundle-from-oci, plus a stricter form pinned to the signer workflow and refs/heads/main. blog-docker-02-desktop.md, "Docker Desktop for the labs", installs Docker Desktop on Windows (WSL 2 backend, per-user) and macOS, walks the Containers, Images, Volumes and Builds views, sets memory, disk, file sharing and Apple silicon emulation for the lab image, runs the labs' follow-along line in PowerShell and bash, and compares Desktop with the Docker Engine on the Hybrid Lab host. Docker Desktop's current release (4.93.0 of 2026-09-28, Engine 29.8.1) and its subscription terms (the Docker Subscription Service Agreement last updated 2026-08-26, and Docker's licensing page) were checked on 2026-09-29 and are dated in the article. Every output quoted was measured on 2026-09-29 on Windows 11 with Docker Desktop 4.93.0 (WSL 2 backend): the full build, smoke.sh passing offline and refusing with the network on, the digest match between the two registries, both gh attestation verify forms and a refused one, the follow-along mount writing as nobody, a uid 1000 mode 755 directory refusing that write, and the bash lines in Git Bash failing (an empty /workspace, and C:/Program Files/Git/workspace/smoke.sh). Not run, and the Desktop article says so: the Windows and macOS installers, wsl --update, the .wslconfig change, and anything on a Mac. Listed in the docs nav marked "(draft)". Nothing is published; both land In Review only when imported from the Content Queue after merge.

  • Lab image: publishing to Docker Hub through a Docker OIDC connection, with no stored token (#779). Owner decision 2026-09-28. publish-lab-image.yml has a third job, Publish to Docker Hub, which runs after Publish to GHCR and only while the repository variable DOCKERHUB_ENABLED is true. It signs in with docker/login-action v4.6.0 (the pin the workflow already used; OIDC arrived in v4.5.0), which exchanges the job's GitHub ID token (audience https://identity.docker.com) for a Docker access token that lasts 300 seconds. That needs the connection ID in DOCKERHUB_CONNECTION and no password. It then copies both images from GHCR to docker.io/hybridcloudworks/hcw-lab and hcw-lab-runner by digest with docker buildx imagetools create --prefer-index=false, checks that the digest Docker Hub reports equals GHCR's, and attests each image under its Docker Hub name with push-to-registry. The digest is kept because, measured with buildx v0.37.1, the flag copies the manifest's own bytes, while the default wraps a single image in a new manifest list with a new digest. id-token: write sits on that job and publish only. A separate job, so GHCR never waits on Docker Hub: a Docker Hub failure turns the run red after GHCR has published and attested, and "Re-run failed jobs" repeats only the copy. Until the owner creates the connection the job is skipped. The Docker side trusts one subject, this repository's main in GitHub's immutable-identifier form, with no environment, because an environment subject carries no branch. New runbook docs/runbooks/docker-hub-publishing.md (the owner's steps, every form value, what success looks like, a failure table, and a one-time manual copy if the organisation's plan cannot create a connection yet); new §4.11 and two §4.2 rows in Required inputs; store 3 rows in Variables and secrets; Docker Hub pull and verify commands in lab-image/README.md, whose GHCR note now records that both packages answer anonymously. The site's "Run it locally" copy and the Docker pages still name only GHCR until a run has published to Docker Hub.

  • The repository is licensed under Apache-2.0. Owner decision 2026-09-28, #780. LICENSE at the root is the Apache License 2.0 text byte for byte as https://www.apache.org/licenses/LICENSE-2.0.txt publishes it (11,358 bytes, LF, SHA-256 cfc7749b…c523d30). .editorconfig gives it indent_size = unset, because 24 of its lines are indented by an odd number of spaces and the text is not reflowed. NOTICE names HybridCloudWorks and carries "Copyright 2026 Saul Patino and HybridCloudWorks contributors". It also says that third-party names, logos and marks (the vendor logos under frontend/public/icons/providers/ and frontend/src/assets/brands/) belong to their owners and are not licensed under Apache-2.0, and that other third-party material, such as the fonts under frontend/public/fonts/, keeps its own terms. scripts/validate-repository-structure.ps1 allows both files at the root and fails when either is missing. The six npm packages (frontend, functions, scripts, vps-agent, edge/availability-probe and lab-host/coder) declare "license": "Apache-2.0". The five lockfiles carry it on their root entry exactly as npm 11.17.0 writes it: each was regenerated from a scratch copy and matched byte for byte. frontend/scripts/package.json is a CommonJS boundary marker, not a package, so it has no licence field. The eleven lab-host Ansible roles said license: UNLICENSED in meta/main.yml and now say Apache-2.0. The site footer drops "All rights reserved." and keeps "© Hybrid Cloud Works.", because a blanket reservation on every page contradicted the code licence. The Terms page's content clause, which begins "Unless otherwise stated", is unchanged. The README has a Licence section. The docs site had nothing that conflicted, and mkdocs.yml's copyright: HybridCloudWorks stays.

  • Home: the providers in two rows with a guide under them; Docker as a service provider, with placeholder pages. Owner request 2026-09-28. The home page strip is two evenly spaced rows, Azure, AWS, GCP and VMware, then GitHub and FinOps, a pipe, and Terraform, Docker and Ansible; each label still links to its hub. On a phone row 2 wraps into its two groups (two over three) and the pipe is dropped, with no horizontal scroll at 360 to 414px. Pointing at a provider, focusing it or tapping it shows a frame one blank line below the rows: a paragraph that says what kind of provider it is (cloud, framework or service) and how this site uses it, checked against the repository, with an icon at the bottom-right that opens the provider's website in a new tab (rel="noopener noreferrer", named "... website (opens in a new tab)"). The frame never changes height: every text it can show is rendered into it invisibly, so it is as tall as the longest at any width, and at 768px and up that is exactly four lines, measured for all nine at 768, 1024 and 1440px in both themes, with the next section's position unchanged. The last provider shown stays shown, a 90ms hover intent keeps a pointer passing over another label on its way to the icon from taking the frame over, the first tap shows and the second follows the link, and the text is a polite live region that neither moves nor holds focus. The copy and types are data in frontend/src/components/home/providerGuide.js, where providerGuide.test.js pins the nine providers, rows, types, websites and a 320-character ceiling. Docker joins the provider registry (providers.js, with docker last so container writing about another provider stays with it; VALID_PROVIDERS, routeFactory.ts, App.jsx, the pre-render list, the header, --docker-blue #1D63ED and a theme-docker, and a Simple Icons mark, CC0) with a landing page on the Terraform template and three focus areas marked coming soon (building images with hcw-lab as the example, the Docker Desktop app, and running an agent in a sandbox, which links to the recipe on /education/labs for now), plus static Blog, Code, Learning, News and Tools placeholders that read nothing from the API. Eight Docker pages pre-render (196 documents, from 188). The header needed room for a ninth provider: the all-providers list now sizes its columns to their labels, the logo can no longer be squeezed out (it was zero pixels wide on the home page at every desktop width, and a few pixels on the hubs at 1024 to 1280px), the empty box beside the menu button gives its width back from lg, and the mobile menu scrolls inside itself instead of running its last links off a 740px screen.

  • Labs page: Coder is credited beside the intro, with its cursor blinking; the intro no longer says the host is onboarded to Azure Arc. Owner request 2026-09-28. On /education/labs the column to the right of the intro was empty on desktop; CoderCredit (frontend/src/components/labs/CoderCredit.jsx) now fills it from xl, top-aligned, with the intro kept at its 48rem, and stacks under the intro at full width below that (a row on a tablet, a column on a phone; no horizontal scroll at 390 px). It shows Coder's wordmark as inline SVG in the theme's text colour and says "We provide these browser labs using Coder, the open-source platform for self-hosted development environments. Every lab workspace is a Coder workspace built from our lab template.", with Coder linking to https://coder.com/ (the site's canonical) in a new tab, rel="noopener noreferrer", named "Coder (opens in a new tab)". No sponsorship is claimed. The square that ends the wordmark blinks like a terminal cursor (.coder-cursor in src/index.css): a 1 s step blink, five times after load and again while the pointer or focus is on the credit, never under prefers-reduced-motion: reduce; five and not forever because WCAG 2.2.2 asks for a pause control on blinking past five seconds. CoderCreditMedia is the one place to swap in a looping mp4 if the owner supplies one. The owner's two logos, cleaned, are kept in frontend/src/assets/brands/coder/, each opening with a comment saying it is Coder's mark, kept only to credit Coder (a comment rather than a README, which the repository policy keeps under docs/); the test fails if the inline wordmark differs from the file. The intro's "a single server onboarded to Azure Arc" became "a single server whose Azure Arc status is on the live card further down this page", and the estate card's intro stopped asserting onboarding too, since #663 has not run yet; both stay true after it does. CoderCredit.test.jsx (11 tests) covers the link, the names, the cursor class, the two files' attribution and the stylesheet's finite blink and reduced-motion rules; public-copy.test.js scans the new file and passes the new sentences.

  • The lab template is published from the host; lab workspaces open trusted, without the AI chat; a missing template no longer shows Coder's error in a pane. Three things from the first live workspace on 2026-09-28. The publishing steps began with winget install Coder.Coder, which reported success and installed nothing, so that day the template was published by hand from the coder container. The coder role now installs /usr/local/sbin/hcw-coder-template-push (root:root 0750), and publishing is one PowerShell line with the owner's short-lived token on the clipboard (lab-host/README.md, "Publishing the template"); nothing is installed on the workstation. The helper takes the token from stdin only (never an argument, never printed, never written to a file), copies every *.tf, .terraform.lock.hcl and README.md from the checkout in /opt/hcw-src, and not the test beside them, into a fresh mktemp directory in the container, runs coder templates push hcw-lab and coder templates edit hcw-lab --default-ttl 1h there against http://127.0.0.1:7080, checks that one version is active and that Coder reports 1h0m0s, prints both, removes the copy on every exit and exits non-zero on any failure. The autostop is coder_template_default_ttl in the role's defaults, whole hours, asserted by the role and rendered into the helper. roles/coder/tests/hcw-coder-template-push.test.sh renders the helper with the role's defaults and runs it against a stub docker and a fake coder in the ansible-lint (lab-host) job. bootstrap.sh does not publish by itself: a token that can publish is as good as root on this host, because the provisioner runs beside the Docker socket, and Coder's API answers it from anywhere (the role's README has the reasoning). Before the template existed, a lab's pane loaded Coder's create page, which answered with Coder's own red "Resource not found" box and a stack trace. The lab launcher now reads GET /api/v2/organizations/default/templates/hcw-lab, the read Coder's create page makes first, once before it frames that page; a 404, a deprecated template or another template's body ends the visit with the site's sentence, Lab workspaces aren't available right now., and no Coder page is loaded. scripts/lab-host-launcher.test.mjs (99 tests) and lab-host-visitor-copy.test.mjs cover it. The workspace template gives code-server four User settings through the pinned module's settings input, checked against module 1.6.0 and VS Code 1.139.1 in code-server 4.139.1: security.workspace.trust.enabled: false (the first workspace opened in Restricted Mode), chat.disableAIFeatures: true (the Chat panel opened asking for a sign-in the site does not offer), workbench.startupEditor: "readme" and telemetry.telemetryLevel: "off"; template.test.mjs pins them and both versions. The owner check in "First admin sign-in" runs the CLI in the container as well.

  • Lab workspaces: the editor opens inside the site's pane. Owner decision 2026-09-28: the lab is reached only through panes on the site, and the panes-only lock is not loosened. Coder's dashboard opens code-server in a new window or tab (open_in has no same-frame value in v2.37.3), which the lock turns into the labs page, so the editor could never open in a pane. A lab's pane now loads the lab launcher, https://coder.lab.hybridcloudworks.com/_hcw/lab/?lab=<id>: four static files in lab-host/coder/launcher/ (no build step, no third-party code) that the coder role installs in /etc/caddy/hcw-lab-launcher and Caddy serves before handle @coder. With the learner's own session, GET only and same origin, it maps the lab through a fixed allowlist to one workspace per learner (lab-lzb, lab-tfv, lab-asc; 16 characters at most, so code-server--<workspace>--<owner> fits one DNS label with a 32-character username), reads /api/v2/users/me (a 401 asks the visitor to use Sign in with GitHub above the pane) and their workspace. A missing workspace gets Coder's own create page with its consent dialog, and a stopped one Coder's own workspace page, in a frame of the same origin; no POST, no CSRF token. It waits (one to five seconds apart, ten minutes or six failed reads at most) for the build to run, the agent to be connected and ready, and code-server's health to be healthy, checks subdomain_name against ^code-server--[a-z0-9-]+--[a-z0-9-]+$ and requires it to be exactly this workspace's name for the signed-in username (/api/v2/users/me), so it can only open the learner's own editor, and location.replaces itself to that name under a fixed suffix, so code-server keeps its own origin. It posts { type: 'hcw-lab', state } to both site names. Caddy adds a strict policy beside the panes-only frame-ancestors (default-src 'none'; scripts, styles, fetches, images and frames from coder.lab only; base-uri 'none'; form-action 'none'), Cache-Control: no-store and nosniff, and no top-level exemption, so a direct visit still redirects. The role copies the four files by name, removes anything else from the directory Caddy serves, and refuses a launcher directory that is not /etc/caddy/hcw-<name>, since it removes that directory whole while Coder is disabled. On the site, LabPanePage.jsx frames the launcher (frame-src, sandbox and allow unchanged), takes a message only from Coder's origin, the pane's own window and a known type and state, shows the state on the toolbar, makes Sign in with GitHub the main action when the launcher finds no session, and treats any message as the frame loading; unavailable shows the page's own unavailable section. The catalogue gains workspaceName, and catalogue.test.js and the new scripts/lab-host-launcher.test.mjs both fail when it and the launcher's map differ; the CI filters now run each side's test when the other side's file changes. scripts/lab-host-launcher.test.mjs (81 tests) drives the logic with a mocked fetch, clock and navigation: the allowlist, the 401 path, create, stopped, starting, the healthy redirect, malformed names and another learner's, the backoff and the cap. It also holds the files (one module script, nothing inline, text only through textContent) and the route. lab-host-visitor-copy.test.mjs scans every launcher message and the page's words, and holds its unavailable to the site's sentence.
  • The lab panes no longer depend on the status token. coder-status now says configured with a valid CODER_URL alone and reachable from Coder's unauthenticated GET /api/v2/buildinfo. CODER_STATUS_TOKEN adds only the templates and the running count; missing, or refused with 401 or 403, the answer is templates: [] and capacity.running: null, and the Function App logs a warning naming the setting, never its value. So the owner's first sign-in works in a pane, and the token's one-year expiry can no longer close every lab. The response shape is unchanged. Every call to Coder now refuses to follow a redirect, which Node's fetch would have followed to another origin with Coder-Session-Token still on it. CoderStatusCard then says only "Coder is reachable." and lists nothing; it had been about to print "0 of 5 workspaces running" and "No lab template is available yet", because capacityWords read Number(null) as zero, which also affected the estate card and is fixed there too. lab-host/README.md, "The status token for the site", now makes the owner's CLI token in Coder's own Tokens page inside a pane, named hcw-setup with the shortest expiry, instead of copying the session cookie out of the browser's developer tools; the hcw-status Template Admin user and its scoped one-year token are unchanged.

  • hcw-vault-set refuses a key the playbook defines. bootstrap.sh passes vault.yml with -e, which outranks every other variable, and the helper took any vault_* name, including the vault role's own (vault_enabled, vault_api_port and so on). It now reads /opt/hcw-src at run time (HCW_SRC_DIR in tests) and refuses, with exit code 2 and the defining file named, any top-level variable of group_vars/, host_vars/ or a role's defaults/ or vars/, and any name a play, the inventory, or a role's task or handler sets (vars, register, set_fact in every spelling); with no checkout, no role, or a file that does not parse, it refuses every key. That catches the vault role's task vars vault_seal_status and vault_state_advice too. None of the thirteen keys in use collides. Its Python now runs isolated (-I), so a yaml.py in the directory sudo ran from can no longer be imported as root. hcw-vault-set.test.sh grows from 39 to 80 checks, and fails without -I.

  • Coder is on for members of the HybridCloudWorks GitHub organisation only, and the vault helper is under Ansible. Owner decisions 2026-09-28: turn Coder on; only members of HybridCloudWorks may sign in; the owner signs in first and becomes Coder's owner, and nothing here creates a Coder user. lab-host/ansible/group_vars/all.yml sets coder_enabled: true and coder_oauth2_github_allowed_orgs: [HybridCloudWorks]; the three vault_coder_* keys were already in the host's vault (checked by name). The Compose file already held what the live host needs (PostgreSQL 18.6 at /var/lib/postgresql, Coder v2.37.3 by digest, CODER_REDIRECT_TO_ACCESS_URL=false from #723, password sign-in off, no CODER_FIRST_USER_*), as did #750's route. One setting is added: CODER_DISABLE_PATH_APPS: "true". Coder v2.37.3 otherwise serves every workspace app on the dashboard's origin too, at /@<owner>/<workspace>/apps/<app>/, where a workspace's JavaScript can call the Coder API. Coder's own guidance is to turn that off whenever a wildcard access URL exists, and the template's code-server is a subdomain app. template.test.mjs now fails without it (11 tests). A new vault_tools role, straight after hardening, makes /etc/hcw/ansible root:root 0700 and installs /usr/local/sbin/hcw-vault-set (root:root 0750), which had been put on the host by hand. The helper sets one vault_* key from stdin without printing it. The role's copy fixes two defects in the hand-installed one. install unlinked vault.yml before writing its replacement (seen with strace on 26.04's uutils install), so a run stopped in between lost the vault; the copy is now written beside it and renamed over it. And a leading byte order mark on the piped value was kept, because str.strip() leaves U+FEFF. The first run after merge reports that task changed. roles/vault_tools/tests/hcw-vault-set.test.sh (39 checks, run as root in the ansible-lint (lab-host) job against a scratch HCW_VAULT_DIR with the job's own ansible-vault) sets two keys and replaces one. It refuses an empty value, seven bad key names, a missing password and a wrong one, each time leaving the vault byte for byte. It checks the file is $ANSIBLE_VAULT;1.1;AES256 ciphertext, root:root 0600, that YAML-like values come back as the same strings, and that nothing temporary remains. Against the hand-installed copy it fails the two BOM checks. lab-host/README.md, "The vault", now sets keys with one PowerShell line per key. You paste the line, copy the value, then press Enter; the PostgreSQL password is made on the host and never shown. With Coder on, a first run without a vault now stops at the coder role, after hcw-vault-set is installed. The README, the reinstall runbook and bootstrap.sh's header say so. The Coder section's status-token steps were measured on Coder Community v2.37.3. The token is scoped template:read and workspace:read, minted by the owner with coder tokens create --user for a user of its own, hcw-status. That user holds Template Admin, the only role below Owner that counts other users' running workspaces (Member, Auditor and User Admin answered count 0). It is a GitHub user at a .invalid address, because --login-type none needs Premium service accounts in v2.37 and password users are refused while password sign-in is off. Unscoped, the same user's token deleted a template; scoped, it got 403. The owner runs the steps from the workstation through ssh hcw-lab and the coder container's CLI, and the token goes to the clipboard, never the screen, before it is seeded at https://hybridcloudworks.com/admin/integrations?tab=keys. The site's panes open only once the card reads reachable (#758), so the owner's own first token is the GitHub sign-in's session, copied from the browser's developer tools. Turning Coder on in the container rehearsal also surfaced a latent defect. coder_workspace_memory_mib and coder_server_memory_reserve_mib had defaults only in the role's argument spec, which Ansible never applies, so the first run with coder_enabled: true stopped at the capacity assertion with 'coder_workspace_memory_mib' is undefined. Both are in the role's defaults now. lab-host/ansible/check-argument-spec-defaults.py, in the same CI job, fails on any option across the 11 roles whose default is written only in its spec, or whose default disagrees with it. What a pane can show from the lab host now speaks to visitors. Coder stopped answers Lab workspaces aren't available right now., the site's own UNAVAILABLE_SENTENCE, and the apex and unclaimed names answer There's no lab at this address. The fail-closed HTTP-only 503 keeps naming the missing vault key for the operator, because an http:// page cannot render in the site's pane. scripts/lab-host-visitor-copy.test.mjs scans every respond body in the three Caddy templates for tool, host and setting names. It pins the Coder answer to the pane's sentence and keeps the operator body on the HTTP-only site.

  • Azure Arc onboarding of the lab host is two owner commands: scripts/lab/Register-LabArc.ps1, then the same with -Connect (#663). The runbook's ten manual steps (a year-long secret from create-for-rbac, a hand-edited vault, a pull request to flip arc_enabled, four az commands for the monitoring, and a manual deletion afterwards) become one run before the hcw-azure apply and one after it. The first run resolves sub-app-site-prod-cus by name and checks rg-lab-hybrid-prod-cus exists. It reads the host over ssh (agent state, arc fact, the vault helper, and which vault_arc_* keys the vault holds, by name only), finds or creates the single-tenant sp-arc-onboarding-lab-hybrid-prod-cus with no role and no credential, and grants id-plat-terraform-prod-cus-01 Resource Policy Contributor on the group only, so the audit policy can be applied. Then it mints a 24-hour client secret with az ad app credential reset --append --end-date and pipes it from a variable into /usr/local/sbin/hcw-vault-set over ssh's standard input with its three identifiers. The value is never printed, written to disk on the desktop, or put on a command line, and it is written as exact bytes with a bare line feed, because a PowerShell pipe on Windows would add a carriage return. The run reads the vault back and deletes the new secret from Entra again if the vault does not hold it. Finally it prints the service principal's object id, the two hcw-azure variables (arc_onboarding_principal_id, HCL off; lab_hybrid_policy_enabled, HCL on, which ADR 0032 decision 3 and #663's acceptance both need) at the workspace's variables page, and the plan to expect, Plan: 5 to add, 1 to change, 3 to destroy. The onboarding role assignment stays Terraform's; the script only reads it. -Connect checks that grant (read only) and stops if a machine resource from an earlier host is in the way. It keeps or re-mints the secret, writes the host's arc fact, runs bootstrap.sh, and waits for Connected. It then deletes every client secret on the registration and all four vault_arc_* keys, with a decrypt, drop, verify, re-encrypt and compare rewrite on the host that leaves every other key byte for byte, and reads both back empty. Last, it installs the Azure Monitor Agent extension and associates dcr-lab-hybrid-prod-cus through az rest (Hybrid Compute API 2025-01-13, Monitor 2022-06-01), so no CLI extension is needed. When connect fails it removes the fact it wrote, so later runs are not stopped at the role. Both runs are idempotent and take -WhatIf. arc_enabled now comes from the host, not the repository: group_vars/all.yml reads the local fact /etc/ansible/facts.d/hcw_arc.fact and is true only for "enabled": true. A rebuilt host is not onboarded whatever the repository says, and a repository-wide true would stop its first bootstrap.sh run at the role's fail-closed vault check, before every later role. Checked on a systemd Ubuntu 26.04 container with the pinned ansible-core 2.21.4. With no fact, the argument spec validated and all 17 arc tasks were skipped. With the fact, the role installed and held azcmagent 1.68.03532.1399, read Disconnected and stopped at its vault check. A missing, non-JSON, "enabled": "yes" or list-shaped fact reads as false. scripts/lab/Register-LabArc.Tests.ps1 adds 126 Pester tests. They test the Entra, Azure and host steps against mocked seams, the secret never appearing in output or errors, and the vault helper receiving exactly the value and one line feed. The host scripts run under bash against stand-in ansible-vault and azcmagent, and the whole script runs, with and without -WhatIf, against stand-in az and ssh that refuse every write. docs/runbooks/labs-host.md "Arc onboarding", lab-host/README.md, the arc role, and Required inputs §4.1 and §4.7 now describe this procedure. The script has not yet run against the tenant or the host. That is the owner's step.

  • Lab workspaces open as panes on the site (#751). Owner decision 2026-09-28: the lab is reached only through panes on the site. #750 made the lab host send every top-level visit, Coder's included, back to /education/labs, so the cards' "Open in Coder" links had become a loop. Each lab now has a page, /education/labs/<id> (frontend/src/pages/shared/LabPanePage.jsx), that frames the workspace deep link /templates/hcw-lab/workspace?mode=auto&param.lab=<id>, checked against the hcw-lab template's name and its lab parameter. The card button, now "Open lab workspace", is an in-site link to it. The frame's sandbox is allow-scripts allow-same-origin allow-forms allow-popups and its allow grants clipboard read, clipboard write and fullscreen to Coder's name and its workspace apps' wildcard, each justified in the page. The toolbar makes the pane itself full screen and goes back to the labs page. staticwebapp.config.json's frame-src gains exactly https://coder.lab.hybridcloudworks.com and https://*.coder.lab.hybridcloudworks.com; connect-src is unchanged, and csp.test.js holds both. GitHub cannot be framed, so sign-in runs in a tab of its own: "Sign in with GitHub" opens Coder's /api/v2/users/oauth2/github/callback, the one path #750 lets through at the top level and where Coder v2.37.3 starts sign-in when no code is present. Coder's closing redirect is a top-level visit that #750 sends to /education/labs, which sends the tab on to the lab's pane (components/labs/labSignIn.js) and, through a storage event, reloads the pane in the first tab. The site records only that a sign-in started and finished, in localStorage, never a credential. The pane opens only when GET public/labs/coder-status says Coder is configured and reachable, because a cross-origin frame cannot report an error page; every other answer, and a frame that has not loaded in 30 seconds, reads "Lab workspaces aren't available right now." Visitor wording names no tool behind the site, and #755's guard now scans the new files and passes the new sentences. The three pane pages are pre-rendered from the catalogue, like the certification pages. ADR 0032 records embedding Coder, rejected when it was accepted, as the chosen approach (amendment "Coder in the site's panes"), and docs/architecture/labs-host.md says what the site side now does. Found while building it and recorded there: Coder opens a workspace app such as code-server in a new window or tab, which #750 redirects, so code-server does not yet open inside the pane. Frontend tests 3,058 to 3,111 (201 to 204 files), on Node 26.

  • The lab host holds every pinned job image before a job needs it, and removes the digests no pin names. The first public "Validate on the lab" job on the live host (2026-09-28) spent its opening seconds pulling the hcw-lab-runner image, about 320 MB, while the visitor waited and the job log filled with pull lines. A new lab_images role runs straight after labs_agent in lab-host/ansible/site.yml. Its plan, roles/lab_images/files/lab-images.mjs, runs on the host's Node.js and imports IMAGES from vps-agent/lib/capabilities.js in the playbook's checkout and the agent's, so the list is the module itself and a pin bump needs no second edit. It also reads the Coder workspace image from the template's image and image_digest locals, which is pulled while coder_enabled is true and kept either way. A pin that is not name[:tag]@sha256:<hex> fails the play naming the entry, and so does a pull that fails. Pulls are community.docker.docker_image_pull by repository@digest with pull: not_present, so a present digest is ok; the tag is dropped because the module splits name:tag@digest at the @ and would pull on every run. Removal is docker image rm by reference, because that module's docker_image_remove and docker_image rebuild a digest reference as name:sha256:..., which Docker 29.8 refuses with invalid reference format. It covers only the pins' repositories and ghcr.io/hybridcloudworks/hcw-lab*, never lists a reference in any other repository, and keeps an image a container still uses until a later run. The runner passes no --pull, so docker run uses a present digest without a registry call. scripts/lab-images.test.mjs (19 tests) holds the helper to both files as they are and the plan to those rules. Verified in a privileged systemd ubuntu:26.04 container, from a bundle of the branch, on Docker 29.8.1's containerd image store: the first run pulled both capability images and the second was changed=0, with exactly the two pinned digests in the lab repositories. A test commit then moved IMAGES.alpine to 3.24.1. --check --diff reported the one pull and changed nothing. The real run pulled the new digest and kept the old one, which a stopped container used. Once that container was gone, the next run removed the old digest's lab references and left an unrelated tag on the same image in place, and the run after was changed=0. With the test host cut off from the network, a shell-echo and a terraform-validate job run through the agent's own runInDocker passed in 403 ms and 577 ms, while a pull of an image the host did not hold failed as unreachable.

  • Admin → Labs: remove a deactivated agent's registration. Owner request 2026-09-28: the VPS reinstalled on 2026-09-26 came back as vps-hostinger-01, and the old host's record, srv939861 (v0.1.0, last seen 2026-08-20), stayed on the Agents tab, Offline and Deactivated, with no way to take it off: #740 built register and activate/deactivate only, and the owner holds no data-plane role. DELETE /api/cms/labs/agents/{agentId} joins the PATCH on that template's one httpRouteByMethod registration in labs-http.js, at the same editor role (lib/labs/agent-registry.js). It reads no body; answers 404 for an unregistered id, so a second DELETE is a 404; refuses an active agent with 409 "Deactivate it first"; and refuses with 409, naming the jobs, an agent that still holds one in lab_jobs (by agentId, which the claim writes): running, or claimed inside CLAIM_LEASE_MS. A claim older than the 15-minute lease is not counted, because the claim path already hands it to the next agent and a deactivated agent could never finish it; counting it would let one stranded job block the removal forever. Then it deletes the lab_agents document and writes one admin_audit_logs row, lab_agent_removed, with the removed document's agentId, oid, lastSeenAt and version. Without a document the agent guard refuses the agent as no-agent-record, the state before its first registration, and Register agent adds it back. On a deactivated card only, Remove opens the repository's ConfirmModal naming the agent and saying its registration is deleted and it can be registered again later; confirming sends the DELETE, toasts and refreshes. The Labs snapshot's refresh no longer drops a request that arrives while a poll is in flight: it reads once more after it, so the card goes when the toast appears rather than coming back from a read sent before the delete. The route inventory now pins both verbs on cms/labs/agents/{agentId} to requireRole at editor and never requireAgent; .azure/api-surface.json carries the route; the Labs host runbook gains step 8 of "Reinstalling the host", retiring the old agent record.

  • "Validate on the lab" opens to the public, locked to the site's pane by origin and Cloudflare Turnstile (ADR 0032 decision 6 revised). Owner decision 2026-09-28: "The lab should only be accessible through 'panes' from my site, lock to that." ADR 0032 gains a dated amendment replacing decision 6's Gate with a lock, and every bound #672 built stays exactly as it was: terraform-validate only, 64 KB, 2 an hour per client, 50 a day, refused while more than 20 are queued or no agent is heartbeating, jobs public: true with a one-day TTL. The lock, in functions/src/lib/labs/public-lock.js, runs before any store read or counter: a POST /api/public/labs/submit whose Origin is not exactly https://hybridcloudworks.com or https://www.hybridcloudworks.com (the CORS allowlist's production origins, now exported from lib/auth/cors.js; both serve the site) is 403 ORIGIN_NOT_ALLOWED before its body is read, a missing Origin included; one without a turnstileToken is 403 TURNSTILE_REQUIRED; the token goes to Cloudflare's siteverify with the secret and the address client-identity.js trusts (new trustedClientIp: CF-Connecting-IP only on a request carrying the origin secret), and anything but success: true for one of the site's hostnames and the lab-validate action is 403 TURNSTILE_FAILED, which is how a replayed single-use token fails; siteverify unreachable, past 5 seconds, non-2xx, non-JSON or refusing the secret is 503 TURNSTILE_UNAVAILABLE. Before siteverify, an in-memory limiter per instance allows one client ten checks in ten minutes and answers the next 429 TURNSTILE_RATE_LIMITED, so junk tokens cannot drive a Cloudflare call per request (from the change's security review, which found nothing above Low). With no secret, or an unresolved Key Vault reference, the POST and the status read answer TURNSTILE_NOT_CONFIGURED, so the status read now says which of switch, Turnstile or agent is missing. The token, the address and the secret are never logged or written to the job. On the builder, the Turnstile widget (Managed mode, shown only when Cloudflare wants the visitor to act) loads only once the lab is open (frontend/src/lib/turnstile.js, landingZone/useLabTurnstile.js); each submission spends one token and asks for the next, and a build without VITE_TURNSTILE_SITE_KEY keeps the button disabled with its reason. The CSP grants https://challenges.cloudflare.com in script-src and frame-src and nowhere else, which csp.test.js holds exactly. Terraform adds labs_public_submission_enabled (default true, the kill switch when set false) as LABS_PUBLIC_SUBMISSION_ENABLED, and TURNSTILE_SECRET_KEY as a Key Vault reference to TURNSTILE-SECRET-KEY, catalogued on the API-keys page and listed in EXPECTED_UNRESOLVED until it is seeded. The widget is not a Terraform resource: the pinned provider's cloudflare_turnstile_widget keeps its secret as a read-only attribute in state and needs an account-scoped token, so the owner creates it in the Cloudflare dashboard, and a test fails if the resource appears. Both plan differences are declared in scripts/assert-expected-plan.mjs. Required inputs, Variables and secrets, architecture §5.3, the API surface and the labs-host runbook ("Opening 'Validate on the lab' to the public") carry the new values and the owner's four steps. The privacy policy (frontend/public/privacy-policy.html, effective date now 28 September 2026) gains one line saying the builder runs Cloudflare Turnstile while Validate on the lab is available, linking Cloudflare's Turnstile Privacy Addendum.

  • Import docs/content drafts into the CMS review queue, never published (#749). Nothing imported a repository draft into the CMS, so reviewing the three lab articles (#737/#744) on the site meant pasting each one by hand. On Admin → Content Queue, Import drafts from the repository lists the blog-*.md files on main (GET /api/cms/content/import-repo/candidates, the public GitHub contents API, the two contract docs excluded) with what each already is on the site; ticking some and importing (POST /api/cms/content/import-repo, { paths }, editor) lands each one In Review with a toast, and the queue switches to that filter. Paths must match ^docs/content/blog-[a-z0-9-]+\.md$ or the whole request is a 400 before any fetch; files are read only from raw.githubusercontent.com/HybridCloudWorks/HCW-HybridCloudWorks/main/ with redirect: 'error', one 8 s deadline over headers and body, and a 256 KB cap. The front matter is read by a port of tooling/workflow.py's frontmatter() over the seven keys blog-template.md defines, and the body goes in untouched, so the landing-zone embeds survive. New drafts are written through createContentDocument (the createContentItem path) with contentStatus: 'in_review', Live: false, source: 'repo' and repoPath/repoCommitSha/repoContentSha256, under an id derived from the path and create-only, so a double import cannot duplicate. Re-importing refreshes the same document only while it is in review (an unchanged file writes nothing), under the read's ETag; an approved, edited, rejected or live article is refused and not touched, as is a new draft whose title a published article already carries. One admin_audit_logs row per path. slug is deliberately not stored: the slug-holders probe is not status-filtered, and a first publish assigns slugify(title) anyway. The three lab drafts are read from disk by the tests and must parse, and CI's functions row now watches docs/content/blog-*.md.

  • Register a lab agent from the admin UI; Register-LabAgent.ps1 finishes the go-live (#740). The API admits a lab agent only when lab_agents/{agentId} binds it to its service principal, and nothing could write that document: no route, a read-only Labs page, and a Cosmos firewall that admits only the Function App. So a registered agent authenticated and was refused with Agent access required. Two routes in functions/src/lib/labs/agent-registry.js, registered in labs-http.js, both editor or above (the level of the other Labs writes; gate 1, the LabAgent app role, stays an Entra grant only a tenant administrator can make): POST /api/cms/labs/agents takes exactly { agentId, oid, jobTypes? }, any other key a 400, with agentId the certificate CN shape, oid a GUID stored lower-cased because the agent guard compares it exactly with the token's lower-case claim, and jobTypes a non-empty subset of LAB_JOB_TYPES (default all five). It creates the document the guard and the claim path read, { id, agentId, oid, active: true, capabilities, registeredAt, updatedAt, updatedBy }, or patches only the registry fields of one that exists, so a heartbeat is never rewound; the same values again write nothing, and a deactivated agent stays deactivated. PATCH /api/cms/labs/agents/{agentId} takes { active } and takes effect on the agent's next call, since the guard does not cache the registry. Every write leaves an admin_audit_logs row (lab_agent_registered, _updated with the previous object id, _activated, _deactivated), and nothing secret is accepted or stored. getLabsSnapshot now carries each agent's active, oid and registeredAt. On Admin → Labs → Agents, Register agent takes the agent id and the object id with every job type ticked, and says in one line that Register-LabAgent.ps1 prints both; each card shows its bound object id and a Deactivated badge, and has Deactivate (which asks first) or Activate. Tests prove a registered document passes require-agent for a token carrying that object id, reaches the claim path with its job types, and is refused once deactivated. Register-LabAgent.ps1 no longer prints JSON and exits 2: no owner script can get the admin's delegated token the route needs, so step 4 prints https://hybridcloudworks.com/admin/labs?tab=agents with the two values to paste and waits for Enter, then continues to the vault and bootstrap.sh and restarts the agent so the verdict reads a clean journal. It skips the wait when the agent is already heartbeating on a service principal it did not just create, so a second run still changes nothing without asking. Docs: the Labs host runbook ("The lab agent's go-live"), lab-host/README.md ("The agent identity") and Required inputs §4.7.

  • Landing Zone Builder: "Validate on the lab", and public lab submission built inside ADR 0032 decision 6's bounds, closed (part of #672). Three anonymous routes in functions/src/lib/labs/public-submit.js, registered in labs-public-http.js: POST /api/public/labs/submit queues one terraform-validate job; GET on the same route says whether a POST would be taken ({ configured, open, code, reason, bounds }, cached a minute); GET /api/public/labs/job?jobId= returns one public job's status and output. All three are closed: unless the Function App setting LABS_PUBLIC_SUBMISSION_ENABLED is exactly "true", which nothing sets, they answer PUBLIC_SUBMISSION_CLOSED before reading the body, the caller or the store. ADR 0032 still holds anonymous submission Gated, and opening it is the owner's revision of decision 6, then that one setting. Open, the bounds are decision 6's and no wider: terraform-validate only (400 for any other type), a 64 KB payload (413 above 65,536 bytes), 2 an hour per Cloudflare-verified client through enforceSubmissionQuota (429), 50 a day globally through a compare-and-increment counter in tool_service_cache (503 until the next UTC day), refused while more than 20 jobs are queued (503 LAB_QUEUE_FULL), and jobs written public: true with a one-day ttl. It also fails closed while no agent registered for terraform-validate is heartbeating (503 LAB_AGENT_OFFLINE, nothing counted, nothing queued) and when the lab cannot be read (503 LAB_STATUS_UNAVAILABLE). The job read serves public: true documents only, with one identical 404 for a missing, admin or expired job, and never returns the payload, the agent or the requester. The queue ceiling is a read before the write, so simultaneous submissions can overshoot it by the ones in flight together; both counters are exact. The explain route's daily counter moved to lib/daily-cap.js so both routes share it, behaviour unchanged. On /tools/landing-zone, "Validate on the lab" sits under the generated files. It asks the status route on mount and stays disabled, with one line saying why and no spinner, until the lab is open. It sends the files unchanged, as a gzipped tar the agent's own parser unpacks into exactly the download, because the capability rewrites registry sources to the vendored copies inside the job (ADR 0032 decision 5). Before sending, it predicts that rewrite with the runner image's own rule (lib/landingZone/labImage.js, held to lab-image/Dockerfile and versions.env by a test) and refuses a build calling a module the image does not vendor. Today that is avm-res-network-virtualnetwork 0.22.2, which every spoke calls, so only a build without the identity, corp and online spokes can be validated on the lab until lab-image vendors it. The frontend CI row now also runs on changes to lab-image/Dockerfile, lab-image/versions.env and vps-agent/lib/docker-runner.js, which those tests read.

  • scripts/lab/Register-LabAgent.ps1: the lab agent's go-live is one owner command (#739). It was several owner steps across three documents, one of them pointing at a runbook row that no longer existed. The script, run with az signed in to the tenant: checks the tenant and prints the az login line when it is wrong; reads /etc/hcw/labs-agent.crt over ssh hcw-lab into a temporary file, with the host checking it against the private key and reporting only whether they match; creates or finds the single-tenant app registration and service principal sp-labs-agent-lab-hybrid-prod-cus-01; appends the certificate with az ad app credential reset --cert --append, ending one second before the certificate does (without --end-date az ends it a year from now, a year before the 730-day certificate); assigns the LabAgent app role on the HCWSite API, the one grant the agent guard checks; writes the four vault_labs_agent_* keys into /etc/hcw/ansible/vault.yml on the host; runs bootstrap.sh; and reads systemctl is-active hcw-labs-agent and the journal to say whether the agent is heartbeating, or which gate refuses it. The vault merge runs as root on the host: decrypt to a root-only temporary directory, replace only those four keys, refuse unless every other key is unchanged, re-encrypt with the existing password file, decrypt again and compare, rename into place, shred. It creates vault.yml when there is none and does not rewrite it when nothing changed; it prints key names, never a value from the file. Re-running changes nothing, -WhatIf changes nothing, and -NextCertificate registers a rotated certificate before the swap. LABS_AGENT_API_BASE is now https://api-azure.hybridcloudworks.com/api in lab-host/README.md and vps-agent/.env.example: the origin lock refuses the lab host at the azurewebsites.net host, and the script probes the Cloudflare path from the host first, naming a Bot Fight Mode challenge if it meets one. The API's second gate, the lab_agents/vps-hostinger-01 registry document, has no write path (no route, no admin page, a firewall that admits only the Function App), so the script prints the document the API needs and says so. Pester tests cover every path that needs no tenant or host, and run the vault merge under bash in CI; scripts/lab-job-types.test.mjs keeps the document's job types in step with the server and agent allowlists. Docs: lab-host/README.md ("The vault", "The agent identity", "Rotating the agent certificate"), Required inputs §4.7 (which also now records portainer_enabled and vault_enabled as true since #729 and the host as provisioned) and the Labs host runbook, "The lab agent's go-live".

  • Lab article series: three how-to drafts for the owner's review (part of #677). docs/content/blog-lab-01-landing-zone.md builds management groups, policy, management and a hub in the Landing Zone Builder and reads what each is for; blog-lab-02-one-container.md pulls ghcr.io/hybridcloudworks/hcw-lab, runs terraform init under --network none and explains the provider mirror, the unpacked layout and the module rewrite; blog-lab-03-agent-explains.md opens the zip in the lab-image/sandbox-template Docker sandbox and reads why AGENTS.md forbids apply. Each carries one landing-zone embed of the same build, lz=mg,policy,mgmt,hub&corp=0&online=0, checked against decodeLz to decode to exactly those four components and to be its own canonical encoding, and each ends with "Explain this component" and a link to /education/labs, with the Coder workspace described as coming. Every output quoted was measured on 2026-09-27 with Docker 29.8.0 against the builder's emission for that build: the public image pulled with no login, terraform version (1.16.4), plain init failing at the module registry offline, hcw-terraform-validate passing offline and under the job runner's full sandbox flags (108 KB .terraform against 31 MB online), and the sandbox template built and running init, fmt -check and validate. The sbx commands are the recipe README's and were not run. Found while measuring: a build with any landing zone does not init offline in the lab image, because the spokes call avm-res-network-virtualnetwork 0.22.2 and the image vendors only 0.15.0; part 2 says so. Listed in the docs nav marked "(draft)"; publishing and the newsletter stay the owner's.

  • ADR 0032 (the learner labs platform) is Accepted, 2026-09-27. Owner approval: "all have been approved to move forward". Its decision 6 is unchanged, so anonymous public lab submission stays Gated until the owner revises that section; the status line and the decisions index say so.

  • ElevenLabs: the last sample replays free, and a new render asks first (owner request 2026-09-27). The owner spent two live checks (257 credits each, 9,486 left on the free plan) only to test playback, because the Audio tab showed a player only for a check run in the same session. Every stored sample is now recorded in admin_config/podcast_last_sample: the media URL with its ?v= version, both voice ids (with names when the voice listing is cached; nothing is listed to find them), the characters billed and the time. GET cms/podcast/elevenlabs returns it as lastSample (or null, with lastSampleError when the read fails), after a HEAD confirms the blob is still there. A sample stored before the record existed, such as the 2026-09-27 render, is still returned (recorded: false), from the blob, the newest podcast:sample usage row, and the saved voices when they were saved before it. The card's Last sample section plays it on every load with its date and voices and says "Replaying costs nothing"; a replay never reaches ElevenLabs. Run live check is now Render a new sample (about 257 credits) and opens the admin ConfirmModal first. The prompt names the credits left and the free replay, and says when the last sample already used the saved voices. Cancel sends nothing. The Podcast voices picker adds "Previews are ElevenLabs's samples and cost no credits". useSetting also returns saved, the stored value, which unsaved edits do not change. ADR 0029 §2a carries a dated note.

  • Podcast voices: the owner picks two ElevenLabs voices the plan allows, by ear, on the Audio tab (#725). The first free-plan live check was refused with 402 paid_plan_required ("Free users cannot use library voices via the API"). The code's Sarah and Aria are library voices to an account created after March 2026. The ElevenLabs card gains a Podcast voices section: a select per host (Maya, Elena) and a list of every voice the key may use, each with its gender, accent and age labels and a play button. A voice the plan does not allow is shown disabled with the reason, never hidden. Two new editor routes back it. GET cms/podcast/elevenlabs/voices lists GET /v2/voices three times with the documented voice_type filter (default, non-community, community), paginated and cached for five minutes, and marks a library voice unusable on the free plan. GET cms/podcast/elevenlabs/voices/{voiceId}/preview proxies one voice's preview. It takes an id, never a URL; it fetches only https://storage.googleapis.com/eleven-public-prod/… from the key's own listing, without the key and without redirects; it stops at 2 MiB and passes on only MP3 bytes. The page plays the preview with Web Audio, so the CSP is not widened. The choice is stored as admin_config/podcast_voices through the Platform settings store (podcast-voices), validated as two different 20-character ElevenLabs ids, and every podcast render reads it first: episodes and the live check. The podcast no longer reads LISTEN_AND_LEARN_VOICE_*, and Listen & Learn is unchanged. The code has no default voice, because ElevenLabs's docs name no id a free account made after March 2026 can use through the API. So a render with nothing chosen refuses before sending, with "Choose the podcast voices first", and the live check waits for a saved pair. A 402 paid_plan_required now says "Choose voices your plan allows" and names the picker. The key needs a third permission, Voices → Read (voices_read), which the secret catalogue and docs/standards/required-inputs.md now list. ADR 0029 §2a carries a dated amendment.

  • Portainer and HashiCorp Vault are on for the lab host (owner decision 2026-09-26). portainer_enabled and vault_enabled are true in lab-host/ansible/group_vars/all.yml, so the next bootstrap.sh run starts Portainer Business Edition 2.45.1 on 127.0.0.1:9443 and Vault 2.1.1 on 127.0.0.1:8200, both loopback-only. Vault comes up uninitialised and sealed; the first administrator, the licence key and Vault's unseal keys are the owner's steps in docs/runbooks/labs-host.md.

  • The lab host refuses a host it was not prepared for; Portainer and HashiCorp Vault join it, loopback-only (owner decision 2026-09-26; #656). The owner's first bootstrap.sh run on the VPS met a host that had never been reinstalled (two self-hosted runners, Portainer, nginx on :80, k3s, Vault, an old agent install). Before it failed at the agent's checkout it had upgraded and restarted Docker, killing a running Dependabot job, enabled ufw and rewritten sshd's settings. The owner decided to reinstall the VPS clean and to keep Portainer and Vault. bootstrap.sh now examines a host it has never accepted, before it changes anything: any Docker container, an installed actions.runner.* unit, Kubernetes units or directories, a foreign checkout at /opt/hcw-src or /opt/hcw-labs-agent, anything else under /opt, and any TCP listener but sshd's and systemd-resolved's. On a finding it refuses with the list, exit code 3, and the two ways on (reinstall, or HCW_ADOPT_NONEMPTY_HOST=1). The first run that passes writes /etc/hcw/bootstrap-host-accepted, every completed run records its commit there, and later runs skip the check. A new portainer role runs Portainer Business Edition 2.45.1 (LTS, by index digest) under the 3 Nodes Free licence. It has a named volume and the Docker socket, and HTTPS on 127.0.0.1:9443 only, which the role refuses to move; there is no Caddy route, and the owner reaches it through an SSH tunnel. A new vault role runs HashiCorp Vault 2.1.1 host-native under systemd, with raft storage, 127.0.0.1:8200 and :8201 only, and TLS from a host-generated certificate. Its download is checked against HashiCorp's GPG signature on SHA256SUMS, with the key pinned by checksum. It never initialises or unseals Vault; those are owner steps whose keys go to the password manager only. This Vault holds lab-host secrets only; production secrets stay in kv-site-prod-cus-01. Both roles are off by default. scripts/version-floors.json gains a vault kind from endoflife.date's hashicorp-vault product, and records Portainer as unsourced (endoflife.date has no product for it). ADR 0032 carries a dated amendment. The runbook gains the host check, the reinstall procedure, the Portainer tunnel (including the setup token 2.45.1 prints) and Vault's initialise and unseal steps. Auto-unseal through Azure Key Vault and the Arc identity is the follow-up, #726.

  • ElevenLabs on the free plan: a credit pre-flight, a non-commercial publish guard, and the account on the Audio tab (#432). Owner direction 2026-09-26: test the podcast voice on ElevenLabs's free plan, 10,000 credits a month at one credit per character on Eleven v3, where one episode is nearly the month. speech/elevenlabs-account.js reads GET /v1/user/subscription (cached in memory for 15 s per key). Before the first dialogue request, synthesizeWithElevenLabs refuses a job the credits left cannot cover in full, sends nothing, and says "ElevenLabs has N credits left of M, this episode needs K; the allowance resets on " with the quota_exceeded code. An unreadable account (a key without user_read) fails closed. A 402 paid_plan_required (a paid-only voice) is no longer reported as out of credit. Owner decision the same day, "Test only, block publish": the podcast records the plan each render ran on (speechTier, speechFreePlan), and approving a transcript whose audio ElevenLabs rendered on the free plan answers 409, naming the non-commercial licence and the upgrade, and writes nothing. The retry route refuses the same way, and the publish job records a free_plan_licence skip. Paid-plan renders publish as before. New editor routes GET cms/podcast/elevenlabs (plan, credits used of limit, reset date, the last render's billed characters, or "Not configured") and POST cms/podcast/elevenlabs/sample (a fixed two-turn, 257-character live check through the real provider, usage source podcast:sample) back a Podcast voice card on the Audio tab. ADR 0029 §2a is amended with the date and the rule, and docs/standards/required-inputs.md gives the key's two permissions, its credit limit and where to seed it. The voices are unchanged, but their status is recorded: Sarah is a Default voice that expires 2026-12-31 and exists only on accounts created before March 2026, and Aria is a Legacy voice the API reroutes to Zoe.

  • PostgreSQL has a version floor, and the weekly job proposes general releases only (#715). Owner instruction 2026-09-26: the newest general release of PostgreSQL. scripts/version-floors.json gains a postgresql kind: line 18, newest 18.6, floor 18.4 (N-2 minors; PostgreSQL numbers releases MAJOR.MINOR), checked 2026-09-26. The pin reader checks coder_postgres_image_tag in lab-host/ansible/group_vars/all.yml against it. It accepts exactly MAJOR.MINOR and reports anything else, such as a moving 18 or a 19beta4, as a finding rather than a pass. update-version-floors reads endoflife.date's postgresql product rather than postgresql.org's versions.json. That keeps one source format and keeps true the workflow's "endoflife.date's v1 API for each kind"; both said 18.6 that day. It adopts a new major at its .2, and skips any major whose latest release is not MAJOR.MINOR or is not yet released, so a beta or release candidate is never proposed. majorMinorFloor joins the floor arithmetic. Nine tests cover the arithmetic, the reader (quoted and unquoted tags, 18, 19beta4, 19rc1, a missing pin), the rule, the file's shape and the updater (minor move, major adoption at .2, five pre-release shapes, a backwards source). Setting the pin back to 16.15, then to 19beta4, turned the gate red with one line naming the file, the pin and the floor. A live --dry-run printed No floor moved. Coder v2.37.3 was checked the same day and is the newest release (ghcr.io/coder/coder:latest resolves to its digest), so it is unchanged.

  • The lab agent and the Coder template pull the Python 3.14 / Debian 13 lab image. vps-agent/lib/capabilities.js pins hcw-lab-runner and lab-host/coder/templates/hcw-lab/main.tf pins hcw-lab to the digests main's publish run 36221001115 pushed for cd9e1af4 (#716), both checked with docker buildx imagetools inspect. .vscode/mcp.json is ignored: it holds per-developer MCP servers and local paths.

  • Node.js on the newest supported line, and version floors enforced in CI (#715, #714). Owner direction 2026-09-25: the latest current release. functions/ moves from Node.js 22 to 24, the newest line Azure Functions Flex Consumption supports (22 and 24 are GA; 26 is not offered): the engines range (^24.19.0), runtime_version in infra/functionapp.tf (an in-place update: the attribute is not ForceNew in azurerm 5.6.0), the functions (azure) CI row, the functions half of coverage.yml, deploy-functions.yml, and update-learn-catalogue.yml, which imports functions/src. Everything else moves from 22 to 26: the frontend/, frontend/scripts/, scripts/ and lab-host/coder/ engines ranges (>=26.8.0), frontend/.nvmrc, and every other node-version: in .github/workflows/. The CI matrix now gives each row its own line (matrix.node-version, defaulting to 26) instead of one pin for all five. scripts/version-floors.json records the newest release and the floor for Python (3.14.7, floor 3.14.5; #714 asked for a separate python-floor.json, folded in here), Node.js (26.10.0, floor 26.8.0; the functions ceiling 24.21.0, floor 24.19.0), Ubuntu (26.04 LTS), Debian (13) and Terraform (1.16.4, floor 1.16.2), each with its source and date. scripts/version-floors.test.mjs reads every node-version:, python-version: and terraform_version: in the workflows, engines.node in every tracked package.json, every .nvmrc, .node-version and .python-version, the python, node, debian and ubuntu FROM lines of every tracked Dockerfile, runtime_version, the lab host's labs_agent_node_version, Ubuntu target and PYTHON_VERSION, lab-image/versions.env's TERRAFORM_VERSION and every required_version, and fails any below its floor naming the file, the line and the floor. update-version-floors.yml re-reads endoflife.date and the Flex Consumption table on Microsoft Learn every Wednesday and opens one pull request when a floor moves; that pull request's CI then lists the pins left behind. The scripts (operations) CI row now also runs on changes to the files the floors test reads, and docs/standards/iac-repository-standard.md names the rule. vps-agent/package.json moves to >=26.8.0 here, because the Ubuntu 26.04 change (#717) moved the lab host to Node.js 26.10.0 and left the package range at >=22. On top of #717 and the Trixie lab image (#716), the floors test finds no pin below its floor.

  • Lab host targets Ubuntu 26.04 LTS, with release-derived apt suites, resolute package pins and every lab host pin at its newest release (#715). The VPS was reinstalled with Ubuntu 26.04 LTS (owner decision 2026-09-26). lab-host/bootstrap.sh and site.yml accept 26.04 and keep 24.04 as a stated fallback, refusing anything else; lab_host_ubuntu_releases in group_vars/all.yml is the list. Docker's version strings name the release, so the pins are now docker_release_pins, one entry per codename: 29.8.1, containerd 2.3.6, buildx 0.37.1 and compose 5.5.1 for both resolute and noble. The arc role derives packages.microsoft.com/ubuntu/<version>/prod from the running release, and because Microsoft signs the 26.04 repository with microsoft-2025.asc (AA86F75E...F748182B) rather than microsoft.asc, the key URL and its SHA256 are per-release maps, each checked with gpgv against its release's InRelease; azcmagent stays at 1.68.03532.1399, the newest in both repositories. Microsoft Learn lists Ubuntu 26.04 on x86-64 as supported for Arc-enabled servers and for the Azure Monitor Agent (read 2026-09-26), so there is no support gap. Ansible's control side now runs on CPython 3.14.7 installed by uv 0.12.19 (release asset SHA256-verified and attested, no curl | sh), because 26.04's /usr/bin/python3 is 3.14.4, below the owner's two-patch floor; modules still run on /usr/bin/python3, the one recorded exception, since python3-apt and python3-docker are built for it. ansible-core stays at 2.21.4, the newest GA release. labs_agent moves to Node.js 26.10.0 from NodeSource node_26.x, and its npm ci stamp now includes the Node pin so a runtime bump reinstalls dependencies. The Coder template's code-server moves from 4.106.3 to 4.139.1. infra-lab/README.md step 1 now says to paste the token-reading line before copying the token, because copying the line overwrites the token on the clipboard. Verified in privileged systemd ubuntu:26.04 and ubuntu:24.04 containers: two bootstrap.sh runs each, the second changed=0; the fail-closed state (apex 503, agent stopped, Coder down); azcmagent installed and failing closed without the vault; and vps-agent's 81 tests passing on Node 26.

  • CI runs Python 3.14, the newest stable release. Owner instruction 2026-09-26: always use the newest Python. The four actions/setup-python pins (the ansible-lint and harness jobs in ci.yml, complexity-delta.yml and docs-pages.yml) move from 3.12 to 3.14. The docs build was run locally on 3.14.7 before the change.

  • scripts/lab/Connect-Lab.ps1: one-command SSH and VS Code setup for the lab host, and the corrected Hostinger read in infra-lab. The script (PowerShell 7, -WhatIf) sets up any desktop to reach the host as hcw-lab: a per-machine ed25519 key at $HOME\.ssh\hcw-lab_ed25519, created only if missing; exactly one Host hcw-lab block in $HOME\.ssh\config, inserted ahead of any Host * or replaced in place, every other line kept, a timestamped backup written first; the Windows ssh-agent service started when it is present and stopped, and the key loaded; then the public key and the step that authorizes it. The host name comes from the LAB_SSH_HOST repository variable when gh can read it, otherwise lab.hybridcloudworks.com; -User root covers the window before the first bootstrap.sh, and -Code and -Connect open VS Code Remote-SSH on /opt/hcw-src or a shell. The printed line for adding a key from a machine that already connects writes root's authorized_keys as well as the login user's, because the hardening role rebuilds hcwadmin's from root's on every run (exclusive: true) and a key only there would be gone after the next one. The private key is never read, printed or sent, and a second run changes nothing and says so. Connect-Lab.Tests.ps1 covers the config rewrite in 29 Pester cases, run by a new step in repository-policy.yml that also fails on zero tests. docs/runbooks/labs-host.md opens with "Connect from a desktop": the Remote - SSH extension, the run with and without a checkout (saved to Downloads and run, never piped to iex), the optional LAB_SSH_HOST variable, the host-key check, and the Web Console as the browser fallback. infra-lab/README.md step 1 printed blanks for every field but the template, because Invoke-RestMethod passes the bare JSON array Hostinger returns down the pipeline as one object (reproduced on PowerShell 7.6 and 5.1); it now reads the token from the clipboard, unwraps the list, prints id, plan, data_center_id, template_id, template_name and state, prints status and body on failure, and blanks the clipboard. plan follows the provider's import at v0.1.23, which keeps the VM's plan field (such as KVM 4) unless the billing subscription lookup returns an item_id. Steps 6 and 7 now use the hcw-lab_ed25519 key and the hcw-lab alias.

  • Azure Arc onboarding for the lab host: onboarding identity, data collection rule, audit policy, Ansible arc role (#663). Phase 3 of #656, the engineering half. infra/lab-hybrid.tf adds the data collection rule dcr-lab-hybrid-prod-cus (heartbeat plus auth/authpriv syslog at Info and above, into the Management workspace, in the workspace's own region), the onboarding principal's only grant (Azure Connected Machine Onboarding on rg-lab-hybrid-prod-cus, planned once the owner sets arc_onboarding_principal_id, because the run identity has no Entra role and a Terraform-minted secret would sit in state) and the audit-only Linux security baseline assignment with IncludeArcMachines true (planned once lab_hybrid_policy_enabled is set, because the run identity cannot write a policy assignment until the owner grants it Resource Policy Contributor on the group). The Arc resource providers join azure_resource_providers; Defender for Servers stays off. The new lab-host/ansible/roles/arc runs after hardening, off by default (arc_enabled: false): azcmagent 1.68.03532.1399 from Microsoft's repository, signing key pinned by SHA256 and held, then azcmagent connect with the credential in a root-only temporary --config file deleted in the same run; skipped once Connected, fail-closed without the four vault_arc_* values otherwise. The Azure Monitor Agent and the rule's association are the owner's two az commands after onboarding, because both target a machine that does not exist before it. Labs host Arc onboarding is the owner procedure, from the principal to Connected, the heartbeat, the compliance state and the disconnect; Required-Inputs §4.1 and §4.7 carry the new names, and the §4.1 variable count, which had drifted to 62 against 63, now reads 65.
  • NVIDIA API Catalog as a purpose-scoped AI provider for content generation (#701). A fourth router provider, nvidia, on the OpenAI-compatible endpoint https://integrate.api.nvidia.com/v1 with NVIDIA_API_KEY as a Bearer token; available by key presence, and an unseeded Key Vault reference counts as no key. Its purpose → model table (CONTENTFORGE_NVIDIA_*_MODEL overridable) defaults to z-ai/glm-5.3 for drafts, deepseek-ai/deepseek-v4.1-flash for analysis and z-ai/glm-5.3-flash for short calls, read from build.nvidia.com on 2026-09-25. It is placed per feature (PROVIDER_PLACEMENT_DEFAULTS in ai-config.js): first for owner-triggered content (Forge drafting and grading, inspector, critique, voice calibration, captions, Listen & Learn and podcast scripts), after the others for the Telegram assistant, and locked off for the anonymous public explain features, image alt text and source grounding — configuration can demote or disable it, never place it in a locked feature. A sliding-window pacing guard keeps each instance at 36 requests a minute (NVIDIA_REQUESTS_PER_MINUTE); a paced call is not sent and fails over at once, as do NVIDIA 429/5xx/404/400 responses after the existing retries. Usage is recorded at zero cost. The AI Engine page lists and orders the provider and sets its placement per feature under "Where AI is used". NVIDIA-API-KEY gains a Key Vault reference in infra/functionapp.tf, a secret-catalogue entry and a Required-Inputs §4.6 row; the owner seeds it.
  • infra-lab: adopt the existing Hostinger VPS by import and add the lab DNS records (#661). A new root module, infra-lab/, for the HCP Terraform workspace hcw/hcw-lab (ADR 0032), pinned to hostinger/hostinger 0.1.23 and cloudflare/cloudflare ~> 5.24. The owner already has the server, and in this provider creating a hostinger_vps is a purchase and destroying one cancels the subscription. So hostinger_vps.lab comes in through an import block whose id is a workspace variable, has prevent_destroy, and ignores changes to plan, data_center_id and password (ForceNew, so a change would be cancel and re-purchase) and to template_id (a change reinstalls the disk). Postconditions turn a mistyped identity variable into a plan error that names the right value. dns.tf writes lab.hybridcloudworks.com (A, the VPS's IPv4) and CNAMEs to it for *.lab, coder.lab and *.coder.lab, all DNS-only. coder.lab needs its own record because *.coder.lab makes it an empty non-terminal, which Cloudflare's wildcard does not answer for. There are no _acme-challenge delegations while there is no lab zone (the ADR 0032 interim). The provider cannot run a post-install script on an existing server, so the first Ansible run is bootstrap.sh over SSH. The owner's steps and the expected first plan, 1 to import, 4 to add, 0 to change, 0 to destroy, are in infra-lab/README.md. Offline contract tests with mocked providers (infra-lab/tests/adopt.tftest.hcl) and the fmt, validate, tflint, test and Trivy steps run in the existing iac-validate.yml contexts, and Dependabot watches the new lock file.
  • The Phase 2 lab image digests are pinned, and the Coder template no longer rewrites /etc/passwd (#698). IMAGES.hcwLabRunner in vps-agent/lib/capabilities.js and the hcw-lab pin in lab-host/coder/templates/hcw-lab/main.tf now point at the digests main published for 02dd9595 (#695), so the three runner-image capabilities find the hcw-terraform-validate, hcw-helm-template and hcw-kubeconform scripts instead of failing with executable file not found. That hcw-lab image gives uid 65534 /bin/bash and /tmp/home, so the template's temporary local.passwd and upload block (#693) are gone.

  • Deploy Functions smoke check polls until the temporary access rule has propagated (#700). The post-deploy smoke test waited a fixed 20 s after adding its ci-smoke-<run id> access rule and made one request, so on run 36175776422 it got 403 "Web App - Unavailable" before the rule reached every front end, and a healthy deploy went red. It now polls /api/health every 10 s for up to 150 s, logs each attempt's status, passes on the first 200 and fails with the last body in the step summary if none arrives. The rule add, the always() removal and the origin-lock check are unchanged. The SCM window's sleep 20 is left as it is: it is waited on by the deploy action, not by a request of this workflow's own.

  • The Coder status token is an expected unresolved secret until Coder can issue it (#682). CODER-STATUS-TOKEN is created by Coder, so it cannot exist before Coder runs on the lab host (#661); the Terraform comment and Required-Inputs §4.7 had asked for it to be seeded before the apply. It is now listed in EXPECTED_UNRESOLVED in scripts/check-unresolved-secrets.mjs, reported every run but not as a failure, and the labs page keeps reading "not yet provisioned". CODER-URL is seeded now with its known value.

  • hcw-lab image Phase 2: transitive AVM vendoring, source rewrite, job labels, helm-template and kubeconform capabilities (#675). Phase 2 of #658. All three vendored AVM pattern modules now terraform init under --network none: lab-image/vendor-avm.sh runs terraform get on each at build time, vendors every registry child once at /opt/avm/<name>@<version> (sixteen name@version pairs across fourteen modules), rewrites each module block inside the vendored copies to the child's relative path with its version commented out, pins every child by tree hash in versions.env (AVM_CHILD_MODULES; the build fails on a mismatch, an unpinned child or an unused pin) and proves with a second terraform get that nothing reaches the registry. The provider mirror is now unpacked, so init symlinks providers instead of extracting 225 MB of azurerm onto the job's 64 MB tmpfs, and TMPDIR=/tmp/run, because a provider opens its socket in TMPDIR and every one died silently on the read-only root; both were measured under the agent's exact sandbox flags, where the Phase 1 image could not validate any azurerm configuration at all. The terraform-validate capability now runs lab-image/bin/hcw-terraform-validate, which copies the payload to the tmpfs and rewrites source = "Azure/<module>/azurerm" blocks whose version constraint a vendored copy satisfies (Terraform's own operators, highest match wins) to that copy, leaving everything else untouched so init fails loudly; the learner's files are never changed (ADR 0032, decision 5). Jobs may carry payloadEncoding: "tar" (base64 of a tar, gzip accepted) for multi-file inputs: the agent unpacks it with its own parser, which accepts regular files and directories only and refuses absolute paths, .., symlinks, hard links and every extension header, under the unchanged 64 KB cap; the server validates the encoding per type and the claim response carries it. Every job container carries --label hcw.lab-job=<jobId>, asserted in docker-runner.test.js, and a job with no plain identifier does not start. Two capabilities join in all three places (capabilities.js, LAB_JOB_TYPES, FALLBACK_JOB_TYPES, now held together by scripts/lab-job-types.test.mjs): helm-template (helm template on a tar of one chart) and kubeconform (-strict against one pinned release of yannh/kubernetes-json-schema bundled at /opt/kubeconform/schemas, no API server). The tmpfs is mounted uid=65534,gid=65534,mode=0700, because Docker mounts it root-owned and the bare Phase 1 flag gave Permission denied on the first write. lab-image/sandbox-check.mjs runs the five jobs through the agent's own docker-runner.js under the full sandbox against the built image, in the publish workflow after the smoke tests. The full target gives uid 65534 /bin/bash and /tmp/home in /etc/passwd, because Coder runs the startup script, code-server and the terminal through the passwd shell (#693). The pull and run commands for the toolchain, PowerShell and bash, are in lab-image/README.md and docs/architecture/labs-host.md for /education/labs (#681) to link. IMAGES.hcwLabRunner pins the Phase 1 publish; the digest that carries this phase's image is bumped in a follow-up once main republishes.

  • Docker agentic sandbox recipe and the "Run an agent against your landing zone" section on /education/labs (#676). Phase 3 of #658. lab-image/sandbox-template/Dockerfile extends Docker's Claude Code sandbox template (docker/sandbox-templates:claude-code, pinned by image index digest) with terraform and the Azure CLI at the versions lab-image/versions.env pins — the build context is lab-image/ so the file is read from its one home, the Azure CLI package string's ~bookworm suffix is swapped for the base image's Ubuntu codename with the version number unchanged, and every download is checked against the pinned SHA256 — and installs AGENTS.md as the agent user's ~/.claude/CLAUDE.md: the folder is a landing zone generated for learning, run terraform init -backend=false, fmt -check and validate, explain the files, never plan, apply or reach a tenant. README.md beside it carries every command, PowerShell then bash: build, docker image save and sbx template load, the sbx policy allow network --sandbox hcw-lz line the deny-by- default sandbox needs for the Terraform registry and GitHub, and sbx run --name hcw-lz --template hcw-lz-sandbox:v1 claude .. On the page, the #676 slot is filled by SandboxSection: three numbered steps (build in the Landing Zone Builder and download the zip, unzip it, create the sandbox from that folder), the sbx run line under each shell through a CommandLine component shared with the lab cards, the first prompt as one quoted sentence, the sentence that local sandboxes are free and cloud ones bill the learner's own Docker subscription and expire after an hour, and a link to the recipe directory. Every sbx fact was read from the Docker documentation on 2026-09-25 (docker sandbox is removed from Docker 29; the CLI is sbx, custom templates are --template <image> on sbx create/sbx run, cloud sandboxes need a Docker Agentic Platform subscription, cannot mount a host workspace and expire after one hour by default) and the page cites the URLs in its source. The image built locally (3.38 GB) and terraform init -backend=false, fmt -check and validate passed inside it against a full builder emission; the sbx commands themselves are the documentation's shapes, not exercised, since the CLI needs a hypervisor the build machine did not have enabled. validate-repository-structure.ps1 allowlists the recipe's README.md and AGENTS.md.
  • Newsletter section "Lab this week" with a daily labs rollup timer (#665). Phase 5 of #656. A new flag-gated timer labsWeeklyRollup (functions/src/functions/labs-jobs.js, FEATURE_FLAG_LABS_WEEKLY_ROLLUP, daily 23:55 UTC, catalogued in infra/functionapp.tf and allowed by the enabled_timers validation) writes one labs:day:<YYYY-MM-DD> document per UTC day to tool_service_cache with a sixty-day TTL: { arcConnected: boolean|null, jobsByType: { [type]: { succeeded, failed, timeout } }, coderRunningMax: number|null }, computed by functions/src/lib/labs/rollup.js from three point reads (the labs:estate and labs:coder-status minute-cache documents and any earlier document for the same day) and ONE grouped count of lab_jobs bounded on the indexed createdAt and the three terminal statuses — no per-job read, no full scan, and never a call to Azure or Coder. Null means not observed, not down: arcConnected is a boolean only when an estate document stamped that day described a configured estate. A rerun for the same day merges — connected once stays connected, the Coder peak keeps the larger sample, job counts are recomputed. The newsletter section lab-this-week (functions/src/lib/newsletter/sections.js, registered last and on by default) point-reads the seven day documents before the build day and renders Arc uptime as days connected out of days observed, lab jobs by type with success counts, and the peak Coder workspace count, every item linking to https://hybridcloudworks.com/education/labs; it renders nothing, and the issue omits it, when no day document exists or none observed the estate or a job. Tests beside each module cover the rollup computation from fixture cache documents and grouped rows, the TTL, the timer's flag gate and disabled skip, the section with data, and the section omitted from a built issue without it; route-inventory.test.js and timer-schedules-utc.test.js now count twenty-two timers.

  • Landing Zone Builder, Phase 4: "Explain this component" and the landing-zone article embed (#670). Phase 4 of #657, frontend half, on the #669 backend. frontend/src/pages/tools/landingZone/LzExplainButton.jsx sits in the teaches panel beside the focused component and, on a click and never otherwise, sends { kind: "landing-zone", componentId, selected, options, teaches } — the five keys the server's validator allows and no other, built from the normalised build and the catalogue's own text — to POST public/cloud-tools/explain through requestLandingZoneExplanation in lib/publicApi.js (the pricing call and it share one postExplanation; the body's kind tells them apart). The answer is shown under a "Generated by AI" label with the model, the time and "cached" in words, is keyed to the exact body it explained so it goes away when the focus or the build changes, and reads the quota (429) and the paused or off states (503) as sentences with no retry, the way the pricing button does — the button, the panels and the click-only rule now live once in pages/tools/explain/ExplainControl.jsx, and scenario/ExplainButton.jsx is what the pricing page sends and the words it uses, its tests unchanged. A landing-zone code fence whose body is the build's share query string (lz=mg,policy,mgmt,hub,fw&corp=2) renders a read-only card in an article: components/content/LandingZoneEmbed.jsx is the lazy shell (the frame and placeholder shared with the pricing embed through the new EmbedShell.jsx) and LandingZoneCard.jsx the chunk, decoding with decodeLz, drawing the diagram through the page's own SVG (moved into landingZone/LzSvg.jsx, with a non-interactive mode that leaves the boxes as plain groups), listing the selected components with a line each, and linking to the same build in the tool. markdownCodeComponents in components/shared/CodeBlock.jsx now dispatches through an EMBED_COMPONENTS map, { language: Component }, so the next tool's fence is one line; the pricing-scenario entry and its tests are unchanged. The fence is documented beside pricing-scenario in docs/content/blog-template.md. Tests: the button's body against a copy of the server's allowlist for every component, each response state, the answer dropped on a focus change; the embed's decoding, card, link, junk tolerance, pre-render and hydration; both fences dispatched by the map.

  • Public labs reads: the Hybrid Lab estate and Coder status, with no browser call to either (#664, #680). Backend halves of Phase 4 of #656 and Phase 2 of #659. GET /api/public/labs/estate reads the Arc-enabled lab host through Azure Resource Graph with the Function App's own managed identity, under a new Reader grant scoped to rg-lab-hybrid-prod-cus alone (infra/lab-hybrid.tf creates the group and the assignment; no new credential exists): the machine's status, last status change, agent version and OS name — never its name, which is the hostname — plus policy compliance counts for the group, whether a vps-agent is heartbeating and how many lab_jobs are queued, and Coder capacity. It answers { configured: false } when no machine exists and 503 when Azure cannot be read, never an invented row, and a side read that fails is null, not zero. GET /api/public/labs/coder-status is the server-side proxy the CSP requires: CODER_URL and CODER_STATUS_TOKEN are Key Vault references to CODER-URL and CODER-STATUS-TOKEN (catalogued for the API-keys page under a new Hybrid Lab section), the token stays in the Function App, and the route returns template names with their active version, the running workspace count and CODER_MAX_WORKSPACES (default 5, the Community cap) after GET /api/v2/templates, /api/v2/workspaces?q=status:running and one /api/v2/templateversions/{id} per template with a 5 s timeout. Both routes are anonymous and cache one document a minute in tool_service_cache, so visitors cannot drive the management plane or Coder; a failed Coder read is cached as unreachable so the card never shows stale numbers. Listed in PUBLIC_ROUTES and .azure/api-surface.json. The hcw-azure apply that creates the group, the grant and the three app settings is the owner's run.

  • Coder Phase 1: Docker Compose on the lab host, the Caddy route, the hcw-lab workspace template with its hardening test, and the Ansible role (#679). Phase 1 of #659, on ADR 0032. lab-host/coder/docker-compose.yml runs exactly two services, coder (ghcr.io/coder/coder v2.37.3) and coder-postgres (postgres 16.15), both by image-index digest held once in group_vars/all.yml and interpolated through a role-written .env; the Docker socket is mounted into coder only, which runs as the image's non-root user with group_add set to the host's docker GID, publishes 7080 on loopback only, disables password auth and Coder's default GitHub app, and reads its secrets from /etc/hcw/coder/coder.env (root, 0600) with the database password in a second file the database alone sees. templates/hcw-lab/main.tf adapts Coder's Docker starter template: a lab dropdown validated against the three catalogue ids, the hcw-lab image by digest, user = "65534:65534", hard limits of one CPU and 2 GiB, all capabilities dropped, no-new-privileges, a per-workspace named volume as the only mount, a per-workspace bridge network created and removed with the container so it never joins the Compose network, and the pinned code-server module on a *.coder.lab subdomain. template.test.mjs (node --test) reads both files as text and fails when the socket appears outside the coder service, anything is privileged, a host_path or mounts block appears, network_mode or the Compose network is referenced, or the user is not 65534; the new coder (lab-host) CI job runs it with docker compose config and terraform validate. The coder Ansible role installs the Compose project, fails closed without a non-empty organisation allowlist, the three Vault keys or on a host too small for coder_max_workspaces, brings the project up or down, renders /etc/caddy/conf.d/10-coder.caddy (503 with a sentence when the service is stopped, so the kill switch is legible), and keeps seven nightly pg_dumps under /var/backups/coder. Found on the way: the Coder agent runs every script through the user's /etc/passwd shell, and Debian's nobody has nologin, so the template writes the image's passwd back with a shell for uid 65534 until lab-image/ does; and coder templates push has no --default-ttl in the current CLI reference, so the one-hour autostop is coder templates edit hcw-lab --default-ttl 1h.

  • The Azure Verified Module pins are checked against the registry every week, and a newer release opens a pull request (#671). Phase 5 of #657. frontend/src/lib/landingZone/avmVersions.js says its pins are "looked up, not remembered"; .github/workflows/update-avm-versions.yml is what looks, Tuesdays at 06:30 UTC and by hand, modelled on update-learn-catalogue.yml down to the two-job split, the App token minted for a few API calls and revoked, the automation environment, and the ready-for-review pull request. frontend/scripts/update-avm-versions.mjs imports AVM_MODULES, asks registry.terraform.io/v1/modules/Azure/<name>/azurerm for each module's latest release, refuses anything that is not MAJOR.MINOR.PATCH tagged v<version>, and for a module that moved rewrites exactly one avm('<name>', '<version>', …) pin, restamps AVM_VERIFIED_ON with the UTC date, downloads the release tarball from GitHub (the proof the tag exists) and, where lab-image/versions.env vendors the module (#658), rewrites its _VERSION and _SHA256 lines with the tarball's sum so the image and the builder cannot disagree. Exit 0 and no writes when nothing moved; --dry-run prints without writing. The summary the pull request carries tables every pin's old → new with its GitHub release notes and says whether the release's required_providers drifted from the pinned requiredProviders — the edit the script does not make, flagged as HAND EDIT NEEDED. Because the versions are in the hcl.test.js snapshots, the workflow refreshes them with npx vitest run -u src/lib/landingZone after a bump and then runs the same suites without -u as the gate, so the pull request carries the Terraform diff a learner would download and a drifted input name fails in the run rather than on the PR. The branch stages exactly the pins, the versions file and the snapshot, asserted by scripts/avm-versions-workflow.test.mjs; the comparison, validation, text edits and summary are the pure half in frontend/scripts/avm-versions-edits.mjs and are unit-tested beside the script; index.test.js now asserts AVM_VERIFIED_ON is a calendar date no earlier than 2026-09-25 instead of naming the day.

  • /education/labs: the browser labs page, with the lab catalogue, Open in Coder deep links, Run it locally, and the estate and Coder status cards (#681). Phase 3 of #659, carrying the frontend halves of #664 (the "Hybrid Lab right now" card) and #680 (the Coder status card) against the contracts those issues fix. frontend/src/data/labs/catalogue.js is pure frozen data — one row per lab with id, title, summary, tools, template, params, articleSlugs and estimatedMinutes, and a test that every row carries every field and no two share an id — starting with the Landing Zone Builder download, a terraform validate walkthrough and an Ansible syntax-check walkthrough. Each card links to https://coder.lab.hybridcloudworks.com/templates/hcw-lab/workspace?mode=auto&param.lab=<id> through safeUrl, and carries the two docker run lines from #658, PowerShell then bash, each labelled with its shell. LabsEstateCard.jsx reads GET /api/public/labs/estate through the new fetchLabsEstate() and renders "The lab host is not provisioned yet." for { configured: false } and, when configured, the Arc status word, the heartbeat age in words, agent version, OS, policy counts, job-runner queue and Coder capacity; CoderStatusCard.jsx reads GET /api/public/labs/coder-status through fetchCoderStatus() and says "not yet provisioned" or "unreachable" before it shows templates and capacity. Every state on the page is a word, never a colour alone. Two slot sections hold the layout for the agent section (#676) and the article list (#677). Wired in App.jsx, STANDALONE_ROUTES, staticRoutes.labs, a new Learn menu in the header (the first header link to /education as well) and a section on /education, each enforced by a test.

  • Landing Zone Builder, Phase 2: the page at /tools/landing-zone (#668). Phase 2 of #657, on the pure module #667 landed. frontend/src/pages/tools/ LandingZonePage.jsx is where a learner assembles an Azure landing zone component by component: a build panel (landingZone/LzControls.jsx) with a checkbox per component, grouped platform and application, a count of 0 to 5 for corp and online landing zones, and one knob per option (region, parent management group, hub and spoke address spaces, private DNS zones, firewall SKU); a teaches panel (LzTeaches.jsx) that follows whichever component was last ticked, clicked in the diagram or asked about, with its hand-written explanation, what it needs, the pinned module that deploys it and its knobs; an inline SVG of the management group tree (LzDiagram.jsx) drawn from layoutDiagram, every box a button that focuses its component, with zoom and pan from the react-zoom-pan-pinch the site already shipped; and the generated Terraform (LzFiles.jsx), one tab per emitted file rendered with the site's CodeBlock, plus a Download zip button.

The build is the URL. useLzState.js is useScenarioState.js adapted to share.js: every control writes ?lz= and the option keys and reads itself back, so a bare URL is the full default build, a shared link reproduces a build exactly, and the pre-rendered page and the first client render agree; a test renders the page to a string twice and hydrates the result without a mismatch. The dependency rule is spoken, not just enforced: a component says "Ticking it also adds Connectivity hub" before it is ticked and "Unticking it also removes Azure Firewall" after, and a spoke range the state moved off the hub is explained in a sentence, kept on the page even once the URL carries only the moved value. An invalid range is refused in the validator's own words and never reaches the URL.

The zip holds exactly the files the tabs show. fflate is the one new dependency, imported inside the click handler so it is a lazy chunk a reader who never downloads never fetches and the pre-render never evaluates; the test mocks it and asserts the entry names equal the tab names and each entry's bytes equal the file's content. Wiring, each held by an existing test or the new one: the lazyPage route in App.jsx, the standalone list in scripts/prerender-entry.jsx, staticRoutes.landingZone, a Tools menu entry in Header.jsx, and a first card on /terraform/tools that opens the page. The Azure education page has no tools list, so it got no cross-link. The three placeholder tool slots are untouched. Helmet sets the title and the canonical; the default build renders the management groups' teaches text and a dozen file tabs, well past the prerender's 420-character floor.

  • The explain route is generalised by kind, and the Landing Zone Builder gets its "Explain this component" backend (#669). Phase 3 of #657, backend half. POST public/cloud-tools/explain reads an optional kind from the body: omitted or pricing is the #613 contract byte for byte (same validator, same canonical hash, same pricingExplain toggle, same stored document — the existing tests are unchanged and the existing cached texts keep serving); landing-zone dispatches to functions/src/lib/cloud-tools/explain/kinds/landingZone.js, whose validator accepts { kind, componentId, selected (<= 12), options, teaches (<= 1200) } against frozen copies of the catalogue's eight component ids and eight option knobs, every string capped, unknown keys refused, and whose prompt asks for two paragraphs — why this component matters for exactly this selection and options, then what changes if it is deselected — grounded in the page's own teaches text and the four Azure Verified Module sources embedded per component, with no tenant and no prices. The cache, the 8 KB cap, the 5 per hour per client and the 200 per day are the same code and the same counters for both kinds: one anonymous AI budget. The landing-zone canonical text starts with kind, so its cache ids cannot collide with pricing's. AI_FEATURES.landingZoneExplain is the kind's own off switch, with its call site in the kind module so ai-call-sites.test.js can see the literal. The kind shape (id, feature, validate, canonical, cacheFields, generate) is documented on kinds/pricing.js; .azure/api-surface.json spells out the landing-zone body for the frontend phase (#670). Tests: kind dispatch, the unchanged default, an explicit kind: "pricing" hashing the same as none, unknown kind 400, the landing-zone validator's refusals, quota and daily pause shared across kinds, the toggle off for one kind while the other serves, and the prompt naming the component and every selected id.

  • The lab host is configured by Ansible, not by hand over SSH (#662). Phase 2 of #656. lab-host/ansible/ holds site.yml and five roles that replace the manual steps the admin Labs page's Setup tab has printed since #577: hardening (a key-only hcwadmin with passwordless sudo whose keys are copied from root's, so the first run cannot lock the host; an sshd drop-in named 00- so it sorts ahead of cloud-init's PasswordAuthentication yes; ufw deny-in/allow-out with 22, 80 and 443; unattended-upgrades rebooting at 04:30; a fail2ban sshd jail on the systemd backend, because Ubuntu 24.04 has no auth.log), docker (Docker Engine 5:29.8.1 with buildx and compose from Docker's repository, all held, with 10 MB x 3 json-file logs and live-restore; the install carries allow_change_held_packages so a pin bump is not refused by the hold it set last time), node_exporter (1.12.1, SHA256-verified, loopback only), caddy and labs_agent. Every version, digest and checksum is in group_vars/all.yml — including the SHA256 of the Docker and NodeSource apt signing keys, because a pin on the versions with no pin on the key that vouches for them is decorative — the collections are pinned in requirements.yml down to community.docker's one transitive dependency, and every role carries meta/argument_specs.yml and a README.

Caddy is host-native and built, not downloaded, because the download API does not pin. The stock apt package and the upstream image both lack the Cloudflare DNS module, and caddyserver.com/api/download — the usual way to add one — always builds the latest release: a request naming a version that does not exist returned the identical 48,173,218-byte binary, so it can be neither pinned nor checksummed. So the role runs xcaddy build v2.11.4 --with github.com/caddy-dns/cloudflare@v0.2.4 inside the official caddy:2.11.4-builder image — pulled and run by its pinned digest, never by tag, so Docker verifies the content and a bumped pin is a new reference that gets pulled — refuses the result unless caddy version names the pin and caddy list-modules lists dns.providers.cloudflare, and writes a .provenance file beside the binary naming the three inputs. The binary's filename carries all three — Caddy tag, module tag, the first twelve hex of the builder's index digest — so bumping any of them rebuilds rather than relabelling an old build. The provenance in the repository is the caddy_* pins in group_vars/all.yml. It runs as caddy.service as the non-root caddy user reading /etc/caddy/env through EnvironmentFile= (owner root, group caddy, mode 0640 — the group read is what lets renewals work — holding the runtime Cloudflare token from an Ansible Vault variable, distinct from Terraform's), serving the three names ADR 0032 puts on the certificate: lab.hybridcloudworks.com, *.lab.hybridcloudworks.com and *.coder.lab.hybridcloudworks.com, DNS-01 answered through the CNAME delegation of the two _acme-challenge names to the lab zone. The site block imports conf.d/*.caddy: the apex placeholder is 00-apex.caddy, and Coder (#679) adds a file rather than editing the template. Without the token the Caddyfile fails closed — an HTTP-only apex answering 503 that imports nothing from conf.d — so a first bootstrap is green, the second turns TLS on, and a missing secret can never put an application route on plaintext.

vps-agent runs host-native as the systemd service the Setup tab described, which ADR 0032 settled. The agent's whole job is docker run, so a containerised agent needs the Docker socket mounted and a Docker CLI baked into an image the repository does not have, for no isolation gain — socket access is host root either way — and the Agents tab's own diagnostics already say systemctl restart hcw-labs-agent and journalctl -u hcw-labs-agent. The role checks the repository out at a pinned sha, runs npm ci --omit=dev once per ref (a stamp file, so a bump reinstalls and a re-run does not), writes /etc/hcw/labs-agent.env with the exact names from vps-agent/.env.example, and generates the key on the host as that file asks. /etc/hcw/labs-agent.pem is root:hcw-labs-agent 0640, no ACLs — root owns it, the service reads it through its group, nobody else can — and .env.example, the Setup tab's step 4 and docs/standards/variables-and-secrets.md now all say exactly that instead of the 0600 a non-root service could never have read. Only the public half, /etc/hcw/labs-agent.crt, leaves the host. The certificate lasts 730 days and nothing renews it by itself, so every run checks it with openssl x509 -checkend and warns within 60 days of expiry, and the README carries the two-step rotation: register the next public certificate first, then swap the PEM. The unit sets TMPDIR=/var/lib/hcw-labs-agent/tmp, because lib/docker-runner.js stages each payload under os.tmpdir() and bind-mounts it into the job container, and the PrivateTmp /tmp the unit otherwise gets is one the Docker daemon cannot see. Reviewing that surfaced a defect the runner has carried since #577: mkdtemp creates the per-job directory 0700 as the agent user while the container runs as 65534:65534, so /workspace was untraversable and every job would have failed before its command ran. prepareJobDir now makes the directory 0755 and the payload 0644 — the directory is per job, holds only the payload copy the API already sent, is bind-mounted read-only and is removed after the job — and docker-runner.test.js asserts the modes. Until the vault holds the identity the unit is installed, stopped and disabled and the env file is removed — explicitly, so removing a value from the vault is a revocation on the next run, not a file a manual systemctl start could reuse — and the play says which values are missing. bootstrap.sh refuses to run when exactly one of the vault file and its password file exists, rather than treating a half-present vault as no vault and turning TLS off. The hcw.lab-job label ADR 0032 requires on job containers is the agent's own change (#675); this role sets no labels.

lab-host/bootstrap.sh — what #661's post-install script will call — installs ansible-core 2.21.4 with pipx, clones the repository at the pinned sha into /opt/hcw-src, installs the collections and runs site.yml against localhost, passing the vault at /etc/hcw/ansible/ when it exists. CI gains an ansible-lint (lab-host) job on the production profile plus a syntax check, gated in-job on lab-host/** the way every other ci.yml job is. scripts/validate-repository-structure.ps1 allows the directory and its READMEs, and scripts/no-wiki-pointers.test.mjs now reads .j2 files, the first tracked extension its coverage check had never seen.

One contract is declared ahead of its role. group_vars/all.yml carries coder_enabled: false and coder_oauth2_github_allowed_orgs: [] for Coder (#679), and site.yml asserts the list is non-empty whenever the flag is true — because an empty CODER_OAUTH2_GITHUB_ALLOWED_ORGS is not a lock, it lets any GitHub account sign in. The README names the real kill switches: CODER_OAUTH2_GITHUB_ALLOW_SIGNUPS=false for new learners, stopping the coder Compose service for everyone. - Landing Zone Builder, Phase 1: the pure module, with no page yet (#667). frontend/src/lib/landingZone/ is the logic behind the Landing Zone Builder (#657), built the way pricingScenarios/ is: frozen catalogues, small pure functions, the state as a value and as a URL, and nothing that reads a clock, a viewport or a tenant. components.js is the catalogue — six platform components (management-groups, policy, management, connectivity-hub, firewall, identity) and two application ones (corp, online, each counted 0..5) — every one with a one-sentence summary, a hand-written teaches of three to five sentences (what it is, why a landing zone has it, what breaks without it), its dependsOn, the module that deploys it, and its knobs: hub CIDR, spoke range, firewall SKU, private DNS zones, region, the two counts and the parent management group, each with a validator and a default.

The module pins are looked up, not remembered. avmVersions.js holds exactly four modules with verifiedOn: '2026-09-25', each read from the registry's v1 API and the repository's latest GitHub release that day: Azure/avm-ptn-alz/azurerm 0.21.0, avm-ptn-alz-management 0.9.0, avm-ptn-alz-connectivity-hub-and-spoke-vnet 0.17.5 and avm-res-network-virtualnetwork 0.22.2. The pattern module for application landing zones, avm-ptn-alz-application-landing-zone-identity-and-access, was on GitHub that day as the unfilled AVM template with no tag, no release and no registry listing, so the builder does not call it: a landing zone is a subscription_placement entry in avm-ptn-alz plus a spoke from the virtual network module instead, and a test asserts every emitted module source is pinned with a version line. The provider constraints (alz ~> 0.21, azapi ~> 2.12, azurerm ~> 4.35, modtm ~> 0.3, random ~> 3.6, time ~> 0.9, Terraform >= 1.12, < 2.0), each module's requiredProviders list, and the platform/alz/2026.08.1 library ref come from the same modules' terraform.tf files at their tags and the library's release list; every emitted module block's providers map is generated from that list, since a block with a providers argument inherits nothing for the providers it leaves out. avm-ptn-hubnetworking is archived; a test asserts the string never reaches any emitted file.

The build is a URL. share.js encodes the state as ?lz=mg,policy,mgmt,hub,fw,id&hub.cidr=10.0.0.0/16&spoke.cidr=10.1.0.0/16&fw.sku=Premium&dns=0&loc=westeurope&corp=2&online=1&root=contoso: lz= lists the selected platform components by short id, the application components travel as their counts with 0 meaning not selected, root= is the management group the alz root is created under (absent for the tenant root group, wired to the module's parent_resource_id), and every default is left out, so the default build — the whole landing zone with one corp and one online — is a bare URL and an empty build is ?lz=&corp=0&online=0. Decoding drops unknown tokens, defaults any option that fails its validator, and normalises through state.js, which closes the selection over dependsOn (a firewall brings the hub; identity, corp and online each bring the tree and the hub they peer to), keeps the counts and the selection in step, and checks the two ranges against each other: a spoke range that overlaps the hub is moved to the first free fallback (10.1.0.0/16, 10.2.0.0/16, then 172.16.0.0/16) and the state carries a warnings entry ({ code: 'spoke-cidr-overlap', from, to }) for the page to show. isLzParam says which keys are the module's, as isScenarioParam does for the pricing page.

hcl/ emits the Terraform the validated pattern would, one module per emitted file. emitFiles returns { path, content } files, only for selected components: terraform.tf, providers.tf (default providers from the workspace's ARM_* variables, an azurerm and an azapi alias per subscription), alz.tf (architecture_name = "alz", one placement per subscription, and when policy is selected all fourteen policy_default_values the pinned library declares — the six AMA and workspace ids from the management names, the three private-DNS values from the hub, the location, two resource-group names and a validated security_contact_email variable — with the one value the build never creates a source for, the DDoS plan, and the DNS trio when zones are off, handled by setting their assignments DoNotEnforce rather than leaving a library placeholder; when policy is not selected the tree is still the alz architecture, which carries its baseline, so every one of the 123 assignments the twelve archetypes make at that ref is emitted with enforcement_mode = "DoNotEnforce" and the defaults the build can supply are still passed, with a comment saying the baseline is present and inert because Policy was not chosen), management.tf, connectivity.tf (the firewall and its policy as blocks of the hub object only when selected, the DNS resources on the option), identity.tf and application.tf (one spoke per landing zone: a resource group, a /24 carved from the spoke range with cidrsubnet, a two-way peering to the hub through the connectivity module's virtual_network_resource_ids, and for corp and identity a route table whose default route is the firewall's private IP when the firewall is selected; online spokes egress directly), variables.tf with GUID validation on every subscription id, the range variables carrying the same rules as the JS validators — a shape check (dotted quad, prefix within HUB_PREFIX_RANGE /8–/24 or SPOKE_PREFIX_RANGE /8–/20, both generated from the constants isCidr and isSpokeCidr use, and cidrhost able to parse it) and, on the spoke range, the hub-versus-spoke overlap rule as a second validation that reads var.hub_address_space (Terraform 1.9 lets a validation read another variable) and is guarded so it never evaluates on a string that does not parse; proven offline with terraform plan on the emitted file: the apart pair plans, a /25 spoke, a /25 hub, a garbage string and an octet of 256 each fail the shape check, and identical, hub-inside-spoke and spoke-inside-hub each fail the overlap message — and a non-empty, /-rejecting check on var.parent_management_group_id, because avm-ptn-alz's parent_resource_id is the parent group's name and the module itself refuses a resource id, subscriptions.tf (every placed subscription id in one list behind an output precondition that refuses duplicates, since a subscription can sit under one management group only; a duplicated id fails the offline plan on that message, a distinct set passes), terraform.tfvars.example with distinct placeholder ids, and a README.md carrying the pattern's prerequisites — HCP Terraform, Owner at the tenant root, a service principal — the spoke pattern in two sentences, and the line that the files were generated for learning and never applied by HybridCloudWorks. format.js aligns = the way terraform fmt does, across runs of single-line attributes and not across a multi-line value or a comment; six builds (default, Basic firewall, hub without DNS, tree only, five-and-five Premium, identity only) pass terraform fmt -check -recursive under Terraform 1.15.8. cidr.js is the carve — corp from the low half, online from the high half, identity at the top of the low half — and diagram.js lays the same state out as a tidy tree of management groups, hub and spokes with their /24s and peering edges, in fixed units so pre-render and hydration agree.

Validators refuse what Number would quietly accept: a CIDR must be canonical (010.0.0.0/8, 10.0.0.0/08, /+8, /8 and /8.0 are all rejected, not read as 10.0.0.0/8) and a landing-zone count is a number or a string of digits only, so a blank field or corp=%20 falls back to the default instead of becoming zero and deselecting.

Three suites, 209 tests, each emitted file and each module block its own row: every component has every field and a teaches of the right length, the dependency closure, the carve, canonical-form and count validators, the overlap fallback and its warning (through normalizeState, setOption and a decoded URL), encode/decode round trips and default omission, decode tolerance, fmt shape on every emitted file, every module source pinned with a version, the archived and the unpublished module absent, all fourteen policy defaults supplied or their assignment not enforced, all 123 baseline assignments DoNotEnforce when Policy is off, both variables.tf cross-checks emitted, committed snapshots of the default and the tree-only builds (the repository's first __snapshots__), and diagram determinism with no overlapping nodes. No page, no route, no dependency; frontend/ vitest goes from 2,272 to 2,481.

  • ADR 0032 records the learner labs platform, and the two documents that misdescribed the labs are corrected (#660). Phase 0 of #656: the decisions every sub-issue in the four lab epics (#656, #657, #658, #659) builds on are written down once, as a Proposed record, before any of them is implemented.
  • docs/decisions/0032-learner-labs-platform.md: the Hostinger VPS is Terraform-managed through hostinger/hostinger in its own hcw-lab workspace so lab state never meets hcw-azure; Ubuntu 24.04 with Docker Engine as the only runtime and no Kubernetes (owner decision 2026-09-24); Azure Arc as the hybrid control plane with heartbeat and auth syslog only and Defender for Servers off; Coder as the learner identity boundary the site links to and never embeds; digest-pinned images from lab-image/ as the one toolchain; and anonymous public submission held Gated with its bounds stated. Listed in the register and the site nav.
  • docs/architecture/labs-host.md is the estate record for the host — provider, plan, OS, what runs, exposure, backup posture and the identities it holds — every row marked planned, with no addresses and no secrets.
  • docs/architecture/architecture.md §5.3 said the browser submits a lab request. It never has: submitPublicLabJob was deliberately not ported (functions/src/lib/labs.js). The section now says submission is admin-only through /admin/labs and enqueueLabJob, and points at ADR 0032 for the gated public path.
  • docs/standards/required-inputs.md §4.7 gains the inputs the ADR names — hostinger_api_token and cloudflare_api_token in hcw-lab, the Arc onboarding credential in Ansible Vault, CODER-URL and CODER-STATUS-TOKEN in Key Vault (read by the underscored app settings) — all MISSING.

  • .github/CONTRIBUTING.md still said new work lands in TODO.md. It lands as an issue on org project 1 with a Priority set (owner decision 2026-09-05, #362); TODO.md keeps the accepted risks and an index.

  • The hcw-lab image: one Dockerfile with runner and full targets, a Terraform provider filesystem mirror, three vendored AVM pattern modules, and a workflow that publishes both to GHCR with provenance attestations (#674). Phase 1 of #658. lab-image/Dockerfile builds from a digest-pinned debian:bookworm-slim and installs terraform 1.16.4, kubeconform 0.8.0, helm 4.3.0 and ansible-core 2.19.13 (the last line that runs on bookworm's Python 3.11), every download checked against the SHA256 in lab-image/versions.env — the one file a version bump edits. The runner target carries /opt/terraform/mirror, built with terraform providers mirror for azurerm 5.7.0 and 4.81.0, azapi 2.12.0, alz 0.22.0, random 3.9.1, modtm 0.4.0 and time 0.14.2, each zip re-verified against the registry's published sum after mirroring, and a TF_CLI_CONFIG_FILE that installs from the mirror only: terraform init under --network none succeeds for anything the mirror holds and fails loudly for anything it does not, instead of hanging on a registry it cannot reach. avm-ptn-alz 0.21.0, avm-ptn-alz-management 0.9.0 and avm-ptn-alz-connectivity-hub-and-spoke-vnet 0.17.5 are vendored at /opt/avm/<name>@<version> from their release tarballs; the fourth module the issue names has no release yet (its repository is still the AVM template) and is recorded as such in versions.env. The image runs as uid 65534 with /workspace read-only, everything it writes under /tmp. The full target adds Azure CLI 2.90.0 from Microsoft's apt repository (key verified by SHA256, package version pinned), kubectl 1.37.1, git, curl, jq and code-server's three prerequisites.

What the offline init proves and what it does not, measured rather than assumed: lab-image/smoke.sh first asserts there is no network, the mount is read-only and the uid is 65534 — an init that passed with network would prove nothing about the mirror — then compares every tool's reported version with versions.env, and runs terraform init -backend=false on a root using azurerm and random and on the vendored management module, both with no network. The management module is the only one of the three whose init needs nothing outside the mirror; avm-ptn-alz calls one registry module (Azure/avm-utl-interfaces/azure) and the connectivity module calls thirteen distinct ones across twenty-one module blocks, and Terraform has no module mirror, so those two need a registry for their child modules. lab-image/README.md says so, with the measured sizes.

.github/workflows/publish-lab-image.yml builds both targets and runs the smoke test in each on every pull request touching lab-image/** (loaded locally, never pushed), and on a push to main pushes ghcr.io/hybridcloudworks/hcw-lab-runner and ghcr.io/hybridcloudworks/hcw-lab tagged with the commit sha and latest, attaches an actions/attest-build-provenance attestation to each, and writes the digests to the job summary for #675 to pin from. Every action is pinned by commit sha; the permissions live on the job. The repository structure policy admits the lab-image directory and its README.

Changed

  • Podbean MCP is retired from the AI Engine; the site reaches Podbean through its API. Seeded on 2026-10-13 (#1114) and retired the same day, on two measurements against mcp.podbean.com:
  • its sign-in server allowlists redirect addresses to Claude, ChatGPT, Cursor and loopback apps. Registration with the site's callback was refused, and a client-metadata document listing it (which Podbean does read) still answered Unregistered redirect_uri, while a loopback address in the same document was accepted;
  • its MCP server answers 401 to a Bearer token minted by a Podbean developer app, so no token the site can hold is accepted.

No web app can connect to it, this site included. podbean-mcp joins DEPRECATED_MCP_SERVERS and is deleted on the next AI Engine load; its credentials-register row goes with it. The developer app's credentials do work against api.podbean.com, which is how publishing (#1117) already reaches Podbean, and where any further Podbean capability for the AI Engine is built (#1115, re-scoped).

  • Approved podcast episodes publish to Podbean; the RSS.com integration is gone (ADR 0029 §1c). Hybrid Cloud Insights moved to Podbean on 2026-10-13, and its feed is now https://podcast.hybridcloudworks.com/feed.xml. RSS.com's API path never ran: RSS.com Max was never bought (#432), so every approval was recorded as not configured.
  • The client: functions/src/lib/podcast/podbean.js replaces rsscom.js. It was written against Podbean's own API documentation data. It exchanges a developer app's client ID and secret for an access token (client credentials) and caches the token for its life less a minute. It authorises the upload, PUTs the MP3 to the presigned URL with no token, and creates the episode with POST /v1/episodes as publish and public. The episode notes are escaped HTML with a link to the article. A re-run updates the episode the document names and never creates a second.
  • Retries: the create is never repeated. The one repeat is a call refused with invalid_token, once, with a fresh token.
  • A lost create is found, not duplicated (#1118). Podbean takes no idempotency key, and CodeRabbit held this to be fixed before merge.
    • When: before any create on a document with an earlier attempt and no episode id, the publish lists the show's newest episodes (GET /v1/episodes).
    • Match: one with the same title, published between ten minutes before the document's first attempt and twenty minutes after its last recorded one, is adopted and updated, with the audio re-sent. The upper bound keeps a late retry from adopting an unrelated episode published later under the same title.
    • Fails closed: a listing that fails or cannot be read creates nothing, and neither does an unreadable attempt time (non-retryable).
    • Recorded: firstAttemptAt and adoptedAt are kept on the record.
  • Credentials stay on the server: no error, log line or recorded failure carries the token, the secret or a URL holding either.
  • Same job, same retry route, new record: publish-podcast-transcript and POST cms/podcast/transcripts/{id}/publish are unchanged. The outcome is stored under host.podbean, with the episode id, page, media URL and file key. host.rsscom is neither read nor migrated, because it only ever held skipped: 'not_configured'. The job's timeout is now 10 minutes, above the client's deadlines including the one re-auth.
  • Integrations: a Podbean card replaces RSS.com's, with its badge, directory row and health probe. Its test makes the connection probe's first token exchange and then reads the show (GET https://api.podbean.com/v1/podcast). The exchange is a new, optional authenticate step on a probe, held to one POST and redacted like the GET. The card answers Connected — <show title>. and records no key verdict.
  • Recording Hub and admin copy name Podbean, and a published episode links to its Podbean page.
  • Public copy: both vendor guards now refuse "podbean" in public copy and public API messages.
  • Keys: the catalogue, the credential register, the expected-unresolved list and the docs swap RSSCOM-API-KEY and RSSCOM-PODCAST-ID for PODBEAN-CLIENT-ID and PODBEAN-CLIENT-SECRET. The client ID is an identifier; the secret is rotated by hand in Podbean's developer portal.
  • The Terraform plan shows four app-setting changes on azurerm_function_app_flex_consumption.hcw, all declared in scripts/assert-expected-plan.mjs. RSSCOM_API_KEY and RSSCOM_PODCAST_ID are removed. PODBEAN_CLIENT_ID and PODBEAN_CLIENT_SECRET are added as Key Vault references to PODBEAN-CLIENT-ID and PODBEAN-CLIENT-SECRET.

  • Owner steps: create a developer app at https://developers.podbean.com/, then paste its client ID and client secret on https://hybridcloudworks.com/admin/integrations?tab=keys. Success is the Podbean card's test answering Connected — Hybrid Cloud Insights.. Until both are seeded, approvals stay on the manual upload path, exactly as before, and the two settings are on the expected-unresolved list, so the secrets monitor stays green.

  • The README is written for someone new to the project, and the engineering detail moved to the docs site (owner, 2026-10-11).

  • README: the README now leads with what a visitor can do on the site:

    • learn a platform;
    • plan with the pricing comparison, decision matrix and Landing Zone Builder;
    • assess an estate with the migration assessment, now live;
    • practise in the labs.

    A start-here table follows, then one short paragraph on how the site is built, and a brief contributor path with a five-line local start. - New docs page: the architecture diagram, the layer table, the repository map, the full "run it locally" guide and the CI/CD description moved, intact, to How HybridCloudWorks is built. It is on the docs site under Architecture and updated for the tool add-ons and the alert fire tests. - Badges: they are grouped as status (CI, CodeQL, docs build, reviews, maintainability, coverage) and links (both sites, the work board, open issues, licence, last commit). The work-board and open-issues badges are new, and every badge was checked to render a real value. - Docs home page: it now opens with a "New here?" section. - Fire record: alert-kv-config-write is recorded, fired at 04:52Z by Invoke-AlertFireTest.ps1 -Test kv-config-write.

  • The HCW Work Board is public, so CONTRIBUTING links it again (owner, 2026-10-10). Open work stays GitHub issues; CONTRIBUTING, TODO.md and the code-review skill now name the board as public and link it (https://github.com/users/saulpatinojr/projects/6), where #1045 had pointed visitors at Issues alone while it was private.

  • The former organisation's board links and the docs CNAME now follow the repository to its owner's account (public readiness). infra/frontend.tf points docs.hybridcloudworks.com at saulpatinojr.github.io instead of hybridcloudworks.github.io (both resolve to GitHub's Pages addresses, so visitors see no change, and GitHub's custom-domain check expects the owner's own host); scripts/assert-expected-plan.mjs declares the update for the plan check. CONTRIBUTING sends open work to GitHub issues, since the Work Board is a private user project an anonymous visitor cannot open; TODO.md and the code-review skill link that board (user project 6), and TODO.md no longer claims the board adds new issues itself: its built-in workflows add sub-issues only, so whoever opens an issue adds it. The scheduled catalogue warning's comment says the same.

  • README: the repository's front door, for a visitor and a contributor (#1033). The README opened as "HybridCloudWorks Website" and read as a migration-era record. It now says what the site is and who it is for, shows the running architecture as a draw.io diagram with the official Azure icons (docs/assets/architecture/hcw-architecture.drawio, exported to SVG so GitHub and the docs site show it alike), with the same picture in words beneath it (built from infra/, docs/runbooks/edge-dns-verification.md and docs/architecture/labs-host.md, so Cloudflare proxies the API hostname only), maps every top-level directory with a link to its README or its docs page (.vscode/ and .qlty/ included), gives each package's install, test, CI and start commands from its package.json, names the runtime lines and leaves the exact floors to scripts/version-floors.json, says which checks CI runs for which package and that the docs site deploys only on a merge that changes its sources, and links CONTRIBUTING, SECURITY, SUPPORT, the code of conduct, the docs site and the licence. Open work is pointed at GitHub issues and TODO.md at the accepted risks. The badge rows are unchanged. The dated history is gone from the page; the migration record was already under docs/history/ and is linked once. The owner chose draw.io over Mermaid, whose boxes are generic and whose layout tangles; the docs hook (scripts/docs/hooks.py) now rewrites an image under docs/ to the site's own copy, so the README's diagram shows on the docs site too. Community health files (owner decisions, #1033): .github/CODE_OF_CONDUCT.md is the Contributor Covenant 2.1, verbatim from contributor-covenant.org with the enforcement contact filled in as the owner gave it (the two hybridcloudworks.com addresses), and .github/SUPPORT.md sends bugs and requests to the issue templates, documentation questions to the docs site and vulnerabilities to SECURITY's private reporting. CONTRIBUTING links both, and the repository-structure policy allows both beside CONTRIBUTING and SECURITY. Two fixes found on the way: scripts/docs/hooks.py stripped a leading . from every repository path (lstrip("./") takes a set of characters), so the README's link to .github/CONTRIBUTING.md on the docs site pointed at github/CONTRIBUTING.md and returned 404; it now strips only a leading ./ or /. And .github/SECURITY.md said accepted risks were recorded in the README, which has not held them; they are in TODO.md.

  • Terraform identity, SEC-1 step two: subscription Contributor comes off once the narrow grants are read back (estate review 2026-10-06, finding SEC-1; #971). scripts/bootstrap-terraform-oidc.ps1 gains -RemoveSubscriptionContributor, off by default. With it the script stops granting Contributor at subscription scope, reads every narrow grant back from Azure (Contributor on each of the nine groups, the HCW Terraform Subscription Scope assignment and a definition holding exactly the JSON's actions, and RBAC Administrator whose condition names Owner and Contributor, on all three subscriptions), refuses with nothing removed if one is missing, and only then deletes the identity's three subscription-scope Contributor assignments, printing Contributor at subscription scope removed for each and reading each back. -WhatIf prints one Would remove line per subscription. A plain run keeps step one's behaviour and grants subscription Contributor back, which is the rollback, so every re-run after step two carries the switch. The audit of what infra/ does outside its nine groups found nothing the narrow grants miss on plan or apply: writes above a group are the custom role's (groups, budgets, provider registration), reads above a group are RBAC Administrator's built-in */read, every cross-group reference has both ends in declared groups, and Contributor carries no data actions, so the custom role gains nothing. The one gap is destroy-time: azurerm's purge of a deleted Cognitive Services account is a subscription-scope delete the role deliberately lacks, so destroying the Foundry account is an owner step. scripts/terraform-identity-grants.test.mjs now classifies every azurerm_*/azapi_* type infra/ uses by the grant that covers it and fails on an unclassified one, on a resource group other than a declared azurerm_resource_group, on a literal Azure id above a declared group (the subscription budgets excepted), on a second delete in the custom role, and if the bootstrap's removal ever runs ahead of its refusal. Set-TerraformRbacCondition.ps1's recovery path already keys on the custom role as well as subscription Contributor, so it keeps working after step two; its comments now say so. ADR 0005 gains the dated amendment "2026-10-07, step two" with the audit table, the gap and the rollback; the deployment runbook, section 0, gains the owner's step-two procedure, the read-back one-liner, and how to read an AuthorizationFailed from the first apply.

  • Lab host: the agent leaves the docker group behind its own socket proxy, the daemon remaps user namespaces, and Coder's workspaces move to a rootless daemon (estate review 2026-10-06, finding LAB-5; #951). The remainder #900 left open. The agent: hcw-labs-agent is in no group but its own (the role takes it out of docker), and the docker CLI it spawns reaches the host daemon through hcw-labs-agent-docker-proxy, HAProxy 3.4.6 (newest LTS) by index digest, on a Unix socket only the agent's group may open, in a container with no network, a read-only root and no capabilities. Its allowlist, in roles/labs_agent/templates/docker-proxy.haproxy.cfg.j2, is the exact calls docker run --rm and docker rm -f make for a job, recorded from Docker CLI 29.8.1 against a stand-in daemon (scripts/lab/capture-docker-cli-requests.mjs, fixture in scripts/fixtures/); no pull, exec, inspect or list. Because a path allowlist cannot see a privileged create, the proxy also reads the create body and refuses privileged mode, host namespaces, the remap's opt-out, added capabilities, devices, mounts, any bind but the job's own read-only directory, unmasked paths, other security options or networks, and non-digest images; it refuses any byte outside ASCII and any backslash, which is what keeps those regular expressions sound against Go's case-folding JSON decoder. scripts/lab-host-docker-proxy.test.mjs evaluates the file's own rules against the recorded calls and the escapes, and CI parses it with haproxy -c at the pinned digest. User namespaces: daemon.json sets userns-remap: default with the containerd image store off (that store does not support the remap), so a job container's root, and Coder's server's and PostgreSQL's, is an unprivileged uid. The two proxies and Portainer opt out to open a socket, as does the one-off Caddy build. The switching run stops every container, restarts the daemon before the later roles (a flushed handler), and copies Coder's PostgreSQL volume (owners shifted into the range) and Portainer's (as is) into the new data root once with hcw-docker-volume-carry, refusing a non-empty destination or a copy older than its source; its test runs in CI as root. -e docker_userns_remap=false is the way back. Coder: the new coder_sandbox role runs a rootless Docker daemon (docker-ce-rootless-extras at the docker-ce pin) as the unprivileged user hcw-coder-docker, with its own subordinate ids and the cpu, memory and pids controllers delegated; coder-docker-proxy now mounts that daemon's socket directory instead of the host's socket, so a privileged container a compromised server or template asks for is that user's, not root's. An authorisation plugin was rejected (ADR 0032, amendment of 2026-10-07). lab_images plans each daemon on its own (--scope jobs, --scope workspaces). A new privilege_checks role ends every run by failing unless the agent is out of the docker group, docker info shows name=userns, a job run as the agent through its proxy succeeds while docker ps and a privileged run are refused, Coder's proxy carries the Compose file's policy with the sandbox socket as its one mount, and that daemon is rootless. Owner steps: re-run bootstrap.sh; running jobs and workspaces stop, and workspaces start again on the sandbox daemon with an empty home volume. The checks to paste afterwards, and what good looks like, are in the Labs host runbook, "Container-runtime privilege separation (LAB-5)".

  • Lab host: Coder v2.38.0 and code-server 4.140.0, with code-server's release notice off (owner, 2026-10-07; #911). The Coder pin moves from v2.37.3 to v2.38.0 (released 2026-10-06; index digest read from imagetools and the registry's Docker-Content-Digest, which agreed; its breaking changes, Tasks removal, MCP private-network CIDRs and HA pubsub, touch nothing this host uses). The workspace template pins code-server 4.140.0 (Code 1.140.0) and passes --disable-update-check: the "a new release is out" notice's link opened a window the site's sandboxed pane could not navigate, which is the blank window of #911, and the release is the pin anyway. Owner steps: re-run bootstrap.sh (pulls the new Coder image and recreates the server), then hcw-coder-template-push so the template version carries the new code-server; existing workspaces take it on their next start.

  • Plan-check declarations trimmed to the one still pending (2026-10-06). The hcw-azure runs after the stale #908 plan was discarded applied the three lab rule predicates, the SEND_REMINDERS flag (false, then true once enabled_timers named it), and confirmed the SEC-2 deletes, the ADR 0032 decision 6 keys and the #816 role removal are in state, so their entries in scripts/assert-expected-plan.mjs are deleted as its header asks. FEATURE_FLAG_PROBE_AI_PROVIDERS is still false in state, so its arming declaration stays.

  • Anthropic is a certification issuer, and the Top issuer stat wraps (owner request 2026-10-05). lib/certIssuers.js gains anthropic (aliases claude, anthropic pbc; learn link anthropic.com/learn; its own colour), so the Certifications editor's issuer combobox offers it and a cert typed as "Claude" maps to it. The Catalog tab's Top issuer card shows two lines instead of an ellipsis for a long label. New certIssuers.test.js pins the registry's shape, the exact-or-alias matching, and what a cert gets from its issuer.

  • Update Cert Catalog sits in the Certifications Hub header (owner request 2026-10-05). The publish button was on the Publishing tab, one tab away from where edits happen; it is now under the Powered-by pill on every tab, renamed Update Cert Catalog. ServicePageHeader gains an actions slot and PublishSnapshotButton a label (the Speaking Events tab keeps "Publish snapshot"). The Publishing tab keeps the snapshot summary and the changes-since diff, and re-reads after a header publish; the delete dialog, the editor and the moved-tab note name the new button. CertificationsPage.test.jsx pins the button in the header and out of the panel, and the tab's re-read after a header publish.

  • About page: a composition root over per-section modules (#842). frontend/src/pages/shared/AboutPage.jsx was one component with 70 return statements and a complexity of 130 under a file-wide eslint-disable complexity, flagged by qlty on every PR that touched it. It now arranges six modules under pages/shared/about/: the hero, the speaking section, the certification registry (its four bodies as four small components), the issuer group, the card and the badge modal, with the data in useCertifications.js and every ordering rule in certificationSorting.js as tables (certificationSorting.test.js pins the orders the page always produced). The eslint-disable is gone; no function in the touched files has more than five returns or a complexity above 18. Behaviour unchanged: the prerendered /about is byte-identical before and after (17,296 bytes, checked with a temporary render test), the card's markup is byte-identical across every branch for a fixture set, and the public-copy, accessibility and existing About tests pass unchanged (4,069 frontend tests in all).

  • AI router: the factory's closures are module-level functions over a context (#843). createAiRouter in functions/src/lib/ai/router.js was one 780-line factory with 24 return statements and a complexity of 137, flagged by qlty on every PR that touched the file. It now builds a context (env, fetch, the clock, log, store, the configuration loader, the NVIDIA pacing guard and the key-verdict reporter) and binds the public surface to it; chain resolution, callWithFailover, the per-provider callers and the generate calls take ctx as their first argument. Two of the hoisted functions were over the threshold on their own, so the OpenAI-compatible request body, the grounded call's usage row and its completion check are their own functions. In the touched code no function has more than three returns and none is above complexity 18 by ESLint's rule. Key handling, the CONTENTFORGE_AI_PROVIDER pin, per-feature placement and routing, failover order, retry and time budgets are unchanged: router.test.js, sync-budgets.test.js, ai-call-sites.test.js, nvidia-provider.test.js and probe-ai-providers.test.js pass without edits (4,604 backend tests in all).

  • NVIDIA's setup note and the seeding reference warn that a fresh key is refused for a while (#701). On 2026-10-04 a key seeded at 08:02 UTC got 403 Authorization failed from the app at 1, 12 and 13 minutes, the same reply a made-up key gets, then passed from a workstation at about 23 minutes and from the app at 37; a key seconds old passed at once from Azure Cloud Shell. Four reseeds and a rotation were spent on a key that was never wrong. The card's setup note and docs/standards/required-inputs.md now say to Test again later before reseeding and to delete the previous key only after the new one passes from the app. notes joins the seed-owned provider fields (frontend/src/lib/aiEngine/seed.js), so the note reaches the stored card rather than only a card seeded after it was written.

  • September's measured cost replaces the estimate (#822). docs/architecture/cost-analysis.md now states September 2026, the estate's first full month in centralus, from Cost Management (actual cost, by resource group and service, read 2026-10-03). The application subscription cost USD 34.32 and Platform Management USD 2.49, so USD 36.81 combined. That replaces the five-day reading of 2026-08-24 and the "USD 15–20" estimate. The page records the query and that the figures can still move slightly while September's billing finalises.

  • The total landed near the combined estimate, but the shape did not. The application subscription cost about twice its estimate, and Platform Management about an eighth: Log Analytics ingestion billed USD 0.00.
  • The largest line is rg-web-site-prod-cus Storage at USD 12.40. That is stsitefuncprodcus01, the Functions host storage (Standard_GRS). Its September metrics show the line is transactions, not capacity: 17.3 million operations (12.0 M GetBlob), 0.64 GB stored. It is named as the first line to look at. Functions, Azure Monitor and Cosmos follow at about USD 6 each.
  • The VALIDATION and OPEN_ITEMS lines record the gap and its drivers. The subscription budget now measures the workload, because the retired-resource tail has cleared.
  • The Static Web App deploy keeps its deployment token, as an accepted risk (#834). Owner decision 2026-10-03, recorded under Accepted risks in TODO.md.
  • Today the deploy job signs in with GitHub OIDC and reads the token just in time through the HCW Static Web App Deployer role. It uses the token for one run, from main only, with a production approval.
  • Moving to the "GitHub" policy would not remove the token. Microsoft still labels the token the recommended policy, and its identity-token example still passes azure_static_web_apps_api_token beside github_id_token.
  • Compensating control: reset the token every 90 days or on any suspicion. Nothing stores it, so a reset breaks nothing.
  • The comments in deploy-azure-frontend.yml that called this open now point at the accepted risk.
  • Production deploys have two required reviewers. Since 2026-10-03 the production environment names saulpatinojr and hcw-architect, and either can approve. Required-Inputs §4.4 and the comments in both deploy workflows said one.
  • The six-hourly cp_sortDate healer is gone (#816, part 2). Part 1 (#829, applied 2026-10-03) made the apply write cp_sortDate onto content and blogs whenever it updates either container, and fail when one lacks it. With that holding, the band-aid comes out.
  • .github/workflows/heal-computed-properties.yml is deleted. No workflow writes computed properties any more.
  • azurerm_role_assignment.github_deploy_cosmos_container_writer and its data.azurerm_role_definition.cosmos_container_writer lookup leave infra/oidc.tf, so the deploy identity holds no Cosmos control-plane write. infra/roles/cosmos-container-writer.json goes with them. The role definition was created by hand, outside Terraform, so the owner deletes it by hand after the apply.
  • scripts/apply-computed-sortdate.mjs loses --apply and its ARM helpers. --inspect stays: it is the one check the apply cannot make, that every date alias in the data is ISO-sortable, and it now runs locally. --apply exits 2 with the usage text, which a test pins.
  • scripts/assert-expected-plan.mjs declares the role assignment's delete and drops part 1's three declarations, which have applied. A declared delete must now name a resource that infra/*.tf no longer declares.
  • The runbook's computed-properties row, Required-Inputs (the custom-role table, CLIENT_ID, and COSMOS_ENDPOINT and COSMOS_RESOURCE_GROUP, now read by nothing), and the comments that described the healer as running now describe what is true.
  • App-settings values pass through tostring() (#837). The azapi_update_resource that strips AzureWebJobsStorage from the Function App's settings now wraps each value in tostring(). Every value is already a string, so the plan does not change. If a value ever comes back as an object or a list, the apply now stops with an error instead of writing it back in that shape. It was held back while T-513 was open, and T-513 closed as a name collision rather than a value problem. The comment in infra/functionapp.tf no longer calls the hardening pending.
  • Comments no longer send readers to TODO.md for open work (#821). TODO.md stopped carrying work on 2026-09-05, but about 120 files still cited it, mostly as (TODO.md T-nnn). A reader who followed one found a file of accepted risks and nothing about the item.
  • TODO.md T-nnn is now T-nnn throughout code, config and current docs. The T-items are closed; each one's record is in this file or in the commit that closed it. .github/CONTRIBUTING.md now says so, and says that a comment pointing at open work names its issue.
  • Other TODO.md pointers now name what they meant: Required-Inputs §4.1, §4.2, §4.5 or §4.6, an ADR, or the changelog entry. Mentions that describe the file itself (the structure gate, the docs build, the accepted risks, the handling rules) are unchanged. So are docs/history/ and the dated architecture records, which are evidence kept as written.
  • The admin Forge Studio page no longer shows "(TODO T-518 / T-607)". The timers were armed on 2026-09-05, so it now says Auto-Forge is honoured by the scheduled forge timer.
  • The five live gaps #821 listed were checked against the estate on 2026-10-02. One is still open: moving the Static Web App deploy off its deployment token, now #834 (owner-gated, on the board). The other four were already closed, and their comments now say how. T-513 was explained as the CORS_* name collision. The expected-plan assertion runs in tfc-plan-check.yml on TFC_TOKEN. The three stale workspace variables were deleted on 2026-09-02. The HCWSite API registration exists and issues v2 tokens.

  • infra/ is touched in comments and one variable description only, so the hcw-azure plan carries no change beyond the permanent baseline.

  • NVIDIA is the backup for content features, and a weekly probe records its speed (#701). Owner decision 2026-09-29. Through the AI Engine's Test, NVIDIA's free trial tier took 56-117 s to answer "ok" with the drafting limits. Capped at 16 tokens and 45 s (#806), it did not answer at all, so the delay is the tier queueing. As first for content features, every synchronous call waited out NVIDIA's budget share (about 35 s, #807) before a paid provider answered.

  • PROVIDER_PLACEMENT_DEFAULTS.nvidia in functions/src/lib/ai/ai-config.js is now order for the eight content features: Gemini, OpenAI and Anthropic answer first, and NVIDIA serves when they cannot. Telegram was order already. The public explain features and alt text stay locked off.
  • A stored placement still wins. A feature an administrator set to First keeps it; the dropdown's order option now reads "In order — the backup", in the owner's words for the decision.
  • The Test's core is testProviderConnection in functions/src/lib/ai/proxy.js, with no HTTP or auth. The Test button and a new timer both call it.
  • The timer is probeAiProviders, Monday 06:15 UTC, in functions/src/functions/schedulers.js and functions/src/lib/timers/ai-provider-probe.js. It runs the Test on every provider with a key, one at a time, so NVIDIA's latency sits beside the paid providers', measured the same minute. It writes status, latencyMs, lastTested and lastTestError, as the Test does, plus lastTestedBy: 'probe'. A timeout is recorded, not thrown. The cost is the Test's, a fraction of a cent a week; its usage rows have their own source, ai-engine:probe ("AI Engine — weekly check"), and admin_test is now a registered source too ("AI Engine — Test").
  • It is flag-gated like every timer: PROBE_AI_PROVIDERS is in local.timer_catalogue (infra/functionapp.tf) and the enabled_timers validation, disarmed until the workspace lists it. scripts/assert-expected-plan.mjs declares both runs of FEATURE_FLAG_PROBE_AI_PROVIDERS, and verify-timer-witness.mjs records that it has no public witness.
  • The AI Engine card reads "Tested 3d ago by the weekly check". It now shows the Test's error message too: the card read lastError, while the Test has always written lastTestError, so a failed Test showed a red badge with no reason.
  • Tests: NVIDIA's own behaviour (request shape, <think> stripping, 429 retries, pacing, timeouts, key verdicts, zero-cost usage, the time budget) is tested with NVIDIA placed first explicitly, because behind three answering providers it would never be asked. New tests cover the defaults, the probe, its timer wiring and the card. route-inventory and timer-schedules-utc count twenty-three timers.
  • Required-Inputs' NVIDIA-API-KEY row describes the placement and the probe, and .azure/api-surface.json names the shared Test.
  • Vault auto-unseal is live on the lab host (#726). The owner accepted the trade on 2026-09-29: root on the host plus the Arc identity can now unseal Vault, and the five Shamir keys are recovery keys. The seal check passed (the Arc identity reads kv-labhybrid-prod-cus-01/keys/vault-seal), a cold copy was taken (/root/vault-before-726.tgz), and bootstrap.sh switched the seal. The owner ran vault operator unseal -migrate with three keys. systemctl restart vault then came back Seal Type azurekeyvault, Sealed false, with the journal reading unsealed with stored key and no keys typed. The grant reads back as one row: Key Vault Crypto Service Encryption User for the Arc machine, on that key alone.
  • ADR 0032's amendment of 2026-09-29 is marked accepted.
  • lab-host/README.md, docs/architecture/labs-host.md and Required-Inputs record the seal as on. The architecture record also records Vault as on since #729.
  • The Docker Hub docs say it is live. Three documents still described Docker Hub publishing as waiting on the owner:
  • docs/runbooks/docker-hub-publishing.md (its status box and "Follow-ups once it is live");
  • lab-image/README.md (the introduction and "From Docker Hub");
  • docs/standards/required-inputs.md (the DOCKERHUB_* rows and §4.11).

All three now record the connection created on 2026-09-29, the first run (36516945081), the matching digests and the verified provenance. The site pulls from Docker Hub (#795), and #779 is closed. - The agent sandbox recipe moved from /education/labs to the Docker hub, at /docker/sandboxes (#774). The "Run an agent against your landing zone" section (SandboxSection, #676) is now the body of a new page, "Run an agent in a sandbox" (pages/docker/SandboxesPage.jsx), under a heading that keeps its old id, agent-heading. The page says when its commands were last checked against Docker's documentation (29 September 2026). Before the move they were read again and still held: sbx run creates a sandbox whose name is new and reattaches with --name, -t/--template takes a container image, and --cloud sandboxes need a Docker Agentic Platform subscription, mount no host folder and expire after an hour by default. On /education/labs the agent slot stays with its heading, so an old /education/labs#agent-heading link still lands on it, and holds one line pointing to the new page. The Docker landing page's third focus area reads "Available now" instead of "Coming soon" and links there, as does the Docker tools page. The Docker hub's header gains "Sandboxes", in the place Terraform's has "Modules". sandboxes is a page type in routeFactory.ts, so the header recognises the static route as a Docker page. The page is in STANDALONE_ROUTES, so it pre-renders, and it is in validate-provider-pages.js. The issue keeps live-check.

  • The labs' "Run it locally" commands pull from Docker Hub. LAB_IMAGE in frontend/src/data/labs/catalogue.js is now hybridcloudworks/hcw-lab:latest. The image went live there on 2026-09-29 through Docker's OIDC connection (run 36516945081, #779):
  • its digest is the same one GHCR serves;
  • its SLSA provenance verifies under both registry names;
  • an anonymous registry token reads its manifest.

ghcr.io/hybridcloudworks/hcw-lab:latest stays a mirror of the same digest, and the lab image README lists both. - CI: every workflow runs on Ubuntu 26.04, named, not ubuntu-latest (#794). Owner rule 2026-09-26: the newest LTS OS. GitHub moves ubuntu-latest from 24.04 to 26.04 over 2026-10-19 to 2026-11-19 (actions/runner-images#14748), so for a month the same job could land on either image. All 36 runs-on lines in the 23 workflows now say ubuntu-26.04, which GitHub made generally available on 2026-09-17. Pinned rather than guarded: a check that failed while ubuntu-latest trailed the newest LTS could only be fixed by pinning, and during the rollout it would pass on one run and fail on the next. The pin is held by the version floors (#715). scripts/lib/version-pins-workflows.mjs reads each job's runs-on as an Ubuntu pin, and a label with no release (ubuntu-latest, ubuntu-slim), an expression or a runner group is a finding. scripts/version-floors.json gains kinds.ubuntu.platformCeilings.runners, the newest LTS GitHub-hosted runners offer as generally available, which every runs-on must equal. It is a ceiling beside the Ubuntu floor because GitHub ships a new LTS's runner image months after Canonical releases it (26.04: released 2026-04-23, preview 2026-06-11, generally available 2026-09-17). When 28.04 is released the floor moves and the lab host goes red at once; runs-on goes red only when update-version-floors.yml, which now also reads the "Available Images" table of the runner-images README, finds 28.04 there without a preview or beta badge. An unreadable README leaves the ceiling where it is, with a warning, as an unreadable Learn table does for the functions ceiling. Tests cover the reader (release labels, quoted, -arm and in flow lists; ubuntu-latest, ubuntu-slim, expressions and groups refused; other systems skipped), the rule (a label behind the ceiling and one ahead of it both refused; runs-on green on the ceiling while the floor is ahead), a gate that every runs-on line in every tracked workflow was read, and the updater (holds while GitHub has not made the new LTS generally available, moves when it has, never past the floor, never to an interim release, never down, soft on an unreadable README). .github/actionlint.yaml declares ubuntu-26.04, because actionlint 1.7.12, the newest release, predates it (rhysd/actionlint#743 adds it), and a test fails a declared label no workflow uses. deploy-azure-frontend.yml's canonical-link check reads find -print0 rather than looping over $(find …) (shellcheck SC2044, already on main), checked against a fixture path containing a space. What 26.04 changes for the tools the workflows use: preinstalled Node.js 22.23.2 → 24.21.0, which only publish-lab-image.yml's sandbox check and the four scheduled pull-request commit jobs run on (built-in modules only); system Python 3.12.3 → 3.14.4, which no job uses (each sets up its own); Docker 28.0.4 → 29.4.2 and Compose 2.38.2 → 5.1.3; jq 1.7 → 1.8.1; and coreutils now Ubuntu's Rust coreutils (uutils). Azure CLI 2.90.0, GitHub CLI 2.101.0, PowerShell 7.6.6 with Pester 5.9.0, Ansible 2.21.4 and Docker Buildx 0.37.1 are unchanged, and Playwright 1.63.0 lists ubuntu26.04-x64 for install --with-deps. Miniconda, Julia, Swift, Mercurial, Pulumi and Java 8 are gone from the image; no workflow used them.

  • Validate on the lab: a report for learners, not the job log (owner request 2026-09-28). The owner's first public run from /tools/landing-zone ended Success! The configuration is valid., but the page printed the raw job log under it: the image pull with its registry, digests and layers, the capability's rewrote module … -> ../../../opt/avm/… block, terraform init listing those paths again, and every provider ending (unauthenticated). GET public/labs/job no longer answers output; it answers report, built by the new functions/src/lib/labs/visitor-report.js: the verdict first (valid, invalid or error, with a plain-words headline), Terraform's own error text for an invalid configuration with the learner's file names and line numbers kept (a path into the lab's copy of a module is cut back to <module>@<version>/<file>), Modules used as name@version from the rewrite lines with one line saying the lab uses offline copies of the Azure Verified Modules so validation needs no internet, and Providers as namespace/name vX.Y.Z. The log is read by allow-list, so a line nobody has seen yet is dropped rather than shown, and a Terraform error that still names the runner after scrubbing is dropped whole and the headline says some detail could not be shown. The raw output stays on the job document, so the admin Jobs view is unchanged. The builder renders the report and never an output field, even when one is present, and reads the report defensively (a known verdict, entries shaped as modules and providers, no runner text). Tests: visitor-report.test.js over three fixtures made from the owner's log (the passing run, the same run with Terraform errors injected, and a job that failed before Terraform ran), plus noise injected at every line; public-visitor-copy.test.js drives those logs through the job route and fails on ghcr.io, sha256:, docker, Pulling, /opt/, ../../../, hcw-lab-runner, image, registry, the host's names or (unauthenticated), with its own token list; public-copy.test.js gains a labRunner term; and LzLabValidate.test.jsx renders what the server's builder makes of the same logs, so the frontend row of CI also runs when those two functions paths change. Also: the owner seeded TURNSTILE-SECRET-KEY on 2026-09-28 and it resolves, so TURNSTILE_SECRET_KEY is off EXPECTED_UNRESOLVED in scripts/check-unresolved-secrets.mjs (an unresolved one now pages like any other), and Required-Inputs marks it SET.

  • Public pages speak to visitors only: no vendor names, no admin-side messages (owner direction 2026-09-28). The owner's words: "Scrub all user-facing pages; remove references to backend workings. The user only knows the front end and doesn't know about tools we use (i.e. Cloudflare, etc.), or when you mention items that the admin needs to resolve." On the Landing Zone Builder the AI explanation's label now reads "AI-generated summary. Check the module documentation before relying on it." (the pricing comparison's, "…Check the sources…"), and the model name, time and cache note under it are gone; POST public/cloud-tools/explain no longer returns model (the stored document and a log line keep it), and every refusal carries a code (EXPLAIN_UNAVAILABLE, EXPLAIN_RATE_LIMITED, EXPLAIN_PAUSED_FOR_TODAY, EXPLAIN_EMPTY, EXPLAIN_FAILED) with a visitor sentence. "Validate on the lab" says "Validation on the lab isn't available right now. You can still download the files and validate locally." for every closed door, "The lab is busy right now…" for a full queue, and words the browser check, the limits and every refusal without naming Cloudflare or a setting; the server's DOOR_REASONS, LOCK_REASONS and the new LIMIT_REASONS say the same, the machine codes (TURNSTILE_NOT_CONFIGURED and the rest) stay for logs and the admin pages, and the frontend maps each code to its own words and never renders reason or error, nor the code (data-door is now closed/busy/open). The labs page drops the host's vendor, the API host, "not provisioned", "endpoint is not published" and the issue numbers of unbuilt sections; the pricing, price-changes and labs status cards say a read failed without the error's text; the submission pages lose the "View Queue" link into /admin and the admin-queue wording, and submitPublicContent throws only visitor sentences; the frameworks, code and coder-corner pages stop naming ContentForge and Framework Studio; and robots.txt stops naming the host. Two tests keep it that way: frontend/src/public-copy.test.js parses every public page, component and hook (admin excluded) and the static files, and fails on a vendor or admin-to-do term outside a per-file allow-list of subject matter and the legal pages; functions/src/lib/public-visitor-copy.test.js does the same for the visitor-facing strings of every module behind a public/* route and drives the lab and explain refusals through the handlers. CI's functions row now also runs when the three frontend files it reads change.

  • Only the site can frame the lab; a direct visit goes to /education/labs (owner decision 2026-09-28). The owner's words: "The lab should only be accessible through 'panes' from my site, lock to that." A new lab_panes_only snippet in the caddy role's Caddyfile is imported into the site block ahead of every route. It covers lab.hybridcloudworks.com, every *.lab name, coder.lab and *.coder.lab, and every response, including the 404 and any route added later. Each response carries Content-Security-Policy: frame-ancestors 'self' https://hybridcloudworks.com https://www.hybridcloudworks.com, with no X-Frame-Options. www is listed because it serves the site with a 200, and 'self' is there for code-server's same-origin webviews. The named matcher @lab_direct_visit sends a top-level browser visit to https://hybridcloudworks.com/education/labs with a 302. That is Sec-Fetch-Dest: document, or Sec-Fetch-Mode: navigate with no destination. Panes (iframe), fetch, XHR and WebSockets (empty), and subresources pass. So do clients that send no fetch metadata: Coder's agents and CLI, curl, and the Function App's status proxy, whose Node fetch sends Sec-Fetch-Mode: cors alone. The Coder route (10-coder.caddy.j2) removes Coder's default frame-ancestors 'self', which would otherwise keep Coder out of the panes. It also lets /api/v2/users/oauth2/github/callback through at the top level, because GitHub cannot be framed. That path only redirects, and the redirect that ends sign-in lands on the labs page. The values are caddy_frame_ancestors and caddy_direct_visit_redirect in group_vars/all.yml. The Coder owner steps in lab-host/README.md now check from the CLI or in a pane, and coder login takes a token. The decision is recorded in docs/architecture/labs-host.md, "Panes only", with a one-line pointer in ADR 0032 decision 4. It takes effect on the host with the next bootstrap.sh run.

  • Lab jobs on the newest images: shell-echo on Alpine 3.24, ansible-check on hcw-lab-runner; every job image held to a floor (#747, #715). The first real lab job (2026-09-28, shell-echo on vps-hostinger-01) pulled alpine:3.20, past its end of life on 2026-04-01. vps-agent/lib/capabilities.js now pins alpine:3.24.2 by its index digest sha256:294b683c..., the value the registry's Docker-Content-Digest header, docker buildx imagetools inspect and the Docker Hub API all reported on 2026-09-28 (3.24.2 was endoflife.date's newest release, 3.24 supported to 2028-06-01).

ansible-check moves off alpine/ansible:2.17.0, a third-party image on ansible-core 2.17 and Python 3.12 that never ran a job under the sandbox: its HOME is /, so ansible-core stopped at Unable to create local directories(/.ansible/tmp): [Errno 30] Read-only file system before reading the playbook. It runs the same argv on the pinned hcw-lab-runner (ansible-core 2.21.4 on Python 3.14.7), with the job tmpfs for ANSIBLE_LOCAL_TEMP and ANSIBLE_HOME=/tmp/run/ansible-home so no warning about ~/.ansible reaches the output. The runner carries ansible-core and no collections, so a module outside ansible.builtin is refused with couldn't resolve module/action, as the job type's description now says. IMAGES.ansible is removed, and capabilities.test.js fails an IMAGES entry no capability uses.

The floors check now reads IMAGES. scripts/version-floors.json gains an alpine kind (endoflife.date alpine-linux, newest line, N-2 patches: 3.24.2, floor 3.24.0), which the weekly updater moves like Python's; a digest-pinned Alpine reference must name its exact release, because a digest does not float. An IMAGES entry that no kind governs, that the repository does not build (hcw-lab-runner is judged through lab-image/Dockerfile) and that no unsourced entry names fails version-floors.test.mjs by name; against main's map it reports 3.20 is below the floor 3.24.0 and alpine/ansible has no floor. lab-image/sandbox-check.mjs now runs every capability, with a case per capability enforced: shell-echo on its pinned alpine digest, and ansible-check on three new fixtures under lab-image/smoke/ansible-check-payload/ (a valid playbook, an unknown play keyword, a collection module). The admin Labs Settings tab no longer tells the owner to pre-pull alpine:3.20, hashicorp/terraform:1.9 and alpine/ansible:2.17.0.

  • The lab agent and the Coder template pull the image that vendors every builder module; lab jobs get 512 MB; the lab articles match (follow-up to #742). vps-agent/lib/capabilities.js pins hcw-lab-runner to sha256:8cc69354... and lab-host/coder/templates/hcw-lab/main.tf pins hcw-lab to sha256:014a81d6..., the digests main's publish run 36320771780 pushed for 7c0a95b2 (#742), both checked with docker buildx imagetools inspect. Those images carry avm-res-network-virtualnetwork@0.22.2, the version every spoke calls, so the Landing Zone Builder's full default build validates offline.

Job memory moves from 256m to 512m (owner decision 2026-09-27) in the agent's default (vps-agent/index.js), vps-agent/.env.example and the labs_agent role's labs_agent_job_memory and argument spec. Measured on Docker 29.8 against the pinned runner, under the agent's own sandbox argv: at 256m the full default build's terraform validate is OOM-killed (exit 247) after an offline init that succeeds; 320m passes, but in 20 to 86 s across four runs, and 512m in 15 to 19 s every time. One job runs at a time on a 16 GB host, so the margin is cheap; the reason is recorded beside the default. sandbox-check.mjs loses its BUILDER_LIMITS override and reads its limits from vps-agent/index.js, so every job, the builder's included, runs at the agent's own limits; it passes all six on the pinned digest. publish-lab-image.yml now runs on pull requests that change vps-agent/index.js for that reason. A new scripts/lab-job-limits.test.mjs fails when the agent's defaults, .env.example, the role defaults and the role's argument spec disagree about any job limit; the scripts (operations) CI filter now covers vps-agent/index.js, .env.example and lib/capabilities.js, which lab-job-types.test.mjs already imported.

The lab article drafts no longer say a build with landing zones cannot init offline. Part 1 keeps its platform-only build, with order as the reason and a pointer to the full build. Part 2 lists every builder module as vendored, shows the full default build's spoke rewrite and its 71 module lines with none downloaded, and turns the left module "spoke_corp_1" entry into what it now means: an image published before 2026-09-27. Every quoted line was re-run against the new images.

  • Coder's PostgreSQL moves from 16.15 to 18.6, with the move rehearsed (#715). 18.6 is the newest release of the newest major (18, supported to November 2030; read 2026-09-26). It is pinned by index digest sha256:5a5a84b1...2da9722, which postgres:18 and postgres:18.6 both resolve to, and which the registry's Docker-Content-Digest and the Docker Hub API confirm. The 18 image keeps its cluster in /var/lib/postgresql/18/docker under a VOLUME at /var/lib/postgresql, and refuses to start with a mount at the old /var/lib/postgresql/data, so the Compose volume now mounts at /var/lib/postgresql. The host holds no Coder data, so nothing is migrated. Rehearsed on Docker 29.8 with the committed Compose file and the pinned Coder v2.37.3. Coder applied its migrations to 585, the newest v2.37.3 ships, and /healthz answered 200. An owner account was created and dumped with the role's backup script, rendered from its template. The dump was restored into a fresh 18.6 with the README's drop, create and load, and a second dump matched it line for line. Coder then started on the restored database, with /api/v2/buildinfo and /api/v2/users/first answering 200. The coder role README's "stay on 16" paragraph is replaced by that record and a procedure for the next major: dump, set the old cluster aside inside the volume, bump, restore. Its two host commands were run against the Compose file: the 18 image refuses to initialise beside another major's cluster, and initialises a fresh cluster once that one is set aside. The backup and restore commands need no path, so they are unchanged. Where the database runs is now recorded as an owner decision (2026-09-26), in the role README's "Where it runs" and as a one-line amendment to ADR 0032's decision 2: it stays this container on the host. The two options not taken were Coder's built-in PostgreSQL (binaries fetched from Maven at start, outside the host's digest pins) and Azure Database for PostgreSQL Flexible Server B1ms ($0.01921 an hour in Central US, about $14 a month in compute, plus storage and internet latency from the VPS).

  • The hcw-lab image runs Python 3.14.7 on Debian 13 trixie, with every tool at its newest release (#714, #715). Owner rule 2026-09-26: every runtime, OS and base image the repository chooses is on its newest supported release. Every stage of lab-image/Dockerfile moves from debian:bookworm-slim (Debian 12, Python 3.11) to the official python:3.14.7-slim-trixie, pinned by index digest; versions.env records why one base serves every stage (the python image is debian:trixie-slim plus CPython, and vendor-avm.sh then runs on the same Python the runner ships). ansible-core moves from 2.19.13 to 2.21.4, the newest line, whose controller range is Python 3.12 to 3.14; its dependencies are no longer Debian packages built for another interpreter but eight wheels pinned in ANSIBLE_CORE_DEPS and installed in pip's hash-checking mode, which also fails the build on any dependency not listed. No Debian python3 package and no pip command remain in the image. The Azure CLI stays at 2.90.0, the newest release, now written as AZURE_CLI_VERSION plus AZURE_CLI_SUITE=bookworm: packages.microsoft.com has no trixie suite and Microsoft's install page says to use the latest Debian suite in that case, and the apt source is the deb822 form that page documents. Terraform 1.16.4, kubeconform 0.8.0, helm 4.3.0, kubectl 1.37.1 and the mirrored providers (azurerm 5.7.0 and 4.81.0, azapi 2.12.0, alz 0.22.0, random 3.9.1, modtm 0.4.0, time 0.14.2) were already the newest releases, each sum re-checked against its publisher's file. The sandbox template's Azure CLI comment now says what it installs, the -1~resolute build from its Ubuntu 26.04 base's own suite, and records why the recipe adds no Python. It also gains HEALTHCHECK NONE, as the lab image has: the base defines no healthcheck, so nothing changes at run time, and it clears the checkov CKV_DOCKER_2 / trivy DS-0026 finding that Qlty reported as blocking once the file was touched. The publish workflow's digest guard, which grepped ^FROM debian:, now checks every FROM in both Dockerfiles: an earlier stage, or exactly the image@digest versions.env records for that file (BASE_IMAGE/ BASE_DIGEST, and the new SANDBOX_BASE_IMAGE/SANDBOX_BASE_DIGEST), and fails on anything else. smoke.sh now also compares python3 --version with BASE_PYTHON_VERSION. Measured with Docker 29.8.0: runner 1.70 GB / 319 MB to 1.71 GB / 319 MB, full 2.69 GB / 491 MB to 2.73 GB / 497 MB (on disk / compressed), sandbox template unchanged at 3.38 GB / 797 MB. Both smoke tests and sandbox-check.mjs pass on the new images. The pins in vps-agent/lib/capabilities.js and the Coder template move in a follow-up, from the digests main's publish run reports.

  • The six radarlint-python findings left on main are fixed, not silenced. #588 counted nine on 2026-09-14 and closed with them outstanding; three had gone with later changes, and qlty check --all --filter radarlint-python on current main reported six, all in files no suite exercises for style. Each is a small extraction, and each keeps the behaviour a test or CI step pins: tooling/workflow.py gains apply_node_policy (the required-or-skipped decision init made inline) and score_node plus overall_status (the per-node scoring evaluate_workflow did in one loop), which brings both functions under the cognitive-complexity limit while the CI round trip — smoke closes empty, dirty refuses then closes abandoned, the handoff validator accepts real work and rejects a wrong agent_id and missing evidence — passes unchanged. scripts/docs/check_redaction.py's main splits into line_findings and scan with the same output, and the gate still passes over the same 72 files. scripts/docs/hooks.py names its docs/ prefix once, and its link rewriter computes the target in one place and returns once — the six-return function Qlty flagged on this very change, fixed in the same pull request rather than carried. scripts/complexity/report.py reads the delta arrow from a direction function instead of a nested conditional. And the Stop guard test's audit-log reader is a module function, which is what the rule wanted from a TestCase method that is not a test. Qlty now reports no issues for the plugin; ruff, compileall, the 13 hook tests and the strict docs build are all green on the result.

  • TODO.md indexes the one open issue, not the nine it had outlived. Its "Where the open items live" table still pointed at #433, #434, #435, #436, #442, #349, #358 and #454, every one closed since 2026-09-09, beside the one row that was still true. Owner decision 2026-09-18: #432 is the only issue that stays open — the audio pipeline's three remaining secrets wait on the ElevenLabs and RSS.com Max purchases — and the five others open that day (#501, #567, #588, #611, #613) were closed with a comment on each recording what was outstanding. The table now carries #432 alone and says where the residual went. Nothing else in the file changed: the accepted risks and the handling rules stand as written.

  • The deployment runbook names the directory the estate lives in, and the az login --tenant line that reaches it. On 2026-09-18 a plain az login from a fresh Mac landed in the owner's home tenant, hybridcloudworks.com, which holds no subscriptions, and closed with The subscription of 'b9e02281…' doesn't exist in cloud 'AzureCloud' — a line that reads as a missing subscription and is a wrong-directory sign-in. The four subscriptions live in the tenant named Default Directory, where the work account is a guest behind MFA, so only an explicit --tenant reaches them. The runbook now says so, gives the sign-in line by the tenant's onmicrosoft.com domain rather than its GUID (the docs redaction gate keeps real GUIDs out of published pages), and shows how to read the GUID back from the session when a variable needs it.

  • The frontend suite runs on vitest's vmThreads pool, and three test files stopped depending on which pool they get (#645). Building a jsdom per test file — 172 of them — was half the suite's tracked time. One V8 context per worker, reused, takes npm test from 84.6 s to 25.0 s and npm run test:coverage from 100 s to 57 s, on the same 4-core machine. CI pays that twice, in the frontend job and in coverage (qlty).

Not isolate: false, which vitest suggests in the same breath. The two are not the same trade. Two test files — one mutating a shared module object and setting a global, the other reporting what it inherited — showed that under isolate: false, once both land in the same worker, the second sees the mutation and the global. Module state crosses file boundaries there, which is exactly the bug #640 was filed as and could not reproduce: passing alone, failing in a full run. It would be nondeterministic too, since whether two files share a worker is a scheduling detail — with only two files they usually do not, so a casual check of isolate: false looks clean. vmThreads keeps per-file isolation; the same probe in the same worker saw a fresh module registry and no global.

Three files had to stop leaning on the old pool first, and each was wrong independently of it. scripts/audit-published-pages.test.js now declares the node environment it always needed — it tests a Node CLI module, touches no DOM, and was failing at collection, so its 15 tests silently did not run while the suite still reported green. setupTests.js states the compression streams a browser provides and jsdom does not implement, which the draw.io parser uses and which only worked before because Node's globals leak into a worker. And AuthCallbackPage reaches its one hard navigation through @/lib/hardNavigate, so its test uses a real URL instead of a fabricated Location — a fabrication that had already caused two bugs of its own, both recorded in that file.

Checked rather than assumed: 2,272 tests under both pools, and coverage identical to the same numerators and denominators. The two lcov.info files are not byte-identical — same 36,985 records, none missing from either, but 398 differ in hit count and none flips covered to uncovered or back, which is why the percentages match exactly and why Qlty, which gates on coverage status rather than on how often a covered line ran, reports what it did before. Nor is the pool the cause: two back-to-back runs of the same pool differ in 164 hit counts, also with no flips. Hit counts are run-to-run nondeterministic here whatever the pool. Peak Node RSS is 3,213 MB against forks' 1,154 MB across a full coverage run — 2.8×, the real cost of reusing a context per worker, and the number to watch as the suite grows. #645 had listed memory as unestablished.

Fixed

  • One Podbean publish per transcript, even when two schedules land at once (closes #1118). An approval (POST cms/podcast/transcripts/review) and a retry (POST cms/podcast/transcripts/{id}/publish) arriving in the same instant could both read the transcript with no publish pending, both queue a job, and two first runs would both create: the episode twice on the public feed. #1117 reconciled a create whose answer was lost; this is the other half, the reservation.
  • The pending marker is now etag-conditional. scheduleHostPublish writes host.podbean.pending with patchDoc's ifMatch set to the _etag the document was read at — after an approval, the one the status write returned — so it lands only on the version the decision was made about. A 412 means someone else wrote first. The document is re-read once: a live reservation on it is theirs, and the loser answers in flight naming their job (202 from review, 409 from the retry) and cancels the job document it wrote, so the stale-queued sweeper can never re-enqueue it. Anything else — the other job already finished, a withdrawal, an unrelated edit — is not a publish in flight, so the reservation is tried once more against the fresh document, from its stable record, so an episodeId just stored is kept under the marker. A document that moves on every attempt queues nothing and throws RESERVATION_CONTENDED; the review handler records SCHEDULE_FAILED and keeps the approval, as it already did for any other failure to queue. A transcript handed over without its _etag is refused before any write. Nothing is enqueued unless the reservation landed.

  • Tests. Two schedules racing on one document produce one job, one marker and one message, the loser in flight; an approval and a retry racing through the handlers agree on one job id; the fresh-document retry keeps the stored episode id; a reservation lost twice cancels its job and queues nothing; a document without an etag writes nothing.

  • Listen & Learn no longer renders dead players for audio the site stopped serving (#1119). Every public Listen & Learn row (the AB-100 set) still pointed at the pre-Azure Firebase bucket, and every file there answered 404 on 2026-10-13, so the audio page showed three players that played nothing and the browser's policy refused the host besides.

  • servableListenAndLearnAudioUrl treats storage.googleapis.com and firebasestorage.googleapis.com as retired: the provider-wide list and the section counts omit such a row, as they omit one with no audio, and the per-certification read blanks its audioUrl so the page shows the chapter without a player.

  • The rows themselves are kept. Regenerating a chapter (Admin → Listen & Learn → Regenerate) writes new audio to blob storage and the row returns.

  • The podcast pages show the show again, now that it is back on Podbean. The public list returned one stale RSS.com upload instead of the three episodes in the feed. Two causes, both from the move on 2026-10-13:

  • The hide rule. RETIRED_PODCAST_MEDIA_HOSTS hid every episode on mcdn.podbean.com (#372). That was right when the show left Podbean on 2026-09-05 and its files 404'd, and wrong once the show returned: every current episode is on that host again. Each host the show has left now carries the moment it left. A row there is hidden only when nothing has refreshed it since. fetchPodcastFeeds upserts every episode in the current feed every two hours, so current episodes show. Rows from Podbean's first tenure, and the RSS.com upload no feed carries, stay hidden. A host the show still uses is never hidden by this rule, so a stalled timer cannot empty the page. The section counts project updatedAt so the sitemap counts the same rows.

  • media-src. Once Podbean had copied the audio, an enclosure went mcdn.podbean.com → 302 → a numbered storage host (s307, s396, s405.podbean.com, one per file). #1107's mcdn.podbean.com alone failed at that hop in headless Chromium, and https://*.podbean.com loads it. RSS.com's three hosts left the list, since no listed episode is on them.

  • Show episodes play on the site again, from RSS.com and from Podbean. The site's media-src allowed RSS.com's two media hosts (#1101), but an RSS.com enclosure redirects through rsscom.pdn.tritondigital.com, RSS.com's download measurement, before the file. Chromium checks media-src on every redirect hop, so every show episode failed at that hop (MEDIA_ERR_SRC_NOT_SUPPORTED), measured in headless Chromium against the live policy on 2026-10-13.

  • The show moved to Podbean the same day: RSS.com's feed now answers 301 with the Podbean feed, all three episodes keep their GUIDs, and the site's ingest follows the redirect. Podbean's enclosures start at mcdn.podbean.com and, until Podbean has copied the audio, redirect back through both RSS.com hosts and the measurement host.
  • media-src adds both hosts, and nothing broader. With them, an RSS.com episode and a Podbean episode each load in Chromium (611 s).

  • csp.test.js pins the list and says to trace an enclosure's hops with curl -sIL before changing it.

  • The read-only Azure identity writes nothing now: the content manifest's origin window has an identity of its own (#1098). github_reader held HCW Function Settings Reader and, for publish-content-manifest.yml's per-run origin window, HCW Function Config Refresh. RBAC is additive, so that one principal could list the Function App's app settings and write them, from any step of the hourly monitor, the secrets monitor or the manifest job on main. infra/oidc.tf said it "can change nothing".

  • github_manifest_window holds Config Refresh on func-site-prod-cus-01 and Reader on rg-web-site-prod-cus, and nothing else, with main-branch sign-in in both subject forms. Reader sits at the group, the reader's own proven shape, so the subscription is discoverable at sign-in; it lists nothing. The manifest job signs in with MANIFEST_WINDOW_CLIENT_ID and skips until it is set.
  • github_reader_origin_window is removed, so the reader holds Reader and Settings Reader on rg-web-site-prod-cus only. The comments that said it could change nothing are true now, and scripts/oidc-subjects.test.mjs fails if either identity gains a grant.
  • scripts/set-github-variables.ps1 seeds the new variable, and ALERT_READER_CLIENT_ID, which #1053 left out of it. The plan checker declares the five creates and the one delete.

  • The lab-host bootstrap tests made their temp folders in the suite body, which Vitest runs even for a skipped suite, while it never runs a skipped suite's afterAll. Every Windows run left a folder behind. They are made in beforeAll now.

  • Cosmos restore: the first drill passed, and the runbook's step 5 now works as written (2026-10-13). The owner restored the latest backups (full/2026-10-11 + 1 delta, 11,971 documents) into a scratch account, and every one of the 62 containers matched the backups document for document. The site's local API served 24 content items and 6 podcasts from it.

  • RTO 1 h 39 min against 8 h; RPO 22 h against 24 h. That time includes finding and fixing three restore-script bugs (#1102).
  • The scratch account, the firewall rule and the read role are gone. The row is in the Drills table, so the daily Check drill age run passes; the next drill is due by 2027-01-21.
  • Step 5 needed two things the page did not have. First, a storage emulator: without Azurite the local host fails every timer listener and exits. Second, a Node count instead of Invoke-RestMethod: content documents carry keys that differ only in case (Title/title), which PowerShell's JSON reader cannot hold, so its count printed 0 against a correct answer. Both are now in the step, with Azurite's clean-up in step 6.

  • check-drill-age.test.mjs read the runbook's table as of a fixed 2026-10-07, which turned the first real row into a date from the future. It now reads it as of today.

  • Cosmos restore: a restore with a daily backup in it completes (found by the first drill, 2026-10-12). The restore script asked every backup layer for every container in the Sunday full. An operational container such as admin_audit_logs is in the full and in no daily delta, by design, so the first delta stopped every restore as "corrupted". No restore that included a delta, which is every day but Sunday, had ever completed.

  • A layer now contributes only the containers its own manifest lists, and a listed container with no file still stops the restore.
  • Against the live backups the dry run reaches the end: 11,971 documents across 62 containers.
  • Each layer's manifest is checked before anything is written: it must name its own run, and list its containers by name. A missing or malformed list stops the restore instead of quietly dropping that day's changes.
  • --verify expects each container to hold what it held before the run plus what the run created, and counts an empty container as 0. The drill's first report flagged 18 false mismatches: 17 empty containers, and one an aborted first attempt had already filled. An independent check found all 62 containers in the test database exactly matching the backups.
  • The script refuses a target whose account name has -prod- in it unless --allow-production-target is given, because a restore writes backup copies over whatever the target holds.

  • The runbook installs func before step 5, takes the container count from the spec instead of a stale 73, explains Azure's Enqueued/Dequeued/Succeeded lines, and deletes the local settings file at teardown.

  • Lab host: every bootstrap publishes every Coder template, with no token (2026-10-12). The coder role's push published hcw-lab alone, with the host's self-renewing rotation credential. amo-guided-lab and na-guided-lab (#1090) could only be published with the owner's token, so each change to them was a manual chore, which the owner had already ruled out on 2026-10-09.

  • hcw-coder-automation push-template publishes every directory under lab-host/coder/templates that holds a *.tf file, hcw-lab first, each by name and each tracked on its own digest. One failing does not stop the next, and the next run retries only the one that failed.
  • hcw-lab keeps the state fields the daily report and the site's drift check read. A state written before this change still counts it as published.

  • The lab-image digest pull request told the owner to fetch a token and push the template after the bootstrap. It now says the run publishes them, and a test keeps the token instructions out and checks the task it names still exists. The runbook's "After a merge" and lab-host/README.md now say the same, with the manual helper kept as the fallback for a host without a working credential.

  • Lab host: bootstrap.sh runs one at a time (#1069, part 2). Two concurrent runs were not serialised across the firewall rebuild's guard-to-unguard sequence, or across two checkouts of /opt/hcw-src.

  • The script takes a flock on fd 9 straight after its root check. A second run waits up to 30 minutes, then refuses having changed nothing.
  • Every pass proves it holds the lock rather than trusting a marker. A pass handed over by exec reuses its inherited fd 9 only when that is the lock file itself. A handover from an older bootstrap.sh, or a hand-set HCW_BOOTSTRAP_HANDED_OVER, takes the lock like any other run.
  • The play inherits the lock, so it holds until every mutator has ended, even if the script is killed first.
  • The lock file is /run/hcw-bootstrap.lock, where only root can create files. It is created 0600 and refused if another user owns it, so no local user can open it and hold the lock. It is opened without truncating, a symlink in its place is refused, and it is never removed. The wait must be a whole number of seconds.

  • Verifying add-on image attestations before the pull (part 1) waits on an owner decision on #1069.

  • Reminders: an editor's save keeps the credential register's rows, and their stamps, beside a malformed owner row (#1041). The save validated the stored sheet as a whole, so one owner row that did not validate (an out-of-band edit) made it keep no register rows at all. Their notified stamps went with them, and the next sendReminders run said a stage again. Each stored row is now normalised on its own: an invalid row is left out, the valid register rows are kept as stored, and a second row under an id already kept is dropped.

  • Forge Studio: "Send to In Review" sends on the first click after the document loads (#1072). useForgeSession's operations read the session through a ref that moved in a passive effect, after paint. A document that arrived from a read, not a click, was on screen while the ref still held the session without it. A click in that gap made Send to review send nothing, and a Save could have sent the previous ETag. The coverage run, being slower, hit the gap, and the page test flaked there. The ref now moves in a layout effect, inside the commit that renders the state. A hook test calls Send to review in the commit that first renders the document: it resolved null before and posts with the ETag now.

  • Lab agent: a job's result is written only while the agent still holds the job (#1057). completeLabJob read the job, checked that the reporting agent held it, then wrote the result with an unfenced patch. Three things can happen to a job in between: an operator cancels it, the canary withdraws it as overdue, or another agent claims it once the 15-minute lease lapses. When one did, the stale agent's result overwrote it. The write is now replaceDocIfMatch under the ETag of the read that found the job held. On a 412 the job is read again and the same rules decide:

  • still this agent's and claimed: the write is retried, at most three attempts in all;
  • anything else: the agent gets 403, 404 or 409, and the job is left as it now is.

The agent already treats a refused completion as an error it logs, so it needs no change.

  • Learn catalogues: the weekly updater passes its own check again, and a re-read can land before the vendor's day (#1091). Two fixes:
  • The weekly updater went red on 2026-10-12, its first run after the 2026-10-05 re-read. That re-read's test pinned the Azure DATA_AS_OF to exactly 2026-10-05, and update-learn-catalogue.yml stamps DATA_AS_OF on every run, so the refresh could never pass. The test now holds 2026-10-05 as a floor for all three catalogues.
  • #1091 asked for AB-400 (available 2026-10-16) and PL-400 (last day 2026-10-30) to be re-read before their days, but a stored status could not move early without the badge moving with it: active showed AB-400 bookable four days before Microsoft opens it. deriveStatus gains rule 2. A row whose new statusReadDate is before its day keeps its current badge until that day: stored retired shows "Retiring" through its expiryDate, and stored active shows "Coming" until its availableDate. The alarm then has nothing to fire on, and the due-soon issue says how to store a re-read this way.
  • Only a row that says when it was read is held. A row stored after its day stays as stored on every clock. AZ-800, retired once 2026-09-30 had passed, stays retired even on the earliest DATA_AS_OF the education index pre-renders with. A first version without the date showed it bookable there.

  • AB-400 is stored active and PL-400 retired, both with statusReadDate 2026-10-12. Microsoft's exam pages, the AB-400 study guide and both dates were unchanged that day, as the Azure catalogue's header quotes.

  • Admin → Integrations → Credentials shows the rotation date you recorded (#1026). On 2026-10-10 and 2026-10-11 the owner recorded the rotation dates of the eight credentials with a rotation rule, twice, and the tab still read as though none had been saved. Every save had landed: the API's request log holds 24 PUT cms/credentials across those two days, all 200. What the tab never did was print the stored date:

  • The Record form, labelled "Rotated on", always opened on today's date, so reopening it to check looked like the save had been lost.
  • The line under a row's age named only where the age came from ("recorded", "Keys tab"). Where a Key Vault write was newer than the recorded rotation, the age came from that write, and the recorded date appeared nowhere.

Now the form opens on the recorded date, and on today only when nothing is recorded. The line under the age names the recorded date whenever one is stored: recorded 2026-08-20, or Keys tab; recorded 2026-08-20 when the Keys tab's write is newer. Tests for both are in credentialsView.test.js and IntegrationsCredentials.test.jsx. - COPILOT_REVIEW_APP_PRIVATE_KEY is recorded as stored (#1063). The owner stored it in the Agents store on 2026-10-11, and gh secret list --app agents lists it. The required-inputs row now reads SET, and the Copilot code review runbook says how to confirm it. A manual dispatch of the setup workflow cannot confirm it, because that run reads Actions secrets rather than the Agents store, so it prints the "not resolvable" notice either way. The first Copilot session after 2026-10-31 is the check.

  • Lab host: the add-ons' egress cleanup works on an empty chain and on its own rules (ADR 0035). The bootstrap after #1059 stopped at "Remove the egress rules this role wrote on an earlier run". The inline pipeline iptables -S DOCKER-USER | grep -F … | sed … | while read rule; do iptables $rule ran under pipefail, and on the host's first run the chain held none of the role's rules, so grep exited 1 and that became the task's failure. Reproduced in Ubuntu 26.04 with iptables 1.8.11 (nf_tables), the host's own versions. The same reproduction showed the next failure waiting: with a rule present, iptables -S prints the comment in quotes, the unquoted $rule split it, iptables answered "Bad argument `egress'", and the loop still printed "removed" and exited 0. Every later run would have deleted nothing while saying it had. It also deleted the role's reject before inserting it again, so the add-ons' subnet ran unfiltered for the length of the rebuild, or for good if the run stopped there (CodeRabbit on #1060).

  • lab-host/ansible/roles/addons/files/hcw-addons-egress-clear, installed as /usr/local/libexec/hcw-addons-egress-clear:

    • guard, in one iptables-restore transaction, puts a reject for the whole subnet at the top of DOCKER-USER (commented … rebuilding) and deletes every rule whose comment is exactly the role's, by its full specification and never by position, so another writer's rule cannot be hit.
    • The role inserts its rules above that reject.
    • unguard lifts it after the last insert.

    A run that stops in between leaves the reject, so the add-ons lose egress rather than keep it unfiltered. Its arguments are held to the shapes they should have before they reach the transaction. A missing chain fails with iptables' own message, instead of being hidden by 2>/dev/null. - tests/hcw-addons-egress-clear.test.sh runs it as root against real iptables, in a scratch chain (17 checks): - an empty chain; - the role's rules among another comment, a comment containing the role's, and Docker's RETURN; - a guard run again after a stop; - a held reject for another subnet, which does not stand in for this one (guard checks its own with iptables -C); - the rebuild under the reject; - several held rejects; - a missing chain; - input that could break out of the transaction.

    CI runs it in the ansible-lint (lab-host) job.

  • Lab host: an add-on with no digest yet no longer stops the bootstrap (ADR 0035). The migration row is enabled with an empty image_digest, which by design deploys nothing and only says so, "because the other roles must still converge". But the addons role asked every enabled row for its vault keys, so on 2026-10-10 the run stopped at "Refuse to start an add-on whose vault key is not set" for a container it would never have started. portainer, vault, the privilege checks and the applied-commit record never ran.

  • lab-host/ansible/roles/addons/tasks/main.yml: the keys are collected from addons_deployed (enabled, with a digest), the same set the test-key check and the container tasks use. A row without a digest gets the "not deployed" note, and its key is asked for once it has a digest.

  • The playbook comment, the role's argument spec and lab-host/README.md say the gate is on a row being deployed.

  • The role-change alerts are a rule per resource type, which Azure accepts (PLAT-4, #964). The apply that created the other PLAT-4 rules refused alert-rbac-write-app and alert-rbac-write-mgmt with UnsupportedCondition: "Alert Rule with category 'Administrative' and subscription scope must have at least one additional filtering condition." azurerm sends resource_types and statuses as any-of groups, and Azure counts only a single-value condition as that filter.

  • infra/observability.tf: each rule became two, one per resource_type (a single equals, the shape of Microsoft's Administrative sample), still Succeeded and Failed only: alert-rbac-assignment-app, alert-rbac-definition-app, alert-rbac-assignment-mgmt and alert-rbac-definition-mgmt. The role-definition rules never fire on an apply, since infra/ declares no role definition; they catch a custom role widened by hand.
  • scripts/assert-expected-plan.mjs declares the four creates in place of the two.

  • docs/runbooks/alerting-and-support.md: the rule table, the fire record, and a fire test for the role-definition rules (a throwaway custom role, named and recorded before it is created, then deleted by that name). ADR 0005's grants table names the four rules.

  • The alert verifier checks every declared rule, weekly, and the runbook stops calling proven delivery unproven (PLAT-4, #964). The estate assessment found verify-alert-state.yml dispatch-only, last run 2026-08-30, reading three hard-coded rules while infra/ declared thirteen across two subscriptions.

  • The rule list is derived, not written down. scripts/lib/alert-declarations.mjs reads every .tf file as one module, with a small HCL scanner (scripts/lib/hcl-blocks.mjs) that respects strings, heredocs and comments, and resolves each alert rule's Azure name, resource group, subscription (from the provider alias through the (sub-…) in its variable's description), count gate and action group. An alert type it cannot read back, a name it cannot resolve, or a rule whose provider and resource group sit in different subscriptions stops the check instead of dropping out of the inventory.
  • scripts/verify-alert-state.mjs reads the live rules through ARM GETs with one az account get-access-token, in every resource group the declarations name, and fails naming each rule MISSING, DISABLED, NOT WIRED to ag-plat-prod-cus-01, or in DRIFT on autoMitigate. It reads the action group too, and fails one that is disabled or has no enabled receiver, printing receiver names and never their addresses. A rule whose create assert-expected-plan.mjs declares is reported as waiting for an apply; a group the identity cannot read is NOT AUTHORIZED, never MISSING. Gated rules are expected per a GATES record of the workspace's gate values, which the tests require for every gate a rule uses.
  • The workflow runs every Monday at 07:53 UTC and on dispatch, still as github_reader. That identity holds no role in the Management subscription, so the capacity and lab rules and the action group report UNREADABLE until the owner grants a read on rg-mgmt-plat-prod-cus: listed in every run and raised as a warning, their state called unknown, while the run passes or fails on what it can read. A weekly red run for a gap everyone knows is how validate-deployed.yml came to be ignored. A refusal anywhere else still fails, and once the workflow can read a recorded subscription the run fails as STALE RECORD until the entry is deleted; the step's coverage output reads partial meanwhile. The report's verdict is always its last line. This change does not widen the identity. A rule live while its count gate is recorded off is DRIFT. The workflow and runbook no longer call github_reader a pure reader: it also holds the origin-window config write on the Function App.

  • Delivery was proven on 2026-08-30: a CLI test notification reached ops-email and then ops-sms (T-709), and the cross-subscription hop had delivered on 2026-08-25 when alert-app-exceptions-prod-cus fired. The runbook's Delivery is unproven section, ADR 0022's consequence, the cost analysis, the deployment runbook's §4 step 3 and the comments in infra/observability.tf now say so, with the CLI commands that re-prove it. The runbook's operator table gains the reachability, export and Telegram rules it omitted, and verify-alert-state.test.mjs fails when a declared rule has no row there.

  • A missing function, an origin lock that is off, and a stranded AzureWebJobsStorage strip each fail a check now (PLAT-2, #962). The estate assessment found all three passing.

  • Registration is checked by name. deploy-functions.yml asserted more than zero registered functions and monitor-functions-registered.yml at least one, so the 2026-08-21 incident (83 deployed, 80 registered, three timers down) passed both. functions/function-inventory.json lists all 248 functions by trigger, and function-inventory.test.js fails, naming each name, when it and the registrations in index.js disagree. scripts/check-registered-functions.mjs compares a live listing with it and names every function missing. The deploy runs it against its own commit, in a step of its own after the firewall windows close, polling for up to three minutes. The monitor runs it against the commit whose package is live, found from the last successful Deploy to Azure Functions step, so a merged and undeployed function is not reported missing. It reads twice more, a minute apart, before believing a miss. It holds actions: read to read that run history. Until a deploy carries the inventory, the monitor says the check is not armed and asserts only a non-zero count, as before.
  • The deploy's origin-lock step read Cloudflare. It curled the proxied hostname, which Bot Fight Mode answers with a 403 whether or not the Azure restriction is on. scripts/assert-origin-lock.mjs now reads the Function App's access restriction from the control plane and holds it to infra/. Unmatched requests must be Deny. Every address in every Allow rule is read, so none can admit everyone beside a permitted one. cloudflare-* rules may list only the declared cloudflare_ip_ranges, and each declared range must be admitted. ci-* windows may admit one address, and this run's window must be gone. It runs whenever the Azure sign-in succeeded, so a deploy that fails early still checks the standing lock. A deploy while functions_origin_lock_enabled is false now goes red. The deploy identity's Website Contributor on the app already covers the read.
  • No Terraform ordering can keep a failed apply from stranding the strip. Terraform skips only the dependents of a failed resource, so an error elsewhere cannot stop the strip, and that was already true. An error on the path itself can: the Function App's update, the settings read, the strip, or a cancel or -target between them. infra/functionapp.tf now says exactly that, and webjobs-strip-ancestry.test.mjs fails if the pair gains an ancestor. The deploy now asserts RUNTIME_CONFIG_WRITER = azapi-strip as well as an absent AzureWebJobsStorage, as the monitor always has. The deployment runbook gives the owner's two PowerShell reads after any apply that did not finish green. It also corrects §5's claim that Terraform "stops scheduling new nodes" on an error.
  • A Docker data-root switch no longer leaves Coder unreachable, and the Coder database volume has one owner. On the first bootstrap after LAB-5 (2026-10-08) the old root's bridges stayed in the kernel, down, on the Coder networks' fixed subnets, and their routes came first. The host's own traffic to Coder, Docker's port proxy on 127.0.0.1:7080 included, went into an empty bridge, so Caddy answered 503 and the site said lab workspaces were unavailable, while Coder and PostgreSQL ran normally and talked to each other. Deleting the two stale bridges brought it back at once.
  • The docker role now runs hcw-docker-stale-bridges on every bootstrap, once the daemon is up. It deletes a bridge only when its network is recorded in another Docker data root on the host, the running daemon does not list it, and nothing is attached, so another program's bridge named like Docker's, or a network the daemon has just created, is never touched. A test runs it against stand-ins for docker and ip and a scratch data root.
  • coder-postgres-data is external: true in the Compose file, and the coder role creates it before up. Compose had warned on every run after the carry that the volume "was not created by Docker Compose". It used the carried data, all 76 MB of it, but nothing said who owned the volume. Now the role does, and docker compose down -v cannot remove it.
  • The lab host's bootstrap no longer stops at 'ansible_managed' is undefined. Since ansible-core 2.19, ansible_managed exists only for the template module. Three copy tasks that LAB-5 added (#987) named it in their content: the systemd delegation drop-in and the tmpfiles entries for the sandbox and agent proxy sockets. The first run after the merge failed at the first of them, after the docker role had moved Docker to its remapped data root and before the coder role had recreated Coder there. Each now writes the literal # Ansible managed, the text the templates render, so a host that ran the one-off -e 'ansible_managed="Ansible managed"' sees no change on its next run. scripts/lab-host-ansible-managed.test.mjs fails on the variable in any task, handler or vars file; ansible-lint does not render task arguments, so nothing in CI had seen it.
  • The Health Hub keeps its results, has a pulse, and shows every status in one place, in one model (#1010, #1011). Until now the hub forgot every verdict on sign-out or in a new tab (live results lived in one tab's sessionStorage, the rest in React state), nothing checked anything unless someone pressed a button, and the status badge wrapped under the description on every probe card but "Publishing failures".
  • One model, written once in lib/status.js: Healthy, Degraded, Critical (it answered but said no: a missing setting, a refused key, a failing check), Offline (it could not be reached, or missed three heartbeats) and Unknown (never checked, or past its freshness window). misconfigured and unavailable are read as Critical and Offline, so stored results and older callers keep working. Offline has its own dark badge with an unplugged icon. The freshness windows are 15 minutes for the pulse and the snapshot, a day for live and session checks, 90 minutes for the session token. The server keeps a copy for the pulse; statusParity.test.js fails if the two disagree on any id, window, classification or verdict.
  • Every probe result is stored server-side, one admin_config document per probe (GET/PUT cms/health/probe-results, viewer to read, the probe's own role to write, ETag-guarded with status transitions). The page reads it on load and every minute while visible, shows each card's age and who checked it, and shows a stale result as Unknown beside its last value.
  • healthPulse, a new timer every five minutes, records what the server can check alone (Cosmos, runtime configuration, Key Vault references, storage, scheduled publishing, publishing failures, queue age, orphaned images, links, the forge, Telegram notices, lab agent heartbeats, the AI providers' and MCP servers' last recorded tests) and its own heartbeat. The pulse line above the tabs says when it last beat. While it is late the hub is Offline; until it first beats, Unknown. It is off until HEALTH_PULSE is added to the enabled_timers workspace variable.
  • Every Health card has the same header, body and footer, with the status top-right in a slot that cannot wrap or shrink, including the Overview and Alerts cards. Cards in a row are the same height.
  • digest.publishingOps had three readers and no writer. The scheduled publisher now records every run there, so the Overview, the Telegram /digest and the Scheduled publishing probe see real runs, and three missed 15-minute runs read as Offline. mergeDigest is ETag-guarded, so timers merging the same day's digest in the same second no longer erase each other's blocks.
  • The admin dashboard's pipeline fits its card with six stages of one size, the Publish stage shows its count, and every number on the page opens what it counts.
  • The pipeline row was flex with a min-w-38 box per stage. At 1280 px the six boxes, five chevrons and gaps needed about 1072 px of a 926 px card: #admin-main scrolled 97 px sideways (17 px at 1440), Live Pages ran past the card's edge, and five of the six descriptions were cut to a few letters. The stages are now an ordered list on an equal grid sized by the card, not the window: six columns from 56rem of card, three from 28rem, two below, every box the height of the tallest. The count sits beside the step number, where it cannot squeeze the label; descriptions wrap to two lines; the chevrons sit in the gaps and only when the six share a row.
  • getAdminDashboardSnapshot returns readyToPublish (approved, forge_ready or published, and not live), shown on the Publish stage and on the sidebar's Publish item. It is the same COUNT getPublishSnapshot now runs for readyTotal, which counted every live page as staged: the ready window includes the published status, and only the list dropped live items in JS. The publish fetch excludes them too, so its 150-row window is not spent on live pages.
  • The stat row is a shared StatTile (components/admin/shared) in the page's own section card: one height, a note line held open on every tile, spans rather than paragraphs. Tiles went from 192 px, with 70 px of nothing under the label, to 124 px. Five columns from 48rem of card; not @2xl, which index.css sets to 1400 px.
  • A type tile's number is still the type's total across every stage, and the section now says so. No admin list shows that set (the Review Queue and the Frameworks and Coder Corner lists all open on needs-review and have no all-statuses view), so the Blogs tile used to land on a list that did not hold its number, with nothing to say so. Each type tile now names status=needs_review in its link and its note says how many are waiting there. The headline count and the live, editor and rejected counts in the header are links too.
  • One 24 px gap between sections (was 32), and the paragraphs that close a block set mb-0 over the p default. On a phone the header's actions wrap below the headline instead of squeezing it onto four lines. The Decision Center below the stat row is the one from #1021, unchanged here.
  • DashboardPage.test.jsx looked for the pipeline with closest('[data-slot="card"], div'); the Card renders no data-slot, so it only ever found the nearest div. It now checks the ordered list of six. e2e/admin-authenticated.spec.js checks the dashboard at 1280 px, 1440 px and on a Pixel 5: no sideways scroll, the row inside its card and centred in it, six boxes within 1 px of one size, no description cut, five even stat tiles. Run against the old row layout, all three fail.
  • "Needs a decision" on /admin is now a Decision Center over every decision waiting, and an article sent to review is one of them (#1013, #1014). It used to show the newest few Review Queue items, and "newest" was the newest of an unordered TOP 30.
  • One read. POST getDecisionCenter (viewer) gathers the Review Queue, Editor work, frameworks and Coder Corner awaiting review, forge-ready and staged content, podcast transcripts, Listen & Learn chapters, newsletter issues, social posts and Forge Studio Queue entries that failed, open workflow alerts, unresolved Key Vault references, due or overdue reminders, and Ambassador deadlines inside two weeks.
    • Each source is one TOP-bounded read ordered by _ts in Cosmos.
    • A source that fails is named in the answer and costs only its own items.
    • Editor-only sources are left out of a viewer's answer and listed as such.
  • Six tabs. Needs a Decision, Frameworks, Queues, Pipelines, Governance and Other Actions, each with its count, newest first. The tab is kept in ?decisions=, so Back from an item returns to it.
  • Each row opens its item at its stage. lib/itemLinks.js is now the one place that maps a content stage to its page. New deep links ring the item where it waits:
    • ?transcript= on Recording Hub › Transcripts
    • ?issue= on the Newsletter and Drafts tabs (it worked on Published only)
    • ?alert= on Health › Alerts
    • ?reminder= on Platform › Reminders
    • ?platform=&exam=&chapter= on Listen & Learn › Review
  • in_review needs a decision. Drafts' Send to In Review and the repository import write it. It was counted on the Editor badge, which the Editor list cannot show, and was missing from the Review Queue's default view and the headline count. It is now in needsReview, the queue's default view, and the Frameworks and Coder Corner "Needs review" filter. The queue orders an article sent from Drafts by when it was sent, so it no longer sorts last for lack of fetchedAt.
  • Recount now re-derives each document's counter marker as well as the counters. The change feed computes every move from the marker, so a recount that rewrote only the counters would have let an in_review marker from before this change take its document out of the wrong bucket on its next edit. Press Recount on /admin once after the Functions deploy. Until then the Editor badge still counts articles in review.
  • The pull request's review closed four ways an item could still go missing:
    • The review queue leaves frameworks and Coder Corner out in the query, on the canonical type, rather than after the read, so a run of them can no longer fill its window. The two type sources match on the same rule, so a Framework typed in another case is in exactly one source instead of none. The queries were run against the Cosmos emulator.
    • Newsletter issues are read by their last write, so a keep or a reject of an old issue is in the window.
    • The queue's other views (live, ready, in progress, rejected) read their window in their own sort field's order, with the rows that lack the field after, instead of the newest-written window.
    • Recount writes the counters only under the ETag they had before its scan. A change-feed update that lands during a run makes it scan again, up to three times, and then say so, instead of writing older counts over newer ones.
  • The element defaults in index.css sit in @layer base, so a utility on a paragraph or heading now applies. The p rule (margin-bottom: 1rem; line-height: 1.7), the h1–h6 rule (color: var(--dark-gray); font-weight: 600), the body rule and the header, footer chrome were outside any layer, and unlayered CSS beats every rule in Tailwind's @layer utilities whatever its specificity: no mb-*, leading-* or text-xs line height could change a <p>, and no text-* colour or font-* weight a heading, typography's prose included. Each keeps its place and its values, so an element that sets nothing renders as before. The provider-theme overrides (:is(.theme-*) header, footer, a) stay unlayered on purpose.
  • Measured on the built site before and after, at 1280 px in both themes: the home page, an Azure blog post and /education/labs change only where an element's own utilities now apply. Headings take their font-bold and written colour, text-sm and text-xs paragraphs their own line height (/education/labs is 108 px shorter), and the article body typography's spacing and prose-p:leading-[1.8]. axe color-contrast over 84 routes in both themes: the same 48 violations before and after.
  • 46 headings on 16 public pages had depended on the old colour. The dark-first pages (the AWS, Azure, GCP and FinOps architecture pages, the podcast page and episode player, GitHub tools and workflows, FinOps tools) set text-white on headings that sit on light surfaces in light mode, which the forced colour had hidden: against the painted background they read 1.0 to 2.6:1 once it moved. They drop text-white and keep the heading colour they always rendered with. The muted text-slate-400 eyebrows on the architecture and education pages gain text-slate-600 for light mode, and the home page's hover-panel eyebrows dark:text-slate-400.
  • ProviderStrip and the Docker pages drop the !m-0, !leading-6 and !mb-0 the old rule had forced on them.
  • Replicate MCP and Hostinger MCP sign in with OAuth Connect instead of sitting red on the AI Engine. On 2026-10-08 both cards read "Error": Replicate's hosted server (https://mcp.replicate.com/sse) answers the REPLICATE_API_KEY bearer with 401 invalid_token because it accepts only tokens from its own sign-in, and Hostinger pointed at http://localhost:8100, where nothing ever listened and whose key was never an app setting. Both servers publish the MCP authorization profile, and nothing here could follow it.
  • functions/src/lib/ai/mcp-oauth.js is that profile: protected-resource metadata (RFC 9728, the 401's resource_metadata first), authorization- server metadata (RFC 8414, OpenID configuration as the fallback), dynamic client registration (RFC 7591) as a public client, PKCE S256 only, resource (RFC 8707), the code exchange and the refresh grant — every request through guardedFetch, https on every hop (its new httpsOnly option), and no redirect followed by a request carrying a secret. A server without S256 or a registration endpoint is refused with a sentence that says which.
  • Three super_admin routes, POST /api/cms/mcp/{serverId}/oauth/start, /api/cms/mcp/oauth/complete and /api/cms/mcp/{serverId}/oauth/disconnect. The pending sign-in lives write-only on the server's document (the state's SHA-256, the verifier, who started it, ten minutes) and is refused for another administrator, after expiry, and on replay. The vendors return the browser to https://hybridcloudworks.com/admin/ai-engine/oauth/callback, a new lazy admin page that completes the sign-in and lands back on the MCP Servers tab with the result. Connect and disconnect are audited.
  • mcp.js refreshes an OAuth Connect server's token when it has under two minutes left, refreshes once and retries once on a 401, and never calls such a server without a connection. A refresh the vendor refuses turns the card amber, "Sign-in expired", with Connect — not red. The refreshPlaudToken timer also refreshes these servers when their token expires within 30 minutes; its name, flag and schedule are unchanged, so nothing needs re-arming. Plaud keeps its own refresh.
  • The seeds and their migration: Replicate keeps its SSE URL and loses apiKeyEnvVar; Hostinger moves to https://mcp.hostinger.com over Streamable HTTP; both gain authType: 'oauth'. The lab-host token's localhost binding and the URL rule's loopback-http exception served only the old Hostinger entry and are gone. A config write that changes a server's URL, transport, key or sign-in method clears its stale status and error, and one that moves a connected server's URL disconnects it. No config write can set the flow's own fields, and reads strip the client secret and pending sign-in as they strip the tokens.
  • The token is the super_admin's vendor account, and Hostinger's tools can change VPS, DNS and domain settings, so calling a Connect server's tools, switching it on or off and changing authType need super_admin; every other server keeps the editor gate. Found by the change's own security review, and fixed here: the config routes now refuse a body naming a nested path (a.b) on either collection. Cosmos patches those inside a field, so an editor could have pointed oauth.tokenEndpoint — and with it the refresh token — at their own host, and allowedTools.0 had slipped past #995's super_admin rule since that change. The callback's iss (RFC 9207) is checked verbatim against the issuer the sign-in started with, and an SSE stream naming a message endpoint on another host does not get the token.
  • The pull request's review closed the rest of the ways round those rules. A PUT is judged on the document it stores, so leaving out enabled or url is a change like naming it, and an omitted authType is kept rather than dropped. A config write cannot set a Connect server's tokens, and deleting one needs super_admin, as Disconnect does. The metadata issuer must be the identifier it was fetched for (RFC 8414): Replicate's origin written with and without the root / is one URL, and any other spelling, path or tenant is refused. The final token write of a sign-in, a refreshed token and a refused refresh's disconnect are each written under the ETag of the document they were decided from, so a Disconnect, a URL move or another instance's newer token that lands meanwhile is never undone. Log lines name no server document.
  • The owner presses Connect on each card and signs in at the vendor: MCP servers — OAuth Connect.
  • The Hybrid Lab's health signals say what is true, and the repository move leaves no trap behind (#1009). From the read-only review of 2026-10-08.
  • The Coder status token's expiry check reads a 404 on its own record as the missing api_key:read scope, as it reads a 403: Coder v2.38 answers an unauthorized read as not found, so the check could never say what to do. The Integrations card now says to re-issue the token with that scope.
  • A graceful stop no longer sends a critical "Lab agent offline" on the spot. The agent is still marked offline at once; checkAgentHealth says "offline" once it has been down five minutes, once per outage (offlineNotifiedAt on the agent's document), and the first heartbeat back sends "Lab agent back online after N min", retried by the timer if it could not go. That morning's 04:30 reboot, back in 43 seconds, had sent a red alert and nothing after it.
  • An agent whose fresh heartbeat says stopping or offline, or that the registry has deactivated, is not online. The goodbye heartbeat wrote lastSeenAt, so the public door stayed open for 90 seconds after a stop; a stopping heartbeat with jobs still draining was stored as busy, and the agent's own 30-second heartbeat said idle during the drain, so the door reopened until the final offline. Both now stay stopping.
  • checkAgentHealth writes its offline mark with the ETag of the read that decided it, so a heartbeat landing between the two wins and no false outage message goes.
  • The public estate card's "Last heartbeat" was Arc's last status change, so a healthy host read as silent for days. It is now "Arc status since", beside a new "Job runner heartbeat" row, and policy checks that can never pass on the lab (the guest-configuration baseline, Defender's update and SQL assessments, the seal-key vault's private link and firewall) count as not applicable, each for a recorded reason, instead of as failures.
  • checkAgentHealth logs marks, recoveries and messages at warn, which host.json ingests, and vps-agent writes its errors and warnings with the syslog priority prefixes <3> and <4>, so the lab host's data collection rule picks them up. What those lines say is content-free: counts and the error's class, never a document id or an error's text (a docker-runner error can quote a path from a learner's payload). The agent writes the job id and the detail on an info line beside each, which stays in the host's journal.
  • Every live reference to the repository under its former owner now names saulpatinojr: the lab host's clone URL, the agent's checkout, the Coder template's clone, and the runbooks' gh --repo lines and raw download. The CMS draft import, which refuses redirects, fetches from the new owner with its draft ids unchanged. bootstrap.sh repoints an existing checkout's origin, and CI fails on the old address or owner/name in any tracked file outside the dated records. Coder's allowed GitHub organisation is unchanged, pending #1016.
  • The labs host pages say what is merged and what the host has applied, and the runbook states the order after a merge: bootstrap.sh, then the Coder template push at once.
  • lab-pins-upstream.mjs escapes the package name in its apt index regex. The fix had been stranded since #899. Commit 7c0e84fa was pushed to feat/lab-supply-chain 25 seconds after #899 merged, which re-created the branch and kept the commit out of main. Unescaped, the dot in containerd.io matches any character (containerdXio passed); this is CodeQL alerts #386 and #390 (js/regex/missing-regexp-anchor). The commit was cherry-picked unchanged onto a branch from main, with its test (a non-matching containerdXio stanza now reads null). Rule recorded: a fix found after a merge goes on a new branch, never on the merged one.
  • Two frontend tests that failed under load now wait for what they assert (#997). Each turned a run on main red on 2026-10-08 and passed on the next run of the same workflow.
  • LabPanePage › full screen queried "Exit full screen" synchronously, though the pane reads full screen through useSyncExternalStore, so the relabelling render could land after the query. It now awaits findByRole.
  • ForgeStudioPage › Send to review waits up to 5 s three times, but kept vitest's default 5 s limit for the whole test, so under load it was killed before it could use its own waits (Test timed out in 5000ms, CI on acbbd716). It now has a 20 s limit.
  • The Docker and lab drafts describe the lab images as published to Docker Hub only. blog-docker-01-building-images.md, blog-docker-03-docker-hub-oidc.md and blog-lab-02-one-container.md in docs/content described GHCR first, with a job that copied each image to Docker Hub by digest. They now describe the publish job's OIDC sign-in (900 seconds, after the smoke tests) and its direct docker push, the digest read back from Docker Hub, docker.io/hybridcloudworks/… names, and the subject repo:saulpatinojr@34853639/…. gh attestation verify names saulpatinojr/HCW-HybridCloudWorks, its output re-measured on 2026-10-08. Part 3's copy-by-hand section went with the copy, and the Lab 03 draft clones from saulpatinojr/HCW-HybridCloudWorks.
  • undici stays on the major Node 24 bundles, and a test now proves the guarded fetch works with the runtime's own fetch. Dependabot's #902 (undici 7.30.0 → 8.11.2) was green on every check. Measured on Node 24.21.0, the Function App's line, with its bundled undici 7.29.1: an undici 8 Agent passed to the built-in fetch throws fetch failed (UND_ERR_INVALID_ARG), while the undici 7 Agent reaches the server. guardedFetch does exactly that for every outbound request it pins, and every existing case injected a fake fetch or dispatcher, so nothing would have failed before production did. guarded-fetch.test.js now sends one request through the real pinnedDispatcher and globalThis.fetch to a local server. CI runs it on Node 24: it passes with undici 7.30.0 and fails with 8.11.2, naming both versions. .github/dependabot.yml ignores undici majors for /functions until the engines move to a Node line that bundles undici 8. #902 was closed unmerged.
  • The version-floor tests judge their literal numbers against a frozen floors file, so the weekly updater's pull requests can pass. The updater's first pull request (#999: python 3.14.8, terraform 1.16.5, 2026-10-07) failed 29 tests, though no rule had changed. update-version-floors.test.mjs compares fixture sources frozen at 2026-09-25 with what the file records, and it read that record from the live scripts/version-floors.json, so any moved floor read as the source going backwards. The rule tests in version-floors.test.mjs state floors as literals ("below the floor 1.16.2"), which go stale the same way. Both now read scripts/fixtures/version-floors-2026-09-25.json, a byte copy of the file on the day those numbers were written. The live file still decides every check of the repository's real pins, and its format is still checked against serialise(). Proven by running both suites with #999's floors applied: 965 passed, where 29 had failed.
  • The lab images publish to Docker Hub only; GHCR is no longer published. The first publish after the 2026-10-07 transfer failed at the GHCR push with permission_denied: The requested installation does not exist (run 37730460748). The repository's token can no longer write the HybridCloudWorks organisation's package namespace, and the copy to Docker Hub, which read from GHCR, never ran. Owner decision 2026-10-08: one registry. The changes:
  • publish signs in through the Docker OIDC connection, with a 900-second token taken after the builds and smoke tests. It pushes docker.io/hybridcloudworks/hcw-lab-runner and …/hcw-lab directly, and attests them there.
  • It no longer holds packages: write. DOCKERHUB_ENABLED now switches publishing on or off as a whole.
  • The separate publish-dockerhub copy job is gone.
  • The two consumers now pin docker.io/hybridcloudworks/…: IMAGES.hcwLabRunner in vps-agent/lib/capabilities.js and the Coder template's image. The digests pinned that day were already on Docker Hub byte for byte, so nothing was rebuilt.
  • scripts/lib/lab-pin-bumps.mjs reads and pins the Docker Hub names. registryOf() now drops an explicit docker.io/, which it had been passing to Docker Hub's API as part of the repository name.
  • The lab host's lab_images role matches hybridcloudworks/hcw-lab* and still matches the former ghcr.io/hybridcloudworks/hcw-lab*, so images pulled from GHCR leave the host once no pin names them.
  • The weekly supply-chain scan reads the Docker Hub pins.
  • The Docker images guide, Docker Desktop guide, lab image README, labs-host architecture page and Docker Hub runbook now describe one registry.
  • The images already on GHCR stay there, public and no longer updated.
  • Azure and Docker Hub OIDC trust the repository's new owner after the 2026-10-07 transfer to the personal account saulpatinojr. The move changed the owner half of every OIDC subject GitHub presents, from HybridCloudWorks@312844660 to saulpatinojr@34853639, while the repository ID 1268997852 stayed. From the next scheduled run every azure/login failed with AADSTS700213, first visible as Monitor Functions Registered and Monitor Unresolved Secrets turning red. Deploys would have failed the same way. github_org now defaults to saulpatinojr, and github_immutable_prefix in infra/oidc.tf carries the new owner ID, so the apply replaces the six federated credentials on the three identities in place; the old-owner subjects are not kept, since nothing can present them. publish-lab-image.yml's EXPECTED_RULE, the Docker Hub runbook's rule and its commands move with it. The runbook gains "After a repository transfer", because the connection's rule in Docker Home is edited by hand. New scripts/check-oidc-owner.mjs compares the owner, owner ID and repository ID in infra/oidc.tf, and the Docker rule, with the GITHUB_REPOSITORY, GITHUB_REPOSITORY_OWNER_ID and GITHUB_REPOSITORY_ID of the run itself. The existing cross-reference could not catch this, because every file moved out of date together. It has no dependencies, and the scripts row in ci.yml runs it on every pull request, outside the path filter. A transfer changes no file, so the pull request that has to catch one may be docs-only. The test suite imports the same functions and asserts that the step stays outside the filter. A future transfer fails a pull request rather than the first deploy.
  • bootstrap-terraform-oidc.ps1 takes the three deployment targets from the naming convention and asks nothing. On 2026-10-07 the picker listed all forty-six subscriptions the owner's sign-in could see, forty-three of them other organisations', as candidates for a Terraform grant, with the right three merely starred. When exactly one sub-app-*, one sub-plat-mgmt-* and one sub-plat-conn-* subscription is visible the script now prints them as [ok] and continues; -ChooseTargets brings the picker back, which then lists only sub-* subscriptions unless -ShowAllSubscriptions is given.
  • Set-TerraformRbacCondition.ps1 conditions every target subscription, not the one the CLI happens to point at. az role assignment list --all reads one subscription, so run from the bootstrap (whose context is the identity's home) it saw Management alone, printed "every row should show a condition" over one row, and on 2026-10-07 left connectivity's RBAC Administrator unconditioned. The script now takes -TargetSubscriptionIds, reads each with --subscription stated, defaults to the tenant's sub-app-*, sub-plat-mgmt-* and sub-plat-conn-* subscriptions, and the bootstrap passes its own targets.
  • bootstrap-terraform-oidc.ps1 no longer replaces a federated credential whose subject differs without being told to. On 2026-10-07 the hcw-azure workspace was found in HCP Terraform's Default Project while the identity's credentials, the script default, infra/backend.tf and the deployment runbook all name Site; every run failed at Azure sign-in with AADSTS700213 naming the presented subject. The script's old behaviour on a mismatch was to delete and recreate the credential for whatever project it was given, which turns a wrong argument into a lockout. A mismatch now stops with both subjects printed and the two ways to agree: move the workspace back into the project the identity names (no Azure change), or re-run with -TfcProject and the new -ReplaceFederatedCredentials switch. terraform-identity-grants.test.mjs pins the default to the project backend.tf and the runbook name, so the three cannot drift.
  • The first digest pull request after #986 refused its own files. scripts/open-lab-pin-pr.mjs read git status --porcelain through a helper that trims its output, so the first line lost its leading space and a fixed three-character slice ate the path's first letter: ab-host/coder/templates/hcw-lab/main.tf was "outside the set" and the publish run's last job exited 2 on 2026-10-07. changedPaths now parses the status code instead of slicing by width, follows a rename to its new name, and has a test for the trimmed first line.

  • The admin portal re-checked access in a loop whenever the answer was not "authorized" (found by the signed-in browser journey, 2026-10-07). onAuthStateChanged told its subscribers about every ACQUIRE_TOKEN_SUCCESS, MSAL emits that for every silent token renewal, and useAdminAuth answers every callback by re-running the admin check, which renews a token. An authorized answer is cached, so that settled after a few rounds; a 401 or an unreachable API is not, so nothing did: against the stubbed identity the page made about a thousand token refreshes and a thousand status calls in six seconds, and in production it would have done the same to Entra and the Function App until the tab closed. A renewal for the account already signed in is no longer reported; a sign-in or a different account still is. entraAuth.test.js pins both, and the browser spec bounds the status calls on both 401 paths.

  • The admin dashboard scrolled sideways on a phone (AP-F1). The Explore grid had an implicit column, sized to its content, and each item's one-line truncated description made that the whole sentence: on a Pixel 5 the cards were 675 px wide in a 393 px column. The grid now declares one minmax(0, 1fr) column below sm, and the browser spec checks that no checked admin route scrolls sideways.

  • The agent health timer threw on every run since the 17:17Z deploy, and that was the app-exceptions alert (2026-10-06). schedulers.js called createNotifier for checkAgentHealth (#894) without importing it, so the handler failed with a ReferenceError every five minutes — three runs and six exception rows a quarter hour, one over the rule's threshold of five — and the agent-offline message could never have come from the timer. The sendReminders timer (#910) had the same hole. The import is now static, and the Functions package gains eslint as a correctness gate (no-undef, unused variables, duplicate keys, unreachable code), run by the functions (azure) CI row; against the previous schedulers.js it reports both sites. The Telegram 403s the heartbeat path logged at 22:21Z and 22:23Z are Telegram refusing the bot or chat, not the code; the owner's direct test names which.

  • Lab alert rules match the host by _ResourceId, not by Computer (LAB-2 follow-up, 2026-10-06). alert-lab-heartbeat-prod-cus fired at its first evaluation after the apply, with the host up: for an Arc machine the Heartbeat, Perf and Syslog tables carry the OS hostname in Computer and the Arc resource name only in _ResourceId, so Computer == "arcs-lab-hybrid-prod-cus-01" matched nothing. The three rules now share one predicate on the lower-cased _ResourceId suffix. The heartbeat change is declared for the plan check; the disk and unit-failure queries are multi-line and the plan shows them beside it. Owner step: confirm the hcw-azure run (3 to change); the heartbeat alert then resolves on its next evaluation.

  • Arc lockdown tasks read each setting with config get (LAB-6 follow-up, 2026-10-06). The first version parsed azcmagent config list --json as a list of name/value objects, which it is not, and the owner's first bootstrap.sh re-run stopped in the arc role before the coder, labs_agent and later roles ran, so #900's proxy and unit changes never reached the host. The two settings are now read one at a time and compared as plain text. Owner step: re-run bootstrap.sh; success is failed=0, then coder-docker-proxy in docker ps.

  • Lab host: the Coder server no longer holds the Docker socket, the agent's install and unit are tightened, the Arc agent is locked down, and ADR 0032's false detection claim is corrected (estate review 2026-10-06, findings LAB-5, LAB-6 and LAB-1's record; owner decision that day). The Coder server container mounted the daemon's socket and joined the docker group, so a vulnerability in Coder — reachable through Caddy by every member of the GitHub organisation — was root on the host, holding every other lab credential with it. A coder-docker-proxy Compose service (tecnativa/docker-socket-proxy 0.3.0, pinned by index digest) now mounts the socket read-only and answers only the API sections the provisioner's Docker provider uses: containers, images, networks, volumes, the lifecycle verbs and the read-only daemon endpoints; exec, build, commit, swarm, system, plugins and daemon auth are refused. The server reaches it over DOCKER_HOST on a control network the two alone share (PostgreSQL stays on the coder network only, #900 review), and the template's provider names the same address; the server has no socket and no docker group, the role no longer reads the group id, and template.test.mjs asserts all of it (three services, the read-only mount on the proxy alone, every proxy policy key, the networks, no port). Stated plainly, as the #900 review asked: this narrows a Coder compromise and does not contain it. The proxy reads paths, not bodies, so a compromised server can still create a privileged container or bind the host's / and start it; containing that needs a body-aware authorisation layer or a separate or rootless daemon for Coder, which ADR 0032 keeps as the revisit trigger with that shape written down. The labs agent's npm ci runs --ignore-scripts (no dependency needs an install hook) and its unit gains address-family and system-call filters plus kernel, cgroup, setuid and realtime protections (MemoryDenyWriteExecute deliberately not: V8 needs it). The arc role sets extensions.allowlist to the Azure Monitor Agent and turns Arc incoming connections off, read-first so the play reports changed truthfully: any Contributor on the resource group could otherwise run a command as root through the control plane. ADR 0032's socket bullet carries the amendment, its validation list names the proxy, and its Cloudflare bullet no longer claims the plan check and drift monitor would show a production DNS edit — neither reads Cloudflare; the only record is Cloudflare's audit log, and the interim steps are the token's client-IP filter and expiry and an audit-log notification. Owner steps: re-run bootstrap.sh; docker ps then shows coder-docker-proxy and the coder template's workspaces still start (the proxy is the only path the provisioner has); sudo azcmagent config list shows the allowlist and incoming connections off. Rollback is a revert and a re-run.

  • The lab host's pins and the lab images are checked against upstream every week, and a pin that falls behind or an image with a fixable HIGH or CRITICAL vulnerability opens an issue (estate review 2026-10-06, finding LAB-3; owner decision that day). Everything on the host is pinned and held — Docker Engine and containerd (which carries runc), Caddy and its Cloudflare module, Coder, PostgreSQL, Portainer, Vault, node_exporter, Node.js — and nothing noticed when a pin fell behind; Dependabot cannot read Ansible group_vars or a digest in a JS module, and the published lab images were never scanned after the build that attested them. New scripts/lab-pins-upstream.mjs reads the ten pins from lab-host/ansible/group_vars/all.yml, asks each publisher for its newest stable version (the Docker apt index, GitHub releases and tags, the HashiCorp index, Docker Hub, nodejs.org) and exits 0 current, 1 behind, 2 when a source could not be read, which is "not evaluated", not "fine". New .github/workflows/lab-supply-chain.yml runs it every Tuesday, scans the two published image digests the host actually runs (read from vps-agent/lib/capabilities.js and the Coder template, never a tag) with the pinned Trivy for HIGH and CRITICAL vulnerabilities with a fix, and opens one issue, or comments on the open one, when either says something is due; a clean week is a green run. It bumps nothing: every bump needs a checksum or digest read from the publisher and reviewed beside the version, and lab-host/README.md "Bumping a pin" is the procedure. Dependabot gains a docker entry for lab-image/, the one thing it can watch there (the base image digest on the FROM lines). The first local run found Docker Engine, Caddy and Coder behind upstream, which is the signal that was missing. Owner step: none to arm it; the first run is Tuesday, or dispatch it from the Actions tab, and work the issue it opens.

  • The browser suites run on every pull request, and the admin spec can fail (estate review 2026-10-06, finding QA-1; owner decision that day). No workflow ran Playwright: the hydration (6 tests), console-error (17 routes) and smoke specs executed only when someone remembered, and e2e/contentforge.spec.js held sixteen tests whose every assertion short-circuited on page.url().includes('admin'), so a blank shell or a removed auth guard passed. ci.yml gains an e2e job, deliberately not a required check for two weeks: it installs Chromium, builds once, serves the build with vite preview, and runs the hydration spec and the rewritten admin spec on desktop Chrome and a Pixel 5 (a new mobile-chrome project; CI retries drop from two to one). Its first run (#899) ran console-errors and smoke too and showed why they cannot run here: a pull-request build has no API origin, so every public page logs "Public data fetch failed" and 32 of 70 cases failed on the environment; those two specs belong to the post-deploy probe of the deployed site (PLAT-5), and the job was scoped the same day. The admin spec now pins what a visitor who has not signed in gets on four admin routes: the sign-in card, no navigation, no menu button, no data; that the card is keyboard reachable and the route titled; that a 375 px viewport never scrolls sideways; and that the public home page never loads the identity library. An authenticated journey needs a stubbed identity and is the next step. The report and traces upload on failure.

  • Admin Portal on a phone: an off-canvas menu with every label readable, a document title on every admin route, and the live counts in the link names (estate review 2026-10-06, findings AP-F1, AP-F2, AP-F3; owner decision that day). The admin shell rendered a 64 px icon rail that could not be dismissed, so a 375 px phone had about 263 px for the page and 71 px with the rail expanded, and the rail's labels were title tooltips, which touch never shows. No admin page set a document title, so the tab, the history entries and the route announcer carried whatever the public site set last (WCAG 2.4.2). The badge counts were aria-labels on spans with no role, which screen readers do not reliably announce. Now: below md the rail is not rendered; a slim top bar carries a menu button that opens a Radix dialog drawer (focus trapped, Escape closes it) with every group and label visible, and any link in it closes it. From md up the rail is as it was. PageHeader sets document.title to the page's title, and AdminLayout sets a registry fallback on every route, so the announcer reads the new page's name. A link with a count is named "Review Queue, 3 waiting" and the badge is aria-hidden. The content column is max-w-5xl with p-4 on phones (max-w-2xl throttled the dense pages on a laptop). The two sub-12 px faded labels in the shell and the page eyebrow are 12 px at the full muted token. Tests: the link name carries the count, the title follows the route, the rail is hidden below md, the drawer opens labelled and closes on navigation.

  • The lab host now pages the owner: heartbeat absence, root disk past 85%, a watched unit failing, and the labs agent going offline (estate review 2026-10-06, finding LAB-2; owner decision that day). Until now the Arc data collection rule shipped heartbeat and auth syslog only, node_exporter listened on loopback for a scrape that never came, checkAgentHealth marked an agent offline in Cosmos and told nobody, and the backup timer failed silently — the public status card said "unreachable" before the owner knew. infra/lab-hybrid.tf adds four host counters (CPU, memory, root disk used and free, 60 s) and the daemon/syslog/kern/cron/ user facilities at Warning and above to dcr-lab-hybrid-prod-cus (kilobytes a day against the cap), and three scheduled-query rules in rg-mgmt-plat-prod-cus on the existing action group: alert-lab-heartbeat (no Heartbeat row in 30 min, stateful, severity 1), alert-lab-disk (root past 85%, hourly) and alert-lab-unit-failed (a hcw-unit-failed syslog line in the last hour). The lab-host hardening role installs hcw-unit-failed@.service, a template unit that logs one line at daemon.err naming the unit that failed, and the Coder backup, labs agent, Caddy and Vault units name it in OnFailure=. The checkAgentHealth timer takes the shared Telegram notifier and sends one message (per the notifier's 15-minute cooldown) when it marks an agent offline, naming the agent, host and last heartbeat; the mark never depends on the message. scripts/assert-expected-plan.mjs declares the three creates; the DCR change is an in-place update the plan shows beside them. The alerting runbook's table carries the three rules; the lab architecture page's control-plane row is now live. Owner steps: confirm the hcw-azure run (3 to add, 1 to change: the DCR), re-run bootstrap.sh on the host so the notifier and the four OnFailure= lines land, and expect the Telegram message within five minutes of stopping hcw-labs-agent once the Functions deploy carries this. The verifier workflow reads the web resource group only, so these three are listed by az monitor scheduled-query list -g rg-mgmt-plat-prod-cus instead (PLAT-4).

  • Security: the Terraform run identity's role-assignment right is constrained, so a run can assign what infra/ needs and never make itself Owner or read the vault (estate review 2026-10-06, finding SEC-1; owner decision that day). scripts/bootstrap-terraform-oidc.ps1 grants id-plat-terraform-prod-cus-01 Contributor + Role Based Access Control Administrator per subscription, and its comment said RBAC Administrator "cannot grant Owner or User Access Administrator". It can: without a condition the role writes any assignment, so a compromised workspace, a leaked TFC_TOKEN, or a malicious provider executed during a plan could have assigned itself Owner, or Key Vault Secrets Officer on kv-site-prod-cus-01, and read every production credential. New scripts/Set-TerraformRbacCondition.ps1 applies Microsoft's constrained delegation: an ABAC condition on every RBAC Administrator assignment the identity holds, built from role names resolved live, that refuses writes of Owner, User Access Administrator, RBAC Administrator, Key Vault Administrator, Key Vault Data Access Administrator, Key Vault Secrets Officer, Key Vault Crypto Officer and Key Vault Certificates Officer — except Secrets Officer to a USER principal, which is the admin_object_ids seeding window — and refuses deletes of Owner, User Access Administrator and RBAC Administrator. Everything infra/ assigns today passes. Idempotent, honours -WhatIf, re-creates an unconditioned assignment at the same scope. The bootstrap script calls it at the end of its role step and its comment now says what the role does. Owner step: run the script once (section 0 of docs/runbooks/deployment-runbook.md); success is one conditioned line per subscription and every row condition present in the listing that follows.

  • One recovery objective pair everywhere, the as-built zone posture written down, a regional-loss procedure, and the first restore drill approved (estate review 2026-10-06, finding PLAT-1; owner decision that day). ADR 0011 said RTO four hours / RPO one hour with mandatory quarterly drills and a zone-aware, ZRS posture; ADR 0028 and TODO.md said RTO 8 hours / RPO 24 hours and that no drill was wanted; the architecture page repeated the July numbers; the restore runbook's header still said "not yet armed" a month after the exporter's first run. ADR 0011 now carries an amendment: the 2026-08-30 pair (8 h / 24 h) supersedes the July targets; Cosmos, the Function App and storage are single-zone with geo-redundant storage, each declined on cost; the 2026-09-09 no-drill decision is recorded and superseded by the 2026-10-06 approval to run the drill once, then quarterly. The architecture page's reliability model and the Well-Architected rows say the same. The restore runbook's header reflects the exporter running since 2026-09-09 and the drill being due, and gains "A regional loss": what survives (RA-GRS exports, code, state, the owner's secret copies), what does not, and the order a restore into a second region would run in, stated as untested and outside the 8-hour objective. TODO.md's accepted-risk row records the approval and deletes itself when the Drills table has a row. Owner step: run the drill (docs/runbooks/cosmos-restore.md, "The drill"); success is one dated row in its Drills table with measured RPO and RTO.

  • Security: the deploy identity trusts the production environment only; its ref-form federated credentials and its unused Cosmos data-plane grants are gone (estate review 2026-10-06, finding SEC-2; owner decision that day). id-site-github-deploy-prod-cus-01 carried four federated credentials: the production-environment pair every deploy actually presents, and a ref:refs/heads/main pair that no workflow has presented since the healer went in #816 and the manifest moved to the reader identity in T-728. The ref pair was the way around the production gate: any job on main holding id-token: write (ten workflows do, for Qlty, Pages and the manifest) could have signed in as the deploy identity with no reviewer and deployed code that runs as the Function App's own identity. infra/oidc.tf deletes github_branch and github_branch_immutable, and with them the two container-scoped Cosmos Data Contributor assignments on content and blogs that outlived their --inspect reader and contradicted ADR 0025's "CI holds no Cosmos data-plane role" (the ADR carries the amendment). scripts/oidc-subjects.test.mjs now asserts the deploy identity trusts environment-form subjects only, so the pair cannot come back unnoticed; scripts/assert-expected-plan.mjs declares the four destroys for the plan check. The reader identity keeps its ref pair: its workflows name no environment and it holds read roles. Owner step: confirm the hcw-azure run — expected 0 to add, 0 to change, 4 to destroy — and afterwards the federated-credential list shows the two production-environment subjects and nothing else.

  • Security: a shared integration key may only be sent to its vendor's host, moving where a credential is sent needs super_admin, and every AI provider or MCP server write is audited (estate review 2026-10-06, finding AP-B1). The key-name allowlist from ADR 0033 stopped an editor naming the Cosmos connection string, but it still let the three keys the seeded servers use be routed anywhere: a server saved at a host the editor controls with apiKeyEnvVar: "VPS_API_TOKEN" received the lab-host token as a bearer on the next Sync, in two requests. ai/mcp-policy.js now carries INTEGRATION_KEY_HOSTS (Firecrawl, Replicate, and loopback for the seeded Hostinger entry) and validateMcpKeyBinding, which the config-collection writes check against the stored half of the pair (a URL move on a server holding a shared key, or a shared key placed on a server at the wrong host, is refused either way), syncMcpTools and mcpProxy check before any call, and resolveMcpAuthHeaders refuses the bearer at an unbound host or with no URL, so a document that slipped past save-time still sends nothing. A write that moves credential routing — a new MCP server, a URL change, a key-name change — needs super_admin, the role that seeds the keys; flipping enabled, renaming, reordering or storing an OAuth token stays at editor. Every ai_providers and mcp_servers PUT, PATCH and DELETE now writes an ai_config_updated row to admin_audit_logs with the actor, the field names, and the URL and key NAME before and after (never a token value); the row is best effort, so a failed audit never reports a landed save as refused. MCP_* settings bind to no host: they are per-server secrets the owner creates for one server. Tests: the binding for every shared key and the MCP_* pass-through, no bearer at an unbound host, Sync and proxy refusing a rogue server without calling it, PUT and PATCH refusals in both directions, the super_admin gate on moves versus the editor toggle, and the audit row's shape with no secret in it.

  • Security: the article scraper and the supporting-document fetch now go through one SSRF guard (estate review 2026-10-06, finding AP-B2). generateArticleDraft and the forge-from-url job fetched whatever URL the editor named with a bare fetch that followed redirects, and the page (or 18 KB of a "KB document") came back into the draft or the source document. From the integration subnet that is a request-forgery primitive: http://10.0.0.5/, the metadata address, or a public host that redirects onto either. The guard the image fetcher has carried since T-734 is lifted into functions/src/lib/http/guarded-fetch.js — http(s) only, localhost refused by name, the host resolved to IPv4 and refused when loopback, link-local or RFC 1918, redirects followed by hand with every hop re-checked, a deadline per hop — and content/scrape.js (the page itself) and content/draft-from-url.js (each documentUrls entry) fetch through it. A refused URL ends the scrape rather than being handed to the reader or headless fallback, which would only fetch the same forbidden target from their side. triggers/fetch-image.js re-exports isPrivateIp and validateFetchUrl from the new module, so its importers and tests keep their names. Copilot's review of #889 tightened the guard four ways, all taken: Azure's platform address 168.63.129.16 (WireServer, plain HTTP), carrier-grade NAT and the multicast/reserved ranges are refused with the rest; the connection is pinned to the address the guard validated, through an undici Agent whose lookup answers only that address (undici becomes a declared dependency; it was already in the tree under cheerio), so a rebinding host cannot pass the check with one record and connect with another, and fetch-image.js pins the same way; an IPv6 literal, and a name the IPv4 lookup cannot resolve, are refusals rather than fall-through to the reader; and the deadline now covers the body, which is read under it and under a byte cap inside the guard ({ response, buffer, text } comes back, not a live Response), so a document origin that sends headers and then stalls is cut off. Tests: the guard's own suite (every refused range including the Azure address, IPv6 literals, unresolvable names, pinned lookup in both callback shapes, a rebinding host, relative and private-bound redirects with re-pinning, redirect limit, streamed and declared body caps, a hung socket and a stalled body), scraper cases proving no fetch and no fallback on a refused target, and a drafter case proving no documentUrls byte is read from a private, link-local or localhost address or across a redirect onto one.

  • Cloudflare DNS records: ignore_changes = [include_shadow_metadata] on every cloudflare_dns_record in infra/ and infra-lab/ (owner decision 2026-10-06: stay on provider 5.27.0). Provider 5.26.0 surfaced the include_shadow_metadata request parameter as a record attribute with a static default, so every pre-existing record planned null → false, the forced write came back with a new modified_on, and the hcw-azure apply after #879 failed with "Provider produced inconsistent result after apply" on docs_pages and azure_functions (upstream #7387, #7396). Ignoring the attribute removes the spurious update; the records were written with identical content and both hostnames answered 200 throughout.

  • Ambassador sub-programs: a cleared parent stays cleared, a program with children takes no parent, and a malformed chain shows rather than hides (#881 review). The seed backfill treated an explicit parentProgramId: null as missing and restored the seeded MCT parent on the next read, so "Additional to → none" could not stick; it now fills the field only when it is absent. checkParentProgram also refuses a parent on a program that already has children (one level only), and the frontend's programsInPlay lists a program whose parent is itself parented at top level, so a chain or cycle two overlapping edits could leave behind is visible and repairable on Settings rather than hidden.

  • CodeQL alert 383 (note). platform-settings.test.js still imported three names from the Listen & Learn speech settings that ADR 0034 slice 5 (#867) removed from that module or left unused; the import block is gone.

  • The weekly certification re-verify could not republish the snapshot (#868). schedulers.js handed createCertReverify a snapshots.publishSnapshots that createSnapshotPublishHandlers never returned, so the Sunday run threw "is not a function" after marking a certificate expired, revoked or renewed, and the public snapshot stayed stale. The handler object now carries publishSnapshots; the timers test pins the name against the real factory.

  • 06-seed-secret.ps1 no longer reports a refused write as done. On 2026-10-05 it printed ANTHROPIC-API-KEY set over a ForbiddenByRbac from az (the data-plane role grant had been skipped), then crashed on the read-back with a null-argument error, and the operator read a seed that never happened as complete. az reports a refused call on stderr and exits non-zero without throwing, so both the write and the read-back now check $LASTEXITCODE, stop the run, and name the missing role as the likely cause; the firewall window still closes in the finally block.

  • GPT-5 models refused every OpenAI and Foundry call over temperature (#849). The owner's first Test of the Foundry card, after the apply and deploy, reached the account and was authorised, then failed with 400 Unsupported value: 'temperature' does not support 0.2 with this model. Only the default (1) value is supported. The OpenAI-compatible request body sent temperature: 0.2 to every provider; the GPT-5 family accepts only the default, so the OpenAI provider, whose defaults moved to gpt-5-mini and gpt-5-nano, has been failing the same way. The table row now decides: NVIDIA's catalogue models keep 0.2, the OpenAI and Foundry rows send none. foundry-provider.test.js and nvidia-provider.test.js pin both sides; the two setup guides say so.

  • NVIDIA's Test timed out with a working key; every NVIDIA default is now GLM-5.3 (#701). The portal's Test sent z-ai/glm-5.3-flash and gave up at 45 s, three times on 2026-10-04, after a freshly rotated key was seeded. A direct call from a workstation with a one-line prompt measured z-ai/glm-5.3 at 2.6 s, z-ai/glm-5.3-flash at 70 s and deepseek-ai/deepseek-v4.1-flash past 120 s, so on the trial tier the "flash" models are the slow ones. Drafts, analysis, multimodal and general calls all default to GLM-5.3 (functions/src/lib/ai/router.js), and the AI Services card offers and describes only that model. The other two keep their zero-cost rows and can return through a CONTENTFORGE_NVIDIA_*_MODEL override. The stored card would not have followed: the page wrote it on 2026-09-25 pinned to z-ai/glm-5.3-flash, and the router obeys a pin for every purpose. providerModelPatches (frontend/src/lib/aiEngine/seed.js) now keeps each stored provider's model list equal to the seed on admin page load, and returns a pin the list no longer offers to the seed's own value, Auto for NVIDIA. A pin still on the list is left alone.

  • Every provider the site routes can be chosen on the review board, Docker, VMware and Ansible included. #776 added /docker/blog/<slug>, and VMware and Ansible have had blog routes longer. But the CMS knew six providers: normalizeProviderName turned Docker, VMware and Ansible into ''. The review board offered Azure, AWS, GCP, GitHub, Terraform and FinOps only, and "Send to Publish Queue" stays disabled until a provider is set, so an article for any of the other three could go nowhere.

  • normalizeProviderName is now derived from PROVIDER_ALIASES in public-reads.js, the server's provider registry, which public-section-counts.test.js already holds to VALID_PROVIDERS. The stored value is the route key with its first letter capitalised, as Aws and Finops always were: Docker, Vmware, Ansible. Every spelling the registry lists (Google Cloud, GitHub, VMware) still normalises. publish.test.js publishes a Docker, a VMware and an Ansible post end to end, to /docker/blog/<slug>, /vmware/blog/<slug> and /ansible/blog/<slug>.
  • PROVIDER_OPTIONS in frontend/src/config/admin.js is now built from CANONICAL_PROVIDERS in lib/providers.js, the frontend registry, rather than kept as a copy. It feeds the review board's picker and the Publish-Ready Builder's provider and landing-zone dropdowns. The editor's provider filter, which had Docker but not VMware or Ansible, now reads the same list. The submission forms keep their own lists on purpose: each offers only what its page can file under. admin.test.js and BlogReviewBoard.test.jsx fail if a provider is registered but missing from the board, and docker-cms.test.js holds the frontend registry to normalizeProviderName.
  • inferProviderFromTags knows every provider too. It returns Docker only when no other provider is tagged: Docker on Azure is Azure, the same rule as docker coming last in providers.js. The server's URL inference and the recording drafter know Docker too, also last.
  • The public submissions API accepted no Docker submission, though the blog and Coder Corner forms have offered it since #775. Worse, its framework and Coder Corner lists had been swapped since the API was written, so it refused every submission from either form. Each list is now its page's list, and submissions.test.js reads the pages to keep them that way. ci.yml runs the functions tests when those pages, admin.js or lib/providers.js change.

  • AZ-800, AZ-801 and Google's PAA re-read on the day their dates passed (#770). From 2026-10-01 the catalogue alarm failed on every branch for three rows dated 2026-09-30. Each vendor page was re-read that day. Microsoft retired both exams as announced: the AZ-800 and AZ-801 pages now say "This exam was retired on September 30, 2026", and AZ-802 is open to schedule with no retirement date. Both rows are retired with retiredDate 2026-09-30 and still point at AZ-802. Google's Professional Agentic Architect page and its certification index are unchanged: the banner still says the beta "is open until September 30!", with no GA date, no new window and no withdrawal. PAA is therefore stored as upcoming with no date, and its description says the beta has closed and GA is not yet dated. The tests that pinned the old statuses now pin what the vendors did, including the Azure page and detail-page tests, which expected AZ-800 to read as expiring.

  • A new budget takes the month it is created in (#820). Azure refuses a monthly budget that starts before the current month, and budget_start_date defaulted to 2026-08-01. From September, any apply that created a budget would have failed at the end of the graph, for a reason unrelated to the change: a fresh subscription, a rebuild or a forced replacement. The start date is now the first of the month the plan runs in (from plantimestamp()), unless the variable pins one. Both budgets ignore the start date once they exist, so the two live budgets plan no change.

  • cp_sortDate is written by the apply that could wipe it, not by a six-hourly healer (#816, first of two). azurerm_cosmosdb_sql_container cannot express computed properties, so an apply that updated content or blogs dropped cp_sortDate. That broke the public list's ordering until heal-computed-properties.yml put it back, up to six hours later.
  • infra/cosmos.tf adds azapi_update_resource.cosmos_computed_properties, one per container that carries a computed property. It is re-run by replace_triggered_by whenever its container changes, and a postcondition fails the plan or apply if the container does not carry the definition.
  • The definition moves to scripts/lib/cosmos-sort-date.mjs. The generated infra/cosmos-containers.json now holds the real query instead of a sentence.
  • scripts/assert-expected-plan.mjs lists these as conditional replacements, and learns to match a declared create by exact address.
  • The healer keeps running until this is seen to hold. #816's second pull request removes it, its role assignments and its --apply path.

The same change restores alert-func-http5xx to the 15-minute window its description states: #250 had given it the availability probe's 30 minutes, and with them the probe's reasoning. - The inputs and secrets docs describe the estate as it is (#819). Most of these pages were last true in August. Each change below was checked against the code or the live estate on 2026-09-29. - required-inputs.md, §4.2 and §4.3. The repository holds two secrets, TFC_TOKEN and MANIFEST_APP_PRIVATE_KEY, not "one", and both are now listed with their workflows, alongside MANIFEST_APP_ID. READER_CLIENT_ID (set 2026-09-06, four workflows) and VITE_TURNSTILE_SITE_KEY (set 2026-09-28) are no longer shown as unset. FUNCTIONS_URL feeds the frontend build's VITE_AZURE_FUNCTIONS_URL, which the doc called retired. STORAGE_ACCOUNT and STORAGE_RESOURCE_GROUP are marked as read by nothing. - required-inputs.md, §4.7 and §4.10. The lab rows read SET from measurement: the Coder references resolve, the Arc fact is present, and the lab agent is registered (its claims and heartbeats answer 200). The outputs count is 24, which adds copilot_review_client_id. - variables-and-secrets.md. The August placement errors are marked resolved or moot. TF_API_TOKEN becomes TFC_TOKEN. The page no longer cites the deleted key-vault.js, getSecret(), openai-client.js or migrate-data.yml. AZURE_OPENAI_ENDPOINT is deliberately absent, and KEY_VAULT_URI is back for the write-only API Keys path. - Runbooks. deployment-runbook.md no longer describes the deleted deploy-infra.yml, and points at §4.1, infra/budget.tf and infra/functionapp.tf. alerting-and-support.md gives the latency alert as P95, and the 5xx alert's 30-minute window as deployed. That window is itself a bug, carried to #816. cosmos-restore.md says 61 of 74 containers, and lists tool_price_history among those not restored. - Elsewhere. iac-repository-standard.md drops the removed agent and issue template. routeFactory.ts drops a pointer to a guide that does not exist. The runner-image rollout in version-floors.json is future tense again. The admin NotebookLM entry records that the consumer product still has no API, while Gemini Notebook Enterprise has a v1alpha one. - The API Keys page no longer states a provider order the code contradicts (#815). Its help text called Anthropic "second" and OpenAI "third", but DEFAULT_PROVIDER_ORDER tries OpenAI before Anthropic. NVIDIA's text still said it wrote "first", though it has been the backup since #812. The order is the AI Engine page's to change, so the four provider entries now point there instead of naming a position, and a test in secret-catalog.test.js refuses any ordinal in them. The same pass fixed module headers that described closed gaps: - jobs.js: the stale-queued gap that platformJobSweeper closed; - key-verdict.js, secrets-health.js and the catalogue: three key-verdict reporters, not two, and four router providers, not three; - ai-config.js: "three answers", not two; - platform-health.js: axios has left package.json, and rss-parser is used elsewhere.

required-inputs.md gives the order by pointing at the AI Engine page, not as positions. - The break-glass secret seeder runs again (#814). scripts/cutover/06-seed-secret.ps1 is for seeding a Key Vault secret when the site is down. It built its list of allowed names from infra/main.tf alone. Since the split (#269) that file holds no references, so every run, -Mode List included, stopped at "Parsed no secret names". It now reads every .tf file in infra/, as functions/test/terraform-source.js does. New Pester tests (06-seed-secret.Tests.ps1) hold it to the repository as committed, and the repository-policy workflow now runs scripts/cutover beside scripts/lab. Its closing hint now prints the real restart and health-check commands instead of placeholders. The docs that placed the references in main.tf, or the script's window in admin_ip_rules, now match. - The Usage tab names the Playground's calls. aiEngine.chat records Playground calls as admin_playground, but SOURCE_LABELS knew only the older admin, so those rows rendered as a raw slug. Both now read "Admin playground", and a test pins it. variables-and-secrets.md also no longer claims the timer flags "still resolve to false today": timers have run since they were armed, so it now says a timer runs when the master flag is on and its name is in enabled_timers. - The secrets monitor can succeed again, so delivery health goes green. monitor-delivery-health's "Can every workflow still succeed?" failed on 2026-09-29 (run 36609411076). The workflow it named, monitor-unresolved-secrets.yml, had failed 10 of 10 runs and had been switched off since 2026-09-14. The only references it found unresolved were AZURE_SPEECH_KEY, already listed as intended, and RSSCOM_API_KEY and RSSCOM_PODCAST_ID, which wait on the owner's RSS.com Max purchase (#432, owner decision 2026-09-18). - Both RSS.com secrets are now in EXPECTED_UNRESOLVED in scripts/check-unresolved-secrets.mjs, with that reason. - Run against the live Function App's references, the monitor now exits 0: all 60 resolve or are unresolved as intended. - Re-enabled, it catches a reference that should resolve and doesn't, such as a lapsed Coder status token. - A provider's name on the AI Engine page follows the code (#701). #806 renamed "NVIDIA API Catalog" to "NVIDIA API" in DEFAULT_PROVIDERS, but the card renders the stored ai_providers document, and seeding only writes a document that is missing. So the old name stayed. - seedAiEngineIfEmpty now patches each stored provider's name, description and icon back to the seed on every page load. These are SEED_OWNED_PROVIDER_FIELDS, which the page gives no way to edit, so a difference is only ever a stale copy. - Fields an administrator sets (model, enabled, order, placement) and unknown providers are untouched. - providerDisplayPatches is pure and tested, including the NVIDIA document as it was stored on 2026-09-25. - A slow first AI provider now fails over before the browser gives up (#701). Generate draft is a synchronous call. The browser waits 90 s and the edge about 100 s. NVIDIA is first for drafting and was allowed 120 s per attempt, and on 2026-09-29 its reasoning models took 56-58 s to answer one word. So the failover to a paid provider started after the page had timed out, and the editor saw a timeout although Gemini would have answered. - Synchronous routes now give the AI router a time budget (budgetMs). Half of it is held back for the next provider, and each attempt's timeout is cut to its share. A retry is made only if a whole attempt still fits. The contract is in the header of functions/src/lib/ai/router.js; the arithmetic is in functions/src/lib/ai/time-budget.js. - Routes with a budget: Generate draft and Create from recording (the existing 75 s handler budget, less 5 s for the work after the model), regenerate newsletter intro and suggest subject lines (14 s), and the social caption Generate button (15 s). The last three answer under the browser's 20 s default. - A call that runs out ends with AI_BUDGET_EXHAUSTED (HTTP 504). It names each provider tried, why it failed, and which were never reached. - Background paths pass no budget and are unchanged: forge jobs, the schedulers, the change feed and the queue workers. NVIDIA keeps 120 s there. - Gemini and Anthropic now honour a per-call timeoutMs, as OpenAI and NVIDIA already did. The portal's Test passes 45 s, so it now bounds those two at 45 s instead of 60 s. - sync-budgets.test.js pins each budget under the browser's timeout in frontend/src/lib/api.js and the edge's 100 s. The functions CI row now also runs when api.js changes. - The AI Engine's Test for NVIDIA answers in seconds, and the card reads "NVIDIA API" (#701). The one-word Test ran with the drafting limits (8,192 tokens, 120 s). NVIDIA's models are reasoning models on a trial tier, so it took 56.3 s and 57.9 s on 2026-09-29 (AppRequests testAiProvider). A slower model outruns the edge's ~100 s request limit and never reports. - The Test now asks for 16 tokens inside 45 s (TEST_MAX_TOKENS and TEST_TIMEOUT_MS in functions/src/lib/ai/proxy.js). - callProvider passes both through to providers whose row sends max_tokens. That is NVIDIA; OpenAI never gets one, because its reasoning models refuse the field. - A stuck provider now fails with "timeout after 45000 ms" instead of hanging. - The card and the keys-page label drop "Catalog". - Required-Inputs marks NVIDIA-API-KEY as seeded. - The home page no longer shows invented numbers or sample content, and two small lines on Home and Learn pass contrast. Every figure and list on / is now read from something real, and homeContent.js says where each one comes from. - Quick Access Hubs. Each hub printed a typed feed count: Azure said 24 while the server reads 7, AWS 32 against 3, GCP 18 against 3, Terraform 12 against 1, GitHub 9 against 4 and FinOps 7 against 1. VMware, Ansible and Docker said "NEW HUB". Each note is now the number of feeds that provider's news page reads, from frontend/src/data/providerFeeds.js. That file is a copy of PROVIDER_FEEDS in functions/src/lib/rss/feeds.js, the list the ingest actually fetches. providerFeeds.test.js fails if the two lists differ by a feed, name or address. HomePage.hubs.test.jsx checks each rendered count against the server file itself. The frontend CI job now also runs when that file changes. ProviderContext's rssFeeds reads the same list: it still named Azure Updates' retired CDN feed and Ansible's old address. - Stat row. The row showed 24+ blueprints, 104+ articles, 22+ modules and 99.9% uptime. - Blueprints is now the count of the staticBlueprints the architecture pages export (24). - Articles is now the published total that GET public/content reports (24 on 2026-09-29). - Modules is gone: the Terraform modules page lists six cards, and nothing counts 22. - Uptime is gone: the availability probe reports only to monitoring that a visitor cannot see. - "Latest from the providers" was ten hard-coded sample headlines, stamped "2 HRS AGO" to "4 DAYS AGO", with a "Last 24 Hours: 3" count. It is now "Latest articles": the six newest published items the blog pages read, fetched in the browser. Each card opens its own page and is dated from the article. Loading, empty and error states use visitor wording. The pre-render shows the loading line rather than a list frozen at build time. - Blueprint carousel. Seven of its ten hand-typed cards opened "Architecture Not Found", and every card said "Snapshot v2.0". It now rotates the 24 real blueprints. AWS's six open their own pages; the rest open their provider's architecture page, where they are shown. The section is now "Reference Blueprints: Architecture Designs by Provider", not "from Cloud Vendors". - Platform Health. - The last row said "GitHub Actions" over GitHub's overall status. It now says "GitHub". - Each row now links to that provider's own status page. Two of them used to open an unofficial AWS feed and Azure's products-by-region table. - The timestamp is now the server's checkedAt, not the time the page asked. - A failed check now reads UNKNOWN instead of staying on CHECKING. - Its light-mode badges took the -800 shade the site's other status badges use. OPERATIONAL measured 3.41:1 and REGIONAL IMPACT 3.03:1. - Contrast. - CatalogueFreshness, the "Catalogue checked against …" line, measured 3.28:1 on the /education tiles in light mode. The home page's "last verified" timestamp measured 4.01:1 in dark mode. Both now use the text-muted-foreground token, and tests pin both classes. - /education is now in e2e/contrast.spec.js and scripts/axe-theme-scan.mjs. - Axe finds no colour-contrast violation on / or /education in either theme.

  • A Google beta that has closed reads "Coming", not available (#770). deriveStatus in frontend/src/lib/certStatus.js used to promote every beta row to active the day after its betaEndDate. That is right for Microsoft, whose betas go live as they end. It is wrong for Google, which closes the beta window and opens GA later: "Beta participants will be notified of their results 4-6 weeks after both the exam window and lab window are closed. You can pass the beta or GA certification" (the Professional Agentic Architect page, re-read 2026-09-29).
  • A row can now say betaClosesBeforeGa: true. From the day after its betaEndDate it renders upcoming, "Coming · beta closed · GA date not announced", or "· GA " when a gaDate is published. It turns active on that gaDate.
  • GCP's PAA carries the flag. So from 2026-10-01, until the row is updated to what Google did, the page no longer offers an exam nobody can book.
  • findStaleStatuses still fails on 2026-10-01 for PAA, AZ-800 and AZ-801. That is the reminder, and #770 carries it.
  • Lab image publishing: the storage-record warnings are gone, and a refused Docker exchange explains itself.
  • Both publish jobs in publish-lab-image.yml now hold artifact-metadata: write, which the attest step needs to write its storage record to GitHub's linked artifacts page. Without it every run warned "Please check that the artifact-metadata:write permission has been included" and "Failed to create storage record" (run 36508154993).
  • That run's Docker Hub job failed at the token exchange with only 400 {"error":"access_denied"}. A new step, Explain a refused Docker token exchange, runs only when the login fails. It requests an ID token for Docker's audience and writes its sub, aud, repository, ref and job_workflow_ref claims to the run summary (never the token), beside the rule the runbook asks the connection to hold, and says whether they match.
  • The runbook's failure table points to that table and to the connection's Failures table in Docker Home.
  • The OFL fonts carry their licence, and a lab pane test no longer races.
  • Google Sans Flex (frontend/public/fonts/google/) and Mona Sans (frontend/public/fonts/github/mona-sans/) are under the SIL Open Font License 1.1, which requires its text to travel with the files. Neither had it. Each now has an OFL.txt copied from its official repository: google/fonts ofl/googlesansflex/OFL.txt and github/mona-sans OFL.txt.
  • Amazon Ember and Bookerly are proprietary and need an owner decision, tracked in #791.
  • In LabPanePage.test.jsx, launcherSays now flushes effects before posting. The page attaches its message listener in an effect, which may not have run when the pane's commit came from the status read resolving outside act. A message sent then was lost, and "shows the launcher's state on the toolbar" failed once in CI on 2026-09-29. Every call is now awaited, and five consecutive runs pass 45 of 45.
  • Register-LabArc.ps1 -Connect no longer fails its own clean-up. After deleting the onboarding secret, the script checked what was left with @(Get-LabArcPasswordCredentials …).Count. That function returns its list as one object (return , …), so the count was always 1, whether the registration held no secret or two. On 2026-09-28 the owner's run connected the lab VPS to Arc and deleted the secret from Entra, then stopped before removing the four vault_arc_* keys from the host's vault, installing the Azure Monitor Agent and associating the data collection rule. The new Confirm-LabArcCredentialsGone assigns the list before filtering it. It also re-deletes and re-reads up to six times ten seconds apart, because Entra's reads can lag its writes. The vault keys are now removed before that check, so a slow Entra read can't leave them behind. Four Pester tests drive it through the real Invoke-LabAzJson with only the az seam mocked, including the one-object case. The same session finished the owner's run with the fixed script, and it ended with the host Connected, the secret gone from Entra and the vault, and the agent sending to dcr-lab-hybrid-prod-cus.
  • Primary buttons use text-primary-foreground, every scope's pair reads, and provider logos read on both themes. --primary is near-black on the light theme, white on the dark one, and the provider's own colour inside a provider's pages, so bg-primary text-white is a blank white box on any dark page outside a provider scope (the two in components/labs were, until #767), and bg-primary text-foreground is unreadable in one theme or both. 37 class strings in 15 files now put text-primary-foreground on a solid bg-primary, hover: pairs included, among them the icon tile on the news and listing pages, whose text-black dark:text-white icon sat in a child element a grep does not see. Measured in Chromium, most were readable where they render today, because a provider scope set a saturated primary; the switch renders the same there and stops being wrong the day the component renders anywhere else. Two scopes' own pairs were under WCAG AA, and changing the class alone would have made them worse: AWS put black on its darkened orange (4.31:1, now white, 4.88:1) and FinOps white on its green (3.02:1, now the site's dark ink, 5.67:1), so the FinOps tools page's "Launch Tool" buttons, 3.01:1 on the dark theme, are 5.67:1 on both. src/primary-foreground.test.js holds both halves: every scope's --primary-foreground on --primary meets 4.5:1 on both themes, and a scan of the parsed source (string and template literals, each branch of a conditional, every cn/clsx combination, each variant state, and the JSX inside a primary surface) fails on any other text colour on solid bg-primary; translucent tints such as bg-primary/10 are allowed, and the allowlist is empty. On the home page, aws.png's white lettering vanished on the light theme and ansible.svg's near-black disc on the dark one; the light theme now shows AWS's one-colour logo as AWS draws it in its own header (aws-on-light.svg; its trademark guidelines forbid recolouring) and the dark theme the reversed Ansible mark (ansible-on-dark.svg, the Simple Icons path, CC0), each with its source noted in the file. The GitHub mark was nearly invisible on the dark theme for a different reason: the pre-renderer resolves the theme as light, and hydration keeps the server's src, so the black Invertocat stayed on a near-black page. ProviderLogo now renders both variants and lets .dark hide one, so the static HTML is right for both themes and the hidden file is never fetched. One table, lib/providerLogos.js, replaces the strip's logo/logoDark fields and the home page's getProviderLogoSrc, whose design cards had been asking for a terraform.png that never existed.
  • Coder sends no usage telemetry. Owner decision 2026-09-28. Coder v2.37.3 ships with --telemetry on by default ("Coder collects anonymized usage data"). lab-host/coder/docker-compose.yml now sets CODER_TELEMETRY_ENABLE: "false": the site's privacy policy names no such processor, and the data would come from learners' sessions. template.test.mjs pins it, and /api/v2/buildinfo reports "telemetry": false after the lab host's next bootstrap.sh run. The lab editor's own telemetry was already off (#764).
  • Coder token steps hold the token in a variable, not on the clipboard. In lab-host/README.md, the owner-check, template-publish and status-token lines all read a Coder token with Get-Clipboard. Each of those lines is itself copied from the README, so on 2026-09-28 copying the status-token line replaced the token, and Coder answered You are signed out to the line's own text. The token is now read once into $t at a masked Read-Host prompt. The new status token is captured in $s and goes to the clipboard only for the paste into the keys page, after which nothing else is copied. The hcw-coder-template-push test pins the new line.
  • CODER_STATUS_TOKEN must resolve. The owner seeded CODER-STATUS-TOKEN on 2026-09-28 (#682). The site's Coder card then listed hcw-lab and a running count, so the token came off EXPECTED_UNRESOLVED in scripts/check-unresolved-secrets.mjs, and an unresolved reference is a failure again. The token expires on 2027-09-28, and #763 is the renewal reminder.
  • The runbook's "Validate on the lab" table matches what visitors see. docs/runbooks/labs-host.md keyed its troubleshooting table on the button's old wording, such as its browser check (Cloudflare Turnstile) is not configured, which #755 took off the public pages. It is now keyed on the API code in the network response, with the exact visitor sentence for each, what it means and what to do, and rows for the states with no code, told apart by the page's data-door and data-check.

  • The lab article drafts link to each other on the site, not to repository files. The six sibling links in docs/content/blog-lab-0*.md pointed at blog-lab-0N-*.md, which works on GitHub only; they now name each article's site URL, /<provider>/blog/<slugified title> (Azure for part 1, Terraform for parts 2 and 3). A title changed before publishing changes the slug, so change these links with it. The importer's test now asserts the drafts carry no repository-link warning.

  • The editor's Problems panel matches CI, and the one open code-scanning alert is fixed. .vscode/settings.json excludes .claude/worktrees/ (agents' temporary checkouts, which fed stale tsconfig.json and workflow diagnostics) and runs flake8 with ruff's default rule families (E4,E7,E9,F), the rules Qlty's ruff enforces; without it the editor flagged 79-column E501 on every Python file CI accepts. hooks/claude_event.py reports a lost audit record on stderr instead of except OSError: pass (CodeQL py/empty-except, alert 363), with the test asserting it. required-inputs.md now records COPILOT_REVIEW_APP_PRIVATE_KEY as MISSING (found absent from the Agents store 2026-09-28), and copilot-setup-steps.yml explains why the Actions extension flags its Agents-store secrets.

  • Register-LabAgent.ps1 pauses for the registration instead of failing. The owner's first live run (2026-09-27) created the Entra app, its service principal, the certificate and the LabAgent grant, then stopped with "Cannot bind argument to parameter 'Lines' because it is an empty string": the registration prompt has blank lines, and Wait-LabRegistration's mandatory [string[]] refused them. It is now [AllowEmptyString()], with a test that feeds it the real prompt.

  • The lab's HTTPS turns on: Caddy's config check loads the token. The caddy role validated the Caddyfile without the environment file the unit reads, so the Cloudflare token was empty during the check and the TLS Caddyfile was refused ("API token '' appears invalid") on 2026-09-27, the first run with a vault. validate now passes --envfile. Proven on the VPS from the branch before merge: failed=0, and https://lab.hybridcloudworks.com serves a Let's Encrypt certificate (valid to 2026-12-26) with HTTP redirecting to it.

  • lab-image: the image vendors every module version the Landing Zone Builder emits, and CI fails when it stops (ADR 0032 decision 5). Found while the lab article drafts were verified (#737). The builder's spokes call avm-res-network-virtualnetwork 0.22.2, and the image carried only 0.15.0, as the connectivity module's child. So the full default build failed terraform init under --network none and passed online. Reproduced on 2026-09-27 against the image built from main: hcw-terraform-validate left spoke_identity, spoke_corp_1 and spoke_online_1 with no vendored version satisfies it (have: 0.15.0), and init failed reaching registry.terraform.io.

The list of builder modules is now written once. Each is an AVM_<KEY>_VERSION / AVM_<KEY>_SHA256 pair in lab-image/versions.env, and the Dockerfile's fetch stage vendors every pair it finds instead of three named calls. 0.22.2 joins the three pattern modules, verified by its release tarball's SHA256, with its one new child, avm-utl-interfaces@0.6.0, pinned by tree hash. 0.15.0 stays beside it, so the two versions coexist as two /opt/avm/<name>@<version> directories. A new AVM_CHILD_CALLS block records which builder module, at which version, reaches which child. vendor-avm.sh prints it with AVM_PRINT_PINS=1 and fails the build when it differs from what terraform get resolved. It also resolves a child at a builder module's exact name@version to that builder copy rather than vendoring it twice. The path layout hcw-terraform-validate reads is unchanged. All sixteen existing child hashes came back identical.

Three checks enforce it. scripts/lab-image-avm-vendoring.test.mjs runs in the required scripts (operations) job with no Docker and no network, and fails when versions.env and avmVersions.js disagree about any module, version or recorded child. Against main's files it reports 23 problems, the first being the missing 0.22.2, and it feeds itself a bumped pin to show that goes red. smoke.sh derives the builder modules from versions.env, checks every recorded child is on disk, and inits each builder module offline. sandbox-check.mjs now validates the builder's real full default build, emitFiles(DEFAULT_STATE) generated at run time, through the agent's own docker-runner.js. Every one of its six registry module blocks must be rewritten, and it ends Success! The configuration is valid. offline. publish-lab-image.yml runs on pull requests that change frontend/src/lib/landingZone/ for that reason. The weekly update-avm-versions.yml pull request now says its scripts check stays red until the child blocks are re-printed. That is by design, because the children of a new release are unknown until the build resolves them.

The builder's "Validate on the lab" prediction (labImage.js, #672) now takes its builder modules from AVM_MODULES directly and holds its one child list to AVM_CHILD_MODULES. The full default build, spokes included, is accepted and sent. A module version the image lacks is still refused, with the module named.

One limit is not the image's. At the agent's default job memory of 256m, that build is OOM-killed during terraform validate (docker inspect: OOMKilled true, exit 247), after an offline init that succeeds. Its sixteen provider configurations each start a plugin process, and 320m passes in 20 s. The host's labs_agent_job_memory is also 256m, and raising it is an owner decision, so sandbox-check.mjs runs that one case at 512m and names the override in its output. Image sizes are unchanged at the README's precision (1.71 GB / 319 MB runner, 2.73 GB / 497 MB full). /opt/avm grew from 5.7 MB to 7.5 MB and each image by 2.1 MB.

  • An unresolved Key Vault reference is never the Cloudflare origin secret. If CF_ORIGIN_SECRET's Key Vault reference failed to resolve, the app read the literal @Microsoft.KeyVault(...) text, which is public in infra/functionapp.tf, and client-identity.js accepted it as proof a request came through Cloudflare, so any Cloudflare-range caller could pick its own rate-limit key. Now that value (and CLIENT_IP_SALT the same way) counts as unset and anonymous routes fail closed. Found by the security review of #738.

  • npm test in frontend/ runs on Windows again: the crash was pdf-parse's canvas addon, not the pool or Node 26 (#720). On Windows the suite died partway with 0xC0000005 (access violation). It did so in 4 runs of 11 under Node 26.10.0, 4 of 12 under 26.5.0 and 6 of 10 under Node 24.21.0, so it was never Node 26's. The pool looked guilty and was not: threads crashed 9 runs of 10 and vmThreads about a third, while forks and vmForks crashed none of 10 each. One file crashed on its own, scripts/update-applied-skills.test.js (4 runs of 6 under threads); all 183 files in src/ together crashed none of 5. It imports pdf-parse, whose pdfjs-dist loads @napi-rs/canvas at import, and 0.1.80, the version pdf-parse 2.4.5 pins, kills the whole process when a worker thread that loaded it exits. A probe with no Vitest in it (one worker loads the addon and exits) crashed 8 runs of 10, and loading it on the main thread only crashed none of 12. Linux never crashes, which is why CI stayed green. Vitest 5.0.2 has no fix for this and needs none.

package.json overrides @napi-rs/canvas to ^1.0.9. The lockfile changes only its own platform packages, and pdf-parse's text for both catalogue posters is byte-identical under either version. With the probe scaled to ten rounds of eight workers, 0.1.80 crashed 17 runs of 17, 1.0.5 8 of 30 and 1.0.9 none of 90. A plain npm test then passed on Windows 30 runs of 30 under Node 26.10.0 (11.0–13.6 s), where --pool=forks, the only way through before, took 31.1–33.7 s. It also passed 10 of 10 under threads, the pool that had crashed 9 of 10, and 5 of 5 each under 26.5.0 and 24.21.0. The pool stays vmThreads on every platform, so CI's Linux run is unchanged. scripts/pdf-parse-canvas-worker.test.js holds the override on every platform. It also runs the probe in a child process, so on Windows a regression fails with its cause named. Against 0.1.80 both of its tests fail. vitest.config.js records all of this beside the pool. frontend/: 191 files, 2,669 tests.

  • tfc-plan-check compares every attribute of an update, not only the Function App's app_settings (#719). scripts/assert-expected-plan.mjs read app_settings alone, so #718's runtime_version "22" -> "24" passed unexamined. Every update in the plan is now compared before against after, leaf by leaf, by a new scripts/lib/plan-diff.mjs. Each difference is named by its path (runtime_version, site_config[0].http2_enabled, app_settings["AzureWebJobs.x.Disabled"]). The allow-list is unchanged: the per-apply RUNTIME_CONFIG_WRITER line and the three azapi replacements. Any other difference fails the check. The repository is public, so the job summary is too, while plan JSON carries secrets in plaintext: a local Terraform 1.15.8 plan showed them in before, after and variables, one left unmarked in terraform_data's computed output. So scripts/lib/plan-report.mjs prints only what is safe to publish. Sensitive values print as (sensitive), unknown ones as (known after apply), a recomputed list or object by its size, and app_settings values as (set). Any string of six or more characters the plan marks sensitive (in resources, outputs or sensitive variables) is masked wherever else it appears, and so are GUIDs and email addresses. runtime_version: "22" -> "24" still prints in full. An update missing its sensitivity markers, or a change with no action list, exits 2. An unrecognised action such as ["forget"], an import and a move are now UNEXPECTED; they used to pass as no-ops (scripts/lib/plan-actions.mjs). The JSON parser's message, which quotes the plan, is no longer printed, and only the first line of the run message is. The job summary names the ref and sha the checker ran from, and a green verdict from a branch other than main no longer says "Safe to confirm". An intended change passes only when the pull request that makes it adds an entry to DECLARED in scripts/assert-expected-plan.mjs, with the exact before and after values. The entry then matches nothing once the change has applied, and is reported as a NOTE to delete rather than a failure. A sensitive or unknown change cannot be declared. check-tfc-plan.mjs prints the declarations, and tfc-plan-check.yml's header says the declarations read are those on the dispatched ref. The tests use a plan-JSON fixture, scripts/fixtures/plan-permanent-diff.json, whose attribute names are held to infra/functionapp.tf. DECLARED ships empty: the live app already runs Node.js 24.

  • Audio from the API plays, and upstream failures reach the page. Two site-wide faults found on the Audio tab on 2026-09-26. The CSP had no media-src, so default-src 'self' blocked every <audio> streaming from https://api-azure.hybridcloudworks.com (Listen & Learn episodes, the Recording Hub, the ElevenLabs live check), which showed 0:00; the CSP now has media-src 'self' https://api-azure.hybridcloudworks.com, held by csp.test.js. And Cloudflare replaces an origin 502 or 504 with its own page, which has no CORS headers, so every handler answering 502 for a failed upstream (Resend, the AI router, the voice preview, ...) reached the browser as "Failed to fetch"; httpRoute now sends those as 500 with X-Upstream-Status, and the handler's reason arrives intact.

  • An ElevenLabs refusal now says why, in ElevenLabs's words. The first live voice listing on 2026-09-26 answered a key that had just read the subscription with 401 and code unauthorized, which the Audio tab showed as "ElevenLabs rejected the key". A permission refusal is now read from detail.code, the legacy detail.status, or a 401/403 whose own sentence names a permission, and every refusal appends ElevenLabs said: "..." (one line, 300 characters, anything shaped like a key replaced with [key]).

  • The lab host's first run survives a freshly booted VPS, and SSH hardening can no longer be lost to a failed run. On the reinstalled VPS on 2026-09-26 the first bootstrap.sh run failed when a first-boot unattended-upgrade held the dpkg lock during the Docker role's package hold, and the sshd drop-in it had already written never took effect: the end-of-play restart handler did not run, and the next run found the file unchanged. Now every apt task waits up to fifteen minutes for the lock (module_defaults in site.yml, DPkg::Lock::Timeout in bootstrap.sh), the three dpkg_selections holds retry for ten, the hardening role flushes its handlers straight after the sshd drop-in, and force_handlers runs notified handlers even when a later task fails.

  • Coder no longer redirects to its access URL behind Caddy. CODER_REDIRECT_TO_ACCESS_URL is false in lab-host/coder/docker-compose.yml: Caddy terminates TLS and forwards plain HTTP, and Coder v2.37.3 answered every such request with a 307 to the https access URL, a loop (measured in #722). A test in template.test.mjs now holds the value. The Coder README's check command uses a glob that works on Node 26, and its expected count is pass 10.

  • The lab host runs the commit it is given, not a pin that lags one merge behind. lab-host/bootstrap.sh pinned HCW_REPO_REF to 04aa9e36, a commit from before lab-host/ existed, so a first run checked it out and stopped at cd lab-host/ansible. The deeper fault was structural: a pull request cannot pin its own merge commit, so the documented "move HCW_REPO_REF and labs_agent_repo_ref in the same pull request" always named the commit before the change, and every lab-host change needed a second pull request to take effect. HCW_REPO_REF now defaults to origin/main. After the clone or fetch it is resolved once with git rev-parse --verify "${HCW_REPO_REF}^{commit}", and that full sha is checked out detached and printed before the play and again as it starts. main's ruleset (a pull request for every change, no bypass actors, strict required checks including ansible-lint (lab-host) and coder (lab-host)) gives the trust the pin gave. HCW_REPO_REF=<sha or ref> still holds or rolls back a host, and an unknown ref stops the run before anything changes. When the resolved commit carries a different bootstrap.sh, the running script hands over to it, so a uv, Python or ansible-core bump takes effect on the run that fetches it rather than the one after. labs_agent_repo_ref in group_vars/all.yml now reads the full sha of the playbook's own checkout, so the agent runs the same commit, including when site.yml is run by hand, and an explicit -e labs_agent_repo_ref=<sha> still holds it back. The npm ci stamp is keyed on the sha the checkout task lands on rather than on the ref string. Stamps left by other commits are now removed first, so releasing a hold reinstalls rather than keeping the held commit's node_modules. Documented in lab-host/README.md ("Which commit runs", with runnable hold and rollback lines), the labs_agent role README and argument spec, docs/runbooks/labs-host.md, lab-host/coder/README.md and infra-lab/README.md step 7. Verified in a privileged systemd ubuntu:26.04 container from a bundle of the branch, with coder and arc off and no vault. The first run was failed=0 and named the sha, and the second was changed=0. After main moved in the bundle, the plain re-run moved the playbook and the agent to the new sha and reinstalled (changed=3). -e labs_agent_repo_ref held the agent at the previous commit while the playbook moved, and releasing it reinstalled. A --check --diff run and a further run were changed=0, an older bootstrap.sh handed over, and an unknown ref was refused.

  • The Azure detail page's catalogue walk no longer fails at random in a full suite run (#640). CertDetailPage.test.jsx's resolves every landing-page slug to a detail page failed once on 2026-09-17, passed on its own, and passed on a re-run of the same suite — the shape that teaches everyone to re-run rather than look.

It was not cross-test interference, which is what #640 assumed and wrote down as the first thing to check. This suite runs on vitest's defaults (pool forks, isolate on), so every test file gets a fresh module registry: a probe that mutated a shared module object and set a global in one file found both pristine in the next. Neither module state nor a stray mock can cross a file boundary here, which leaves file ordering — the third hypothesis — nothing to expose.

It was a budget. The walk renders all 101 Azure exam pages, and each one suspends on a real dynamic import of that exam's study-guide module, so the test goes through Vite's transform server ~51 times — while in a full run that one server is serving four worker processes at once. None of that contention exists when the file runs alone, which is exactly why it passed alone. Measured on the 4-core container: 1.4 s alone, 1.67-2.19 s across four full runs, against vitest's 5 s default. One slow moment covers the rest of that distance, and this suite has them — App.routes.test.jsx's un-mocked lazy route, the same shape, swung 1.05 s to 2.70 s between two runs of the same suite on the same machine. Oversubscribing the CPU reproduces it on demand: 4.75 s in a full run, and, running that file alone, Error: Test timed out in 5000ms. at 5,083 ms — the failure mode named, not inferred.

Both tests now state the budget their own work needs: 30 s for the walk, which is the figure test:coverage had already settled on naming this exact test, and 20 s for the route. The route's was a second and quieter bug — it asks findByText to wait up to 10 s, which under a 5 s test budget it could never be given, so the generous timeout added for CI load had never once applied.

  • Gallery uploads no longer refuse a file for having the wrong name (#631). getGalleryFileExtension took the blob path's extension from the filename while the content type came from file.type. The upload route requires the two to agree (checkUploadMediaType), so a valid file with a name that disagreed was a 415: photo.jfif — what Windows has written for JPEGs saved from a browser for years — and any PNG someone had named .jpg.

It uses imageExtensionFor now, as the submit page's slot uploads did in #632, which derives the extension from the declared type. imageUpload.js had documented this exact trap and carried PUBLIC_IMAGE_EXTENSIONS to avoid it; both call sites were simply not using it. An unsupported type is named outright rather than building a path the route then rejects for an unrelated-looking reason.

  • Stage 3's "Upload Images" half had never worked (#630). Article slot images on /admin/submit went to the content blob container. Every container is private in Terraform — "public" means reachable through the media delivery route, and only the containers in PUBLIC_MEDIA_CONTAINERS are — so the upload route answered 200 with url: '' by design, and the page stored that empty string.

An empty string is falsy, so {slotUrls[key] && …} never rendered the uploaded row. The operator pressed Upload, the spinner finished, and nothing appeared: no error, no link, no confirmation. Downstream, collectSelectedSlotImages drops the slot on .filter(Boolean(item.url)), so canPreview and the article's hero were computed as though no image had been uploaded at all. Only AI-generated images worked — and the reason is instructive: that path has carried if (!imageUrl) throw all along, which is exactly the guard the upload path lacked. It has one now.

Slot images go to covers, the container Terraform describes as "content cover images, served via the media route". Nothing already in content becomes reachable.

SVG is no longer offered, and nothing is lost by that. admin-uploads.js recorded that this picker offered SVG because these went to a private container; in a publicly served one it is refused, since served anonymously an SVG is a scriptable document. An SVG slot upload produced url: '' like every other type, so it never worked either. GIF and AVIF are newly offered.

The blob path's extension now comes from the file's declared type rather than its name (imageExtensionFor), which is half of #631: the route requires the two to agree, so photo.jfif — what Windows writes for a JPEG saved from a browser — was a 415 on a perfectly valid file.

The write is module-level over a state bag, as linkWrites.js is, so it has seven tests without mounting a 2,859-line page.

  • Gallery uploads produced images nothing could display (#602). Manual uploads on /admin/gallery went to the content blob container. Every container is private in Terraform — "public" means reachable through the media delivery route, and only the containers in PUBLIC_MEDIA_CONTAINERS are — so the upload route answered 200 with url: '' by design, and the page wrote that empty string into the gallery record. Nothing failed: the image stored, the record saved, and the result was a row that rendered no thumbnail and handed an empty hero URL to the submit page through "Use this image".

Uploads go to covers now, which Terraform describes as "content cover images, served via the media route" — exactly what a gallery image is used as. Nothing already in content becomes reachable; the alternative of adding content to PUBLIC_MEDIA_CONTAINERS would have exposed everything already sitting in it, which is the disclosure decision blob-paths.js warns about, and it is not taken here.

Two things stop this being silent again. The page now refuses to create a record when the route returns no URL, so a container that cannot serve fails loudly at the upload instead of persisting a broken row. And the file picker offers only the five types a publicly served container accepts, naming any rejected file rather than letting the route answer 415 for the whole batch — covers refuses SVG, because served anonymously it is a scriptable document.

ImageGalleryPage had no tests; it has six, including the empty-URL case that is the defect itself. Writing them surfaced a second bug in the fix: the rejection message was being set on deleteError, which fetchGallery clears on every refresh — and the refresh runs immediately afterwards, so the message was wiped before it could be read. It has its own state now.

Changed

  • The App installation token is revoked when the job ends (#641). It was living out its full hour after the work that needed it had finished.

revokeInstallationToken calls DELETE /installation/token, authenticated by the token itself rather than the App JWT, and the two PR-opening workflows call it in a step with if: always() — so a failed push or a failed gh pr create still cleans up, which is the case where a live credential matters most.

It cannot fail the job. The script returns 0 whatever GitHub answers, and counts a 401 as success because an already-invalid token is the end state wanted. A run that did its work should not turn red over cleanup.

copilot-setup-steps.yml deliberately does not revoke, and now says so at its own mint step. It hands the token to Copilot's MCP server, which reads it after that job finishes; revoking there would leave code review holding a dead credential. Making all three consistent would have broken it.

This is what actions/create-github-app-token does in its post step and the one thing the in-repo minter did not. The header's reason for preferring the in-repo version is also corrected: it claimed the action's commit SHA could not be resolved when it was written, which was a limitation of that session rather than an argument. The real reason — no third-party code in the job holding the most powerful credential in the pipeline — still stands and is what the header says now.

Five tests, 29 in that file.

  • Six secrets-outside-env findings fixed, and the triage record corrected (#567). The three workflows that never adopted the repository's own convention now scope their secret-holding jobs to environment: automation.

These had been reported as resolved and were not. The 2026-09-16 verification on #567 said zero zizmor findings at all three personas. Six mediums stood at the auditor persona, untriaged. A persona is not a severity filter — the counts table now reads 42, not 36.

It was never a new decision: production gates the two deploys, copilot the Copilot setup, github-pages the docs. These three simply never followed. It is also what Microsoft's MCSB v2 DS-3 asks for, at criticality Must have.

automation and not production, deliberately. These jobs open pull requests and read a Terraform plan rather than deploying, and two run on a schedule. A required-reviewer rule would leave a scheduled run waiting up to 30 days, not failing — so the environment must stay ungated, which the triage record now says outright.

The repository's own OIDC guard caught an imprecision, which is what it is for. scripts/oidc-subjects.test.mjs failed, claiming the Azure login in publish-content-manifest.yml would now present an :environment: subject and fail with AADSTS700213. It scanned the whole FILE for environment: and for the client-id without asking which job held either. GitHub composes the subject per job, and the login job declares no environment. The scanner is per-job now, which is strictly stronger: the old shape could not see a file where one login job names an environment and another does not.

Nothing in a job that calls azure/login was touched.

Half of this is the owner's, and the order is load-bearing. Declaring the environment changes nothing by itself — a repository secret is still readable from a job in any environment. The control arrives when the secret becomes an environment secret, and that must happen after this lands: reverse the order and the job resolves an empty secret, reports "not set — skipping" and passes green while the automation is dead.

Also recorded so nobody re-researches it: TFC_TOKEN has no OIDC alternative. HCP Terraform's API takes a bearer token and nothing else.

  • The Stage cards move out of SubmitUrlsPage, finishing #634. The page is 2,916 lines to 775 across five PRs, and no function in it or its twelve modules is over the complexity gate any more.

All four cards, the page chrome and the section-block data are now their own modules. StageTwoCard (CCN 10) and StageThreeCard (CCN 13) came apart into sub-components on the way: Stage 2 drew the same removable-row list three times, for files, document URLs and KB articles, differing only in the remove handler.

Moving them exposed five references the new files never imported — PROVIDER_OPTIONS, BLOG_LANDING_ZONE_OPTIONS, getPublishTargetLabel (twice), isSupportedDocumentUrl, and a namespaced imageStage. call — and npm run lint reported 0 errors for every one. ESLint does not run no-undef on these files, so an unbound identifier is a ReferenceError at render time and silent everywhere else. Each would have broken the admin page the first time an operator opened it.

So the cards now have smoke render tests. That is the same failure as #629, where a rename left three <select> blocks pointing at dead names while lint, build and every unit test passed because nothing rendered them.

Qlty raised four similar-code findings on the extracted contentBlocks.js, and they were right. Twenty section-block entries each wrote their heading out three times — as title, as heading, and again at the head of template. A block builder derives the last two now.

That closes a latent bug as well as the duplication: the readiness checklist matches a section by its heading while the button that inserts it is labelled by its title, so an edit to one that missed the other would quietly stop the checklist finding a section visible in the draft, and the draft could never be marked ready. All twenty already held the invariant; it is now unbreakable and tested. The file is 182 lines to 65, and the regenerated values were proved identical to the originals before the swap.

Tests 175 to 193.

  • Stage 1's machinery moves out of SubmitUrlsPage (#634). Last of the machinery; only the Stage cards' markup is left.

Three modules: pageMeta.js for the pure derivations, promptStage.js for the prompt library, and builderSnapshot.js for the sessionStorage draft. loadPromptLibrary (CCN 16) and inferProviderFromUrl (CCN 11) are gone — the first split so its four cancellation checks are no longer tangled with its error handling, the second replaced by a table.

Three parallel switches became one table. getPreviewSection, getPublishTargetLabel and getPromptLibraryPagePath each carried their own copy of the content-type mapping, so adding a type meant three edits and three chances to miss one.

The prompt library's status lines were each built twice — once in the load effect, once in the matching change handler — so the wording could drift between the page loading an assignment and an operator making the same one by hand. One definition each now.

The snapshot's read and write lived apart, a module-level reader and an inline effect, which is how two lists of the same fields end up out of sight of each other. Both are in one module with a test that fails if a setter is added to one list and forgotten in the other.

_hasSelectedImage was declared once and read nowhere. That is the second dead declaration found in this file hiding behind a leading underscore, after _MAX_STAGE_TWO_SUPPORTING_FILES in #636 — the no-unused-vars exemption that makes the prefix useful also makes dead code invisible.

Also unified: the uploaded-wins-over-generated rule, which #637 left written twice in imageStage.js. One slotImageSources now, read by both. All 31 of that module's existing tests pass unchanged, which is the proof.

Tests 114 to 175. The page is 2,175 lines to 1,784, its total CCN 268 to 164, and its returns 20 to 11.

  • Stage 4's persist machinery and StageFourCard move out of SubmitUrlsPage (#634). Third of four.

The create payload and the two ways to save are now in components/admin/submit-urls/persistStage.js, and StageFourCard — at CCN 20 the single most complex function in the file — is its own module, split into seven pieces. Its complexity was never one hard decision; it was fourteen small JSX guards across 215 lines.

Stage 4 is the only stage that writes anything durable, and until now its payload had no test at all — it was assembled inline in a click handler. It has twenty-four, including one that pins Live: false and approvedForBlog: false: saving is not publishing, and if those ever default true a save silently becomes a publish.

Nothing Stage 4 does has changed. persistContentItem still returns null on every failure path including the readiness gate, because both callers branch on it — a falsy id must not produce a success message or navigate away. Both save buttons are still disabled while either is in flight, so the draft cannot be written twice. The duplicated Title/title and Content/content/postContent casings are the stored schema's and are kept, now with a test saying why.

Also moved: the image-selection helpers joined imageStage.js, where they belong — which makes visible that the uploaded-wins-over-generated rule is written twice in that file. The two agree today; unifying them is a behaviour-adjacent change left for the last stage rather than slipped into a move.

Tests 76 to 114. The page is 2,587 lines to 2,175, its total CCN 357 to ~278, and its returns 25 to 20.

  • Stage 2's draft machinery moves out of SubmitUrlsPage, with tests (#634). Second of four.

Source URLs, supporting documents and the generate-draft call now live in components/admin/submit-urls/draftStage.js as module-level functions over a state bag, as Stage 3's did in #635. handleSubmitDraft (CCN 18) and handleSupportingDocumentUpload (CCN 15) are gone, split into pure builders (effectiveSourceUrlsFor, draftRequestFor, normalizeDraft) and thin writers.

Nothing Stage 2 does has changed. The supporting-document read stays all-or-nothing, so one unsupported or oversized file in a batch adds none of them rather than a partial set the operator then has to reconcile; uploads and document URLs still share one five-item budget rather than having one each; and a URL still sitting unadded in the input still counts toward the draft call, so typing one and pressing Generate works without pressing Add.

Two incidental removals. _MAX_STAGE_TWO_SUPPORTING_FILES was declared once and read nowhere, surviving only because the leading underscore exempts it from no-unused-vars — and it was a stale duplicate of the constant that does the work. The six useCallback wrappers around these handlers bought nothing either: no consumer in this file is memoized and none of the six appears in a dependency array, so their stable identity was never read.

Tests 31 to 76. The page is 2,770 lines to 2,587, its total CCN 417 to 357, its returns 37 to 25, and functions over the CCN gate 8 to 6.

  • Stage 3's image machinery moves out of SubmitUrlsPage, with tests (#634). First of four; the stages are the seams, and the state block already named them.

SubmitUrlsPage.jsx is a 1,088-line component holding 54 useState, 44 returns and nine functions over the repository's CCN gate — and eight tests, all covering the one function #632 had already pulled out. Two of its features shipped broken and undetected under exactly that ratio: slot uploads never worked at all (#630), and a valid .jfif was a 415 (#631). Both were found by reading the code during an unrelated fix, not by a test and not by a report.

Upload, generate, remove, resolve and gallery-delete now live in components/admin/submit-urls/imageStage.js as module-level functions over a state bag — the linkWrites.js shape the hub series established and that uploadSlotImageFile already used here. The page keeps the state and five one-line wrappers.

Nothing Stage 3 does has changed. The count in the "Generated N image slot(s)" summary is still taken from what was collected rather than from what was requested, so an abandoned run still reports what it actually stored; removeGeneratedImage still refuses to clear a slot locally when the server-side delete failed, which would otherwise leave an empty slot backed by a live blob.

Tests went from 8 to 31. The page is 2,916 lines to 2,770, its total CCN 472 to 417, its worst function CCN 19 to 5, and its returns 44 to 37; the new module has nothing over the gate.

  • The Stop guard's blocking path is tested, and the lint config stops measuring the wrong thing (#588). PR B of the burn-down.

hooks/claude_event.py had one assertion in CI: that it exits 0 with no state. That is the fail-open half. The blocking half — an armed workflow with missing handoffs, where the hook is supposed to refuse the Stop — had no coverage at all, so a regression there would have been silent in the only direction that matters: the guard quietly never fires and the session ends on a workflow whose audit trail claims work that never ran. hooks/test_claude_event.py covers both halves in 13 cases, written and run green against the old code before any of it was touched.

The nine-branch fail-open cascade inside main is now one function per question — read_event, audit, armed_workflow_id, validation_fails, block_decision — each answering "allow" for everything it cannot read, so the contract is stated once rather than once per condition. main is six lines. In tooling/workflow.py, the state tree's file names are four path helpers instead of literals repeated at every call site (a rename could be applied to the write and missed on the read, and the tool would report "workflow not found" against a workflow it had just created), and main is a dispatch table rather than a chain of seven returns.

.editorconfig was measuring indentation in files where indentation is not a style choice. Of the findings counted on #588, 468 were Markdown — where four spaces opens a code block, three continues an ordered-list item and two nests a bullet — and 65 more were PowerShell comment-based help aligning its continuation lines under .SYNOPSIS, which is what Get-Help renders. Both now state no opinion on indent_size. .vscode/ states none on indent_style either: VS Code wrote tasks.json with tabs and the other three with spaces, so asserting tab to clear the first simply moved the same finding onto the rest.

Measured with a local reimplementation of the checker's rules (its own binary cannot be fetched in this environment): 576 findings before, 13 after. The 13 that remain are aligned SQL inside template literals and aligned docstring continuations — deliberate formatting in code that Prettier and ruff already govern, and not worth a blanket exemption for every .js file.

  • Workflow hygiene: the token is granted where it is used (#588). zizmor at its pedantic persona reported 34 findings across the 20 workflows; it reports none now, at all three personas.

Five High excessive-permissions. deploy-functions, heal-computed- properties, monitor-functions-registered, monitor-unresolved-secrets and verify-alert-state each granted id-token: write at workflow level. Each has exactly one job, so moving the grant onto that job removes no privilege that exists today — it stops a second job added later from silently inheriting the OIDC token. Stated plainly because a "High" here is hygiene, not a live hole.

Seven workflows had no concurrency block. The monitors and the healer never cancel a run in flight: a cancelled six-hourly check is a skipped check rather than a check done later, and heal-computed-properties writes cp_sortDate onto documents, so an interrupted run stops partway through a container. The two pull-request checks do cancel, since a new push supersedes the run it replaced; repository-policy cancels on pull requests only, because its run on main is the record that what landed satisfies the policy.

Eleven undocumented-permissions and five anonymous-definition. The permissions now carry a comment on each entry — zizmor reads a trailing comment, so several blocks that already had a paragraph above them were still reported. dependency-review gains name: dependency-review, deliberately identical to its job id: that job runs on pull_request, so its check-run name is a status context the ruleset can require, and renaming it would leave every pull request waiting for a context that is never reported (ADR 0026, T-523).

Six secrets-outside-env findings remain at the auditor persona only. Binding those three workflows to a GitHub Environment is a deployment- configuration change rather than workflow hygiene, so it stays on #588.

Added

  • Linkie Hub and Labs Hub: finished to the Newsletter Hub standard (#577). The last two hubs in the series. /admin/linkie is 127 lines over Links / Analytics / Settings (components/admin/linkie), and /admin/labs is 97 lines over Dashboard / Jobs / Console / Agents / Settings (components/admin/labs), down from 1,076 and 693.

Labs gains two tabs, and they are the point. The job table was the bottom third of Dashboard with its columns stopping at the exit code, so a job that failed yesterday could be seen to have failed and not why: job.output was rendered only for whichever job the Console had just submitted. Jobs rows expand to it. And the live connection state was the top card of Setup, under six install steps, so "the agent is disconnected" was answered with provisioning instructions whether or not anything needed provisioning. Agents separates the three states fleetState names: online, stale — an agent that has connected and stopped, which wants systemctl restart and not a reinstall — and none.

Linkie's Connection tab became Settings with ?tab=connection redirecting, and ?profile= survives a tab change: every posts and analytics path is profile-scoped, so a link naming a tab without its profile would land on whichever profile sorts first.

One live-page rule, shared by the three surfaces that must agree. isLiveRecord and getLiveUrl existed in LivePagesPage, the Social Hub and Linkie. isLiveRecord genuinely matched byte for byte; getLiveUrl did not — the Linkie copy was the ||-chain-into-a-nested-ternary that Qlty flagged on the Social Hub's copy in #623, carried unnoticed in a second file while the first was fixed. They are lib/livePages.js now. One difference is deliberately not resolved: LivePagesPage has a getContentPublicPath fallback the hubs lack, which changes which pages an operator can post about, so it keeps its own resolver and the module header records the divergence.

Labs had no tests at all, including the agent-staleness rendering T-309 was filed against; it has 12 page tests and 16 helper tests now. Writing them found labsView.js calling toMillis without importing it, and found that four structural guards in labsPolling.test.js had gone blind — they read LabsPage.jsx as text, and this refactor moved their subject into components/admin/labs/, so all four passed vacuously against the 97-line page left behind. They read the whole hub directory now, and assert the glob matched something.

  • Recording Hub: organised by duty, not by provider (#576). /admin/recording-hub had one tab per SERVICE — Podcast, and Plaud with three sub-tabs of its own — so "where do I approve a transcript" depended on which service produced it, and the OAuth setup sat two clicks inside a working tab. It is five duties now at the Newsletter Hub standard: Recordings, Transcripts, Episodes, Distribution, Settings. Providers are named in the header and inside the tabs. The page is 118 lines and the two provider components (1,350 and 598 lines) are gone, re-cut into components/admin/recording-hub.

The page had no ?tab= at all. It held the selected tab in useState, so no link could name a tab and Back could not leave one — the first thing resolveTab fixes. Both provider ids redirect: podcast to Transcripts, and plaud to Recordings rather than Settings, because that tab opened on its Library sub-tab and the recordings are what such a link was for, not the OAuth setup two sub-tabs along. The Plaud sub-tab ids and the host/feed words redirect too, and own-properties-only lookup keeps ?tab=constructor from reaching Object.prototype.

Distribution is new, and is the reason the split was worth doing. The RSS.com host record was one line tucked under each transcript, so "did anything fail to publish" meant scrolling the whole list and reading every row. The same records are now grouped by state with failures first and the retry beside them. hostState decides the grouping and HostLine renders by the same four states, rather than two if chains that could drift.

Transcripts and Distribution share one read, because approving a transcript is exactly what creates the record Distribution shows; two reads could disagree about one transcript, and a publish that a stale list says never happened is the worst thing that tab could say. The connection read joins them because Settings connects what Recordings needs. Race-safe per #555: a generation on reads, an in-flight guard on writes, and the unknown versus disconnected distinction the page already documented — a check that could not run is not a check that ran and said no.

fmtDate was two functions. Each provider tab carried its own copy, identical except that only the Podcast one guarded Number.isNaN, so the same malformed timestamp rendered as '' on one tab and "Invalid Date" on the other. The guarding one won.

Plaud routing to ContentForge and transcript publishing are unchanged; the 24 existing end-to-end assertions are unchanged except where the split moved what they read, and 33 more cover the tab resolver, the helpers, the new navigation and Distribution.

  • Complexity Delta: report the complexity a pull request adds, not the complexity already here (#624). A new Complexity Delta workflow and scripts/complexity/.

Why it is a delta gate. Measured at the head of main on 2026-09-16, lizard finds 479 functions over CCN 8 — 322 over 10, 117 over 15, 43 over 20, 9 over 40, with 253 of them in frontend/ and 198 in functions/. A whole-tree run at CCN 8 is therefore red 479 times over on its first pull request and on every one after it, INCLUDING pull requests that remove complexity. A check that cannot go green is a check nobody reads, and this repository already has one of those in monitor-unresolved-secrets.yml. So the question asked is the one with a green answer available every time — did THIS pull request make it worse — and the 479 are left to be worked down separately.

Every function in a changed file is compared against itself at the merge base. A function that is new and over the threshold, or over it and higher than it was, is reported; one that is over it but no higher is carried without complaint; one that got simpler is reported as a win. Carrying is the point — editing one line of a CCN 30 function must not be refused, or the backlog becomes unfixable by the thing asking for it to be fixed. A total-CCN line covers what per-function verdicts miss, since twenty new functions at CCN 5 trip no verdict and are still 100 CCN of new branching.

Functions are matched by (path, name), never by line number, and one that is new at its path is first paired with a same-named function that left elsewhere in the same diff. That pairing is not polish: lizard's JavaScript tokenizer drifts with file context, and getLiveUrl measures 7 inside the 1,551-line SocialHubPage.jsx against 10 as the byte-identical body in socialView.js — the only textual difference being the export keyword. Without it, splitting a large file reports every function in the new files as newly added complexity, taxing the one refactor this repository most needs to stay cheap. Anonymous functions never pair, because lizard names every arrow function (anonymous) and matching on that pairs the 40th callback in one file with the 3rd in another.

Report-only by default, so findings arrive as annotations on the changed lines and the check stays green; COMPLEXITY_ENFORCE arms it and COMPLEXITY_THRESHOLD moves it. The trigger is not path-filtered, for the reason ADR 0026 records.

Two notes for whoever reads this next. lizard . -C 8 -W does not do what it reads like: -W is --whitelist and takes a filename, so that command exits 2 from argparse before measuring anything — lowercase -w prints warnings, and lizard already exits 1 on any. And the gate is held to its own standards: render came back at CCN 20 and compare at 14 on the first draft, and then Qlty put the single module at file-complexity 72 — the gate's own file being the thing the gate exists to prevent. It is now four modules (model, measure, compare, report) behind the CLI, none over CCN 8. Nineteen unit tests cover the comparison, now without needing lizard installed to import it, and run as a workflow step before the gate judges anything.

CodeQL contributed the other half. It read the field named ccn as a credit card number — its "private" heuristic covers card numbers, and the message interpolating two of them produced two "clear-text logging of sensitive information" alerts on one line. The field is now complexity, which is the clearer name regardless, and the flag is --threshold. Its third finding was real: a print in an except ImportError at module scope runs during import, so the missing-dependency report moved into main.

.py joined the wiki-pointer guard's TEXT list in the same change. That guard fails when a tracked extension carrying five or more files is neither scanned nor explicitly exempt, and scripts/complexity/ took the repository from four Python files to six — the test doing exactly the job its comment claims, making the next language added a decision rather than an oversight. Python here is prose: three of those six open with a docstring telling an operator what to do, so a pointer at the retired Wiki inside one would be as dead an instruction as a pointer in a runbook.

  • Cloud pricing comparison, Phase 3 backend: price history, a newsletter section and "Explain this number" (#613). Three parts on the same refresh.
  • Price history and change detection (lib/cloud-tools/history.js). After each region's cache document the refresh-tool-pricing job writes one tool_price_history snapshot per region per UTC day (history:<region>:<YYYY-MM-DD>, a map of <serviceId>:<provider> cells with pricePerUnit, unit, sku, currency, source: live|baseline) and diffs it against the newest snapshot at least 7 and at least 30 days old — found by point reads of day ids N..N+6, never a query — into price-changes:<region> in tool_service_cache: { region, asOf, windows: { '7d': { since, sampleDay, items }, '30d': {…} }, sampleDays }, each item { serviceId, label, provider, unit, sku, from, to, deltaPct }, live-against-live only (a live/baseline flip, a unit change and an unchanged price are not changes), largest move first. A history failure is logged and recorded in the run summary; the cache document stands. GET public/cloud-tools/price-changes?region= serves the document as a four-field projection; before any comparison exists it answers empty windows and sampleDays: 0, cached a minute. The container is generated from the manifest (regenerate, /id, 400-day TTL) and classified regenerable for the export.
  • Newsletter section "Cloud price changes" (id cloud-price-changes, lib/newsletter/sections.js): the 7-day window of price-changes:us-east-1 — region fixed, an email has no picker — as "AWS · Virtual machines: $0.192 → $0.201 per hour (+4.7%)" linking to /tools/comparison?region=us-east-1, rendered through the same item shape and renderer as its siblings. With no moves the section is omitted from the issue entirely; there is no "no changes" filler. On by default, as the registry rule (#557) makes every new section; the Newsletter Hub's Content form lists it to turn off or cap.
  • "Explain this number": POST public/cloud-tools/explain (lib/cloud-tools/explain/, functions/cloud-tools-http.js), anonymous, takes the priced scenario the page already shows ({ region, scenarioId, scenarioLabel, extras, egressGb, results: [{ provider, total, base, segments, unavailable }] }, 8 KB, validated field by field, unknown keys refused) and answers { explanation: { text, model, generatedAt, cached } } — two paragraphs from the AI router's default chain on which provider is cheapest and by how much, what the extras add, and one thing that could flip it, from the numbers given and nothing else. Bounded four ways: cached by a hash of the canonical body for 7 days (a shared link costs one model call), 5 per hour per Cloudflare-verified client on the submission counter (429), 200 per UTC day across everyone through a compare-and-increment counter (503 "paused for today"), and 503 "not available" before any counter moves when the router cannot serve it — no key, or the new Pricing explanations toggle on AI Engine → AI features switched off, which is the owner's off switch for the one anonymous AI call. Anything URL-shaped is stripped from the model's text before it is stored.
  • Cloud pricing comparison, Phase 3 frontend: price changes, explain, second region, article embeds (#613). Four additions to /tools/comparison and one to articles, against the two Phase 3 routes the backend PR adds: GET public/cloud-tools/price-changes and POST public/cloud-tools/explain. Until that PR is deployed the card and the button show the server's error rather than a number.
  • Price changes (pages/tools/PriceChanges.jsx), under the table: what moved in the region's list prices over the last 7 or 30 days, one row per move — "AWS · Virtual machine $0.192 → $0.201 /hour ↑ +4.7%", rises and falls coloured and signed — with "No changes in the last 7 days" and "No history yet" as their own states and an as-of line. Follows ?region=.
  • Explain this comparison (scenario/ExplainButton.jsx): posts the computed comparison (provider, total, base, segments, unavailable — no lines, so it stays under the server's 8 KB cap) and shows the two paragraphs under "Generated by AI from the numbers above — check the sources before relying on it", with the model, the time and a note when the answer was cached. Only on a click, never on load; disabled while any provider is unavailable; 429 reads "try again in a while", 503 shows the server's sentence, anything else offers Retry. The answer is keyed to the numbers it explained and goes away when they change.
  • Compare with (scenario/CompareRegion.jsx): a second-region select on the results, state in ?compare=, that fetches that region once and puts "US West: $782.10 (+10%)" under each provider's total and "Cheapest in US West: Google Cloud" under the list; loading and failure inline. An id the API did not list, or the region already shown, is ignored and dropped from the URL on the next change; a previous region's answer is never shown as the new one's. lib/pricingScenarios/compare.js holds the arithmetic; share.js carries compare= with the rest of the scenario.
  • Article embeds (components/content/PricingScenarioEmbed.jsx): a fenced block with language pricing-scenario whose body is a scenario query string renders the scenario card read-only inside a post, with an "Open in the comparison tool" link. Wired through the code override every markdown renderer shares (components/shared/CodeBlock.jsx), so blog posts, RichTextBody and the ContentForge preview all get it; the card is a React.lazy chunk a post without the fence never loads, and a post with it pre-renders and hydrates cleanly (test with renderToString → hydrateRoot). Documented in docs/content/blog-template.md.
  • Cloud pricing comparison, Phase 2: scenarios and extras (#613). A "Price a scenario" card above the Phase 1 table on /tools/comparison: pick a shape (Static site + API, Three-tier web app, Event-driven backend, Data platform, Kubernetes platform), edit its quantities, and see a monthly and yearly bill per provider as one stacked bar each — the base cost and every extra as its own segment, the cheapest highlighted and the others marked "+N% vs cheapest". Priced from the same cached list prices as the table, in the same region.
  • Extras from a pull-down: backup, zone redundancy, a DR level (pilot light, warm standby, active-active — one at a time) and a 1- or 3-year commitment (drawn as a dashed reduction at the end of the bar), plus an egress select (100 GB to 20 TB). Every extra is priced off the same eight meters, so nothing on the page comes from a number the price list did not supply.
  • The maths is visible: "Show breakdown" lists every line as quantity × factor × unit price, and "How this is calculated" renders each extra's rule and the ASSUMPTIONS table — every multiplier per provider, with a link to the provider page it was read from (frontend/src/lib/pricingScenarios/).
  • Never silently zero: a provider with no price for a service the scenario uses shows "unavailable: " instead of a total, and a total with any catalogue line carries the "catalogue price" badge.
  • Shareable URL: the scenario, extras, egress and edited quantities live in the query string beside ?region=, and Copy link copies it (or shows it to copy by hand when the clipboard refuses).
  • Cloud pricing comparison, Phase 1 (#613). /tools/comparison — "Pricing Comparison" in the Tools menu, formerly "Pillar Comparison" and a Coming Soon page — is a real page: one row per service shape (VM, serverless, object storage, relational and NoSQL databases, Kubernetes, messaging, CDN) with a list price per provider on AWS, Azure and Google Cloud, read from the daily server-side cache through GET public/cloud-tools/pricing?region=.
  • The region is in the URL (?region=), so a comparison is a link.
  • Each cell says what it is: the price to four significant figures, the provider's SKU beneath it, a "catalogue price" badge when the provider's price list could not be read and the site's own figure stands in, and "unavailable" when there is neither. The cheapest cell on a row is highlighted, and every cell links to the provider's pricing page.
  • An as-of line says when the prices were refreshed, or that the cache is stale, or that it has never been filled.
  • Integrations Hub → Cloud → Cloud pricing cache: a card whose test reads the cache and reports its age and its live / catalogue / unavailable counts, and whose Refresh now enqueues the refresh-tool-pricing job and re-reads once it finishes.

  • Scenarios, monthly estimates and the remaining extras are Phase 2.

  • Cloud Tools pricing: the timer, the job and the public read (#613, backend half). The ported live-pricing library in functions/src/lib/cloud-tools/ had no caller; now it has three.

  • refresh-tool-pricing job (functions/src/functions/cloud-tools-jobs.js, lib/cloud-tools/refresh.js): for each of three region options (us-east-1, us-west-2, westeurope — pricing/regions.js, asserted against PROVIDER_REGION_MATRIX) fetches all eight catalog services from AWS, Azure and GCP one service at a time and upserts one tool_service_cache document per region: { id: 'pricing:<region>', region, refreshedAt, ttlMinutes: 1440, services[], counts { live, baseline, unavailable } }. A region failing is logged and the others continue. Editor-enqueueable ({ type: 'refresh-tool-pricing', payload: {} }, optional payload.regions), ten-minute budget, failure to Telegram like the export job.
  • refreshToolServiceCache timer, daily 02:00 UTC, behind FEATURE_FLAG_REFRESH_TOOL_SERVICE_CACHE and the schedulers master switch; enqueues the job rather than running 24 lookups inside a timer. Catalogued in infra/functionapp.tf and allowed by the enabled_timers validation; the twentieth catalogued timer and the twenty-first registration, and every prose count was moved with it. The once-per- process "disabled — skipping" Warning is now lib/timers/flag-gate.js, shared by schedulers.js and this timer.
  • GET /api/public/cloud-tools/pricing?region= (lib/cloud-tools/public-pricing.js): one point read of the region's document, Cache-Control: public, max-age=900, freshness reported through cacheFreshness and never repaired on the request path. No document yet is a 200 with refreshedAt: null, stale: true and every cell unavailable, so the page renders "not refreshed yet" rather than an error. Unknown region is 400.

  • Owner decision 2026-09-15: the two KNOWN_UNIT_MISMATCHES services (compute-serverless, database-nosql) get no baseline fallback — fallbackBaselineFor answers null for them, so a live miss is an absent row counted as unavailable, never a number in the wrong unit. Recorded in the baseline.js header and pinned by a test.

  • Linkie posts can carry an image (#501). On /admin/linkie, Add a Post has an optional image row. The image can be uploaded from the computer into the public covers container (PNG, JPEG, WebP, GIF or AVIF, up to 15 MB), or chosen from the image gallery, and shows a preview with a Remove button.

  • Push Published Content sends the article's cover image.
  • Current posts show their thumbnail.
  • The field is thumbnail_url, confirmed from a real Linkie post response. It is sent only when an image was chosen, so posts without one are unchanged.

  • Shared upload code: file reading and upload live in lib/imageUpload.js, which the Image Gallery page now shares.

  • Social Hub: tabs by duty, with an Accounts tab and a Published calendar (#575). /admin/social had four tabs already, but the page was one 1,551-line component holding the Publer client, the view helpers and every tab's JSX. It is now five tabs at the Newsletter Hub standard — Compose, Queue, Published, Accounts, Settings — and the page is 93 lines. The panels, the Publer client, the shared pieces and the pure helpers live in components/admin/social.

Accounts is its own tab, and Settings answers one question. The connected profiles were a card inside the Connection Settings tab, below the credential tile; they are the list an operator checks before composing, not something set once, so they have a tab. What is left on Settings is whether the proxy can use its credentials — which is all the browser can honestly say, since the key lives in Key Vault and the proxy never sends it back (FINDING-04) — plus where each of the settings this hub does not own is actually set.

Published groups by the day a post went out and reports per-account outcomes. A flat list of posts is not a history. Posts Publer returns with no timestamp are kept in an Undated group rather than dropped: a post with no stamp still published, and losing it from the record would be the worse error. postResults reports only the state and error Publer actually sent — an account it said nothing about is shown with no state rather than as a success, which is the #463 mistake (a job whose every account failed reported as a success) one layer up.

Each tab reads its own data, so one failure cannot blank another. The accounts read and the live-pages read are separate hooks that Compose, Published, Accounts and Settings call for themselves; a missing Publer key leaves the content picker usable, and a failed content read leaves the account list visible. There is nothing here for two tabs to disagree about — unlike the Listen & Learn Hub, no tab writes state another tab shows — so this is the right side of that trade. useRecentContent carries a generation, so a superseded read paints nothing, and a failed read empties the list rather than leaving rows beside an error saying they could not be read (#555). The Queue's deletes are guarded in flight: a second click while the first is unanswered is ignored rather than sent twice.

Deep links are ?tab=, and now they are validated. The page read searchParams.get('tab') || 'compose' and tested it against four literals, so an unknown id rendered the header with no panel under it at all. resolveTab reads own properties only, so ?tab=constructor cannot reach Object.prototype and hand a function to the panel lookup. connection — the word on the old tab bar, which read "Connection Settings" over an id of settings — lands on Settings, and profiles, calendar and scheduled land where their content went. All four ids the page shipped with still resolve to themselves.

Composing and scheduling are unchanged. The two endpoints, the delete form, the job poll and the envelope reading are the #463 and #397 code moved verbatim into social/publerApi.js; their tests moved with them and every assertion is unchanged. The tab bar is now the shared HubTabs, so the Social Hub gets the role="tablist" contract and arrow-key navigation the other hubs have.

  • Listen & Learn Hub: tabs by duty (#574). /admin/listen-and-learn was one scroll — the generate form, the grounding panel, the list of sets and the chosen set's episodes, stacked — and is now a tab per duty at the Newsletter Hub standard: Generate, Review, Published, Settings. The page went from 622 lines to 99; the panels, the episode card, the voice field and the view helpers live in components/admin/listen-and-learn.

Review and Published are one component and two configurations of it. The first draft was two files differing in a filter, an empty-state sentence and a doc comment, and identical for thirty-five lines either side — the duplication #588 counts, and two places to change the approve button. They are now SetEpisodesTab with a keep predicate. Failed episodes sit on Review rather than Published: they are work needing a decision, and the card already explains the failure.

Three Qlty structure findings blocked the first push and are fixed at the source rather than silenced. useListenAndLearn came back at complexity 34 with 10 returns, so the reads and writes are now module-level functions over one state bag, each with a single exit — the shape useCertifications documents for exactly this reason, and the reason the hook stays inside the budget as it grows. queuedMessage carried 6 returns; its two no-provider reasons moved into a frozen table read through hasOwnProperty, which drops it to 4 and means a reason of constructor cannot reach a function. Every assertion on both is unchanged.

The reads are race-safe the way #555 hardened the Newsletter Hub. Every episode read goes through one generation counter, so opening set B while set A is still loading paints B — an A that answers late is dropped rather than shown under B's heading. That is a bug a bare cancelled flag does not catch, because both reads are on the same mount. A failed read empties the list rather than leaving the previous set's episodes beside an error saying they could not be read, and approvals are guarded per episode so a second click while the first is unanswered is ignored rather than sent twice.

A run keeps reporting while the operator is on another tab, because generating and progress live in the hook rather than in the Generate tab — a run takes several minutes, and leaving the tab must not lose the line that says what it cost.

The stored voice default is derived, not copied into form state. null means "not chosen yet", so the default appears when the settings load; '' means the operator chose "Stored default" deliberately. Both send no model, and keeping them distinct is what lets a late settings response fill an untouched field without overriding a choice already made. The previous shape wrote the default back into state from an effect, which is a second source of truth for one value and is what react-hooks/set-state-in-effect objects to.

Deep links are ?tab=; the old section words (sets, episodes, voice, grounding) and anything unknown land where their content went. resolveTab reads own properties only, so ?tab=constructor cannot reach Object.prototype and hand a function to the panel lookup — pinned by a test, as it is on the Certifications Hub.

Settings links to Platform settings for the voice default rather than offering a second editor over the same document, and records why certification episodes are grounded on the official study guide alone: one outdated video demonstrating a retired portal flow would otherwise be laundered into an episode that sounds authoritative, and exam prep is exactly where confidently wrong is worse than absent.

  • Certifications Hub: tabs by duty (#572). /admin/certifications moves from one page (1,107 lines) to five tabs on the shared HubTabs bar, deep-linked with ?tab=:
  • Catalog: stats, search, issuer filter and the card grid.
  • Featured: featured certifications in display order.
  • Renewals: expired certifications and those due within 180 days, soonest first.
  • Publishing: the Publish snapshot button, when the public snapshot was last published, and what changed since.
  • Settings: image rules, verification sources and hidden items.

Old ids such as expiring, hidden and publish land on the tab that now holds them. The list loads once at page level with a generation guard, and a read that raced a write is issued again. Writes have a per-certification in-flight guard, and the editor's save and upload have their own. The Publishing tab reads the public snapshot on its own, so a failure in either read stays on its tab. PublishSnapshotButton gains an optional onPublished callback, and publicApi.js gains fetchPublicSnapshot.

  • Speaking Events Hub: tabs by duty (#573). /admin/speaking-events is now five tabs on the shared HubTabs bar, deep-linked with ?tab=:
  • Upcoming: future sessions, soonest first.
  • Past: delivered sessions, newest first, with slide and event links.
  • Sources: when Sessionize was last read, a preview of what a sync would create or fill in, Sync from Sessionize, and manual entries.
  • Publishing: the public snapshot and what a publish would write now.
  • Settings: the Sessionize speaker ID, read-only, with a link to the Integrations card that stays its only editor.

Old ids such as events, sync and publish land on the tab that now holds them. Each read is generation-guarded, and a failed refresh clears its rows. Save, delete and sync each have an in-flight guard, and the session lists wait for both reads so Enrich cannot create a duplicate record. Behaviour changes: the store is re-read after a partly failed sync, and event links render only for http(s) URLs.

  • Health Hub: Code and Security tab, and a Qlty card on Integrations (#569). A new Code and Security tab reads GET /api/cms/code-quality (from #590) the first time it is opened, not on page load, and shows:
  • Qlty's grade tiles: maintainability, security, coverage, duplication and technical debt;
  • open issues by level, and the security total with its split;
  • categories, top rules and top files, linked to Qlty;
  • an as-of time, and a warning when the summary was truncated.

Qlty not configured is a notice linking to the Integrations Keys tab, and a refusal is a retryable error. The loader uses the Health Hub's generation guard, so a slower read never overwrites a newer one and a failed refresh clears the numbers. The copyable report gains a Code and Security section once the tab has loaded. The Integrations Hub gains a Code quality group with a Qlty card (QLTY-API-TOKEN) whose test calls the qlty connection probe and names the account the token belongs to. ?tab=qlty, quality, security and code-quality open the tab.

  • Health Hub: tabs by duty instead of one long scroll (#569). /admin/health now has four tabs on the shared HubTabs bar, deep-linked with ?tab=:
  • Overview: the at-a-glance strip, readiness, publishing and operational signals.
  • Alerts: workflow alerts with their filter and actions, and their own outcome line. Before, it was written onto the smoke-test card.
  • Checks: smoke tests, token claims, admin registry, the Labs probes and Re-run identity checks.
  • Report: the Markdown report and Copy report.

Old ids such as diagnostics, signals and verify land on the tab that now holds that content. The header and tab bar always render: a failed ops-health read is an error on Overview and Alerts, with Try again, while Checks and Report carry on. The strip's observed badges say UNKNOWN rather than zeros until the snapshot arrives. Every snapshot read (first load, refresh after a smoke test or alert action, Try again) goes through health/useOpsSnapshot.js, which has a generation guard. A slow read cannot overwrite a newer one, and a failed read clears the numbers instead of leaving them beside the error. Shared state stays on the page, so switching tabs never refetches or reruns a probe. The Code and Security tab follows in a separate PR.

  • Code and Security summary API for the Health Hub (#569). GET /api/cms/code-quality (editor) reads this repository's Qlty project server-side with the new QLTY-API-TOKEN secret (app setting QLTY_API_TOKEN): project metrics, and every open issue paged 100 at a time. It answers condensed totals by level, category and security, the top ten rules and files, and Qlty's own grades and coverage, cached for ten minutes. No finding message, snippet or fingerprint leaves the server, and log lines carry only the route, status and invocation. Checked live on 2026-09-14: it counts 1,353 open issues and 48 security issues, equal to Qlty's Total Issues and Security Issues metrics. The token gets a Code quality section in the secret catalogue, so it can be seeded from the Integrations Keys tab, and a qlty connection probe (GET https://api.qlty.sh/user). The Health Hub tab that shows the summary follows in a separate PR.

  • Integrations Hub: tabs by duty instead of one long page (#570). /admin/integrations is now four tabs at the Newsletter Hub standard, split into frontend/src/components/admin/integrations/. Overview is one grid of every service, broken and not-configured first, with Test all: each service's existing test, run one at a time, with YouTube left out because its test spends daily quota (its tile says so), and the time each ran this session. Services shows one group at a time (?group=); each card keeps its description, docs link and test, and lists the keys it uses by name and light only. Keys holds every catalogue credential under its group, the services that use each one, the unchanged paste and generate actions, and "Other credentials"; one write runs at a time, and no value is fetched or rendered. Identity is the Entra configuration panel as it was (#519). Tabs deep-link with ?tab=, moved or unknown ids land where their content went (Overview by default), and /admin/connections and /admin/api-keys now redirect to Overview and Keys. Each tab reads its own data with its own loading and error states, reloads are generation-guarded, and a failed refresh clears what it replaced. No API, probe or seeded value changed. There is no Activity tab: key writes keep only the latest writer per key and no admin route reads admin_audit_logs, so that history needs a backend first (noted on #570).

  • Platform Settings Hub: one tab per concern, plus a change history (#571). /admin/platform is now four tabs at the Newsletter Hub's level of separation, deep-linked with ?tab=: Content defaults (default covers, and a link card to Newsletter Hub → Settings rather than a second form over the newsletter document), Social automation (autoposting, the only tab that asks Publer for accounts), Audio (podcast feeds and the Listen & Learn voice) and Change history. Each tab mounts only while open and loads its own settings with its own loading, error and Retry states, so a failure in one never blanks another; an unknown ?tab= lands on Content defaults. Every save sends the same body as before. useSetting gains the #555 race guards: a generation number on loads, an in-flight ref on saves (a double submit sends one PUT), and a failed load clears the working copy and hides the form. Change history reads the new GET /api/cms/platform-settings/history?setting=&limit=&after= (editor): platform_setting_updated rows from admin_audit_logs, newest first, TOP 1..100 (default 50), with the setting validated against the spec registry and every refusal a 400 before Cosmos. Each entry is { id, at, actor, setting, summary }, where the actor is the display name or else the Entra object id and never the email the row also holds, and paging is by timestamp through nextAfter. New audit rows record userName from the token's name claim so the table can say who. The tab shows time, who, setting and key: value summary chips, with a setting filter, Refresh and Load more.

  • Qlty set up in the repository, with test coverage uploaded from CI (#568). .qlty/qlty.toml is committed, so a local qlty check --all and Qlty Cloud run the same plugins: the linters that work here (actionlint, editorconfig-checker, eslint for frontend/ and scripts/, prettier, ruff, radarlint-python, ripgrep, tflint) and the security scanners (bandit, checkov, gitleaks, osv-scanner, radarlint-iac, trivy, trufflehog, zizmor). ESLint stopped failing with Cannot find package 'globals': the init-written package_filters dropped globals (and react, which the frontend config reads), so each package's own ESLint config now installs complete. Generated and vendored paths are excluded (frontend/data/**, public JSON and fonts, dist, coverage, site, .agentic). No security finding is silenced in qlty.toml; triage uses each scanner's own ignore file, recorded in docs/security/scanner-triage.md (#567). frontend/ and functions/ gain npm run test:coverage (V8 coverage, coverage/lcov.info with repository-relative paths; functions/ adds @vitest/coverage-v8 4.1.11); npm test and the required CI jobs are unchanged. The new, non-required coverage.yml workflow runs both on pull requests and pushes to main and uploads them in one qltysh/qlty-action/coverage step pinned to v2.3.0's commit. The upload authenticates with GitHub OIDC, so no Qlty token is stored; pull requests from forks, which cannot mint one, skip the upload with a notice.

  • Use a Resend template as the newsletter's design (#557). Newsletter settings on the Mailing List Settings tab gain a Template block: keep the built-in design (the default, and what existing settings read as) or pick one of your published Resend templates, listed from GET /api/cms/mailing-list/templates; unpublished templates show disabled, and when the list cannot be loaded the saved choice is kept and the reason is shown. Resend broadcasts take no template id and template variables cannot loop over articles, so the site does the merge (lib/newsletter/template-layout.js): the week's body, rendered exactly as the built-in design renders it, goes at {{{NEWSLETTER_BODY}}} (required, exactly once); {{{NEWSLETTER_SUBJECT}}}, {{{NEWSLETTER_PREHEADER}}} and {{{NEWSLETTER_PERIOD}}} are filled with escaped text; {{{UNSUBSCRIBE_URL}}} is rewritten to {{{RESEND_UNSUBSCRIBE_URL}}}; and the postal address and unsubscribe link are appended when the template lacks them. The template is fetched with a five-minute in-process cache per id, dropped when a different template is saved. The preview and the test send use the template, or the built-in design with a visible templateProblem warning when it cannot be fetched or used; approval is refused with 409 TEMPLATE_UNUSABLE before the issue is claimed, so a newsletter never goes out in a design other than the one chosen. The plain-text part is unchanged, and the built-in HTML is pinned byte-for-byte by render.test.js.

  • Choose where the newsletter signup box appears, and its wording (#557). Newsletter settings on the Mailing List Settings tab gain a Signup form block: show the box in the footer on every page, at the end of blog posts, both, or nowhere, and set its heading (up to 80 characters) and blurb (up to 240), with character counts and a live preview that uses the site's own box and cannot subscribe anyone. It saves with the rest of the settings. The site reads the choice from a new anonymous GET /api/public/newsletter/signup-config, which answers { placement, heading, blurb } and nothing else from the settings document, is cached for five minutes, and answers the defaults with 200 when the settings cannot be read. The footer and the blog post share one request per page load, render the wording as plain text, and show the defaults (both places, the original wording) while it loads, if it fails, and in the pre-rendered HTML, so a site whose settings were never saved looks exactly as it did before. The public subscribe and confirm routes are unchanged.

  • Choose what each newsletter issue is built from (#557). Newsletter settings on the Mailing List Settings tab gain a Content block: tick which sections go in, put them in order with up and down buttons, set the most items each may show (1 to 20), how many days back to look (1 to 31), whether the AI intro is written and in which tone (professional, friendly, short and direct, enthusiastic). It saves with the rest of the settings and the form shows what the server stored. The issue builder reads these for both the Build this week's issue button and the Monday automatic build, neither of which passes a fixed seven days any more; an explicit days on the job still wins, clamped. Turning the intro off builds with no AI call and no intro error; the regenerate-intro button uses the saved tone. Settings saved before this read as the old behaviour (every section, 12 items each, seven days, a professional intro), with no migration. Nothing here sends email.

  • Resend domains, recent emails and API logs on the Mailing List Settings tab (#504). The page half of the Resend domains, logs and emails API (#553), frontend only. Three cards below the connection check, each loading when the tab opens and failing on its own (503 says Resend is not configured, 429 says how long to wait), so none can hold up the newsletter settings. Sending domains lists each domain with its status and region; opening one loads its DNS records (type, host, value with a Copy button, priority, TTL, per-record status) with a note to add them at Cloudflare, a Verify button that says the check runs in the background and offers Refresh, and open and click tracking switches that PATCH only the switch that changed, with a warning that open tracking needs the tracking subdomain CNAME and is recommended for broadcasts only. An Add domain form takes a name (checked as a hostname before sending; the server decides) and a region, defaulting to us-east-1, and opens the new domain on its records. There is no delete; a note links to resend.com/domains for that. Recent emails shows the server-masked recipient, subject, time and last event, and API logs the time, method, endpoint and a status coloured by 2xx, 4xx or 5xx, both paging with Load more. Opening a log shows its redacted request and response bodies as pretty-printed text, never as HTML. The only writes are domain add, verify and tracking, all publisher-only on the server; a refusal is shown in the server's words.

  • An Audience tab and per-issue metrics on the Mailing List page (#504). The page half of the Resend metrics and audience API. A new Audience tab, between Published and Settings, shows the Newsletter segment's subscribed, unsubscribed and total counts (with when they were counted, and a note when they are a lower bound), then the contacts fifty at a time with Load more and a debounced email search that says it searches the loaded page. Each row can be unsubscribed, resubscribed or removed; those are the only writes, they are publisher-only on the server, a refusal shows the server's message, and Remove asks first because it deletes the contact from Resend. Contacts are addressed by Resend id, never by email. On Published, a sent issue now shows delivered, unique opens and clicks with their rates, bounces, unsubscribes and complaints above its email, its top ten links, and a collapsible list of who opened, clicked or bounced; a scheduled issue says metrics come after it sends. The month header adds that month's delivered count and open rate, loaded after the calendar and left out if Resend cannot answer. A 503 says Resend is not configured and a 429 says how long to wait, which lib/api.js now carries on the thrown error as retryAfterSeconds.

  • Resend metrics, audience, domains, logs and templates for the Mailing List page, in the API (#504). The owner wants Resend's numbers and settings on the site rather than in Resend's dashboard; this is the API half, and the Metrics, Audience and Settings → Resend pages follow. New routes under /api/cms/mailing-list/: metrics (by broadcast, or by newsletter issue, defaulting to the last 30 days), a broadcast's clicked-links and recipients, one page of the Newsletter audience plus a cached audience/summary of subscribed and unsubscribed counts, domains with their DNS records, logs (bodies only on the detail, with credentials and email addresses redacted), recent emails with masked recipients, and templates. Reads need editor. No route sends email. The ones that write to Resend — unsubscribing or deleting a contact, and adding, verifying or changing tracking on a domain — are publisher-only; deleting a domain is deliberately not offered. Every id, cursor, date, address and limit is validated before Resend is called, a Resend refusal comes back as a 502 carrying Resend's reason (a rate limit as a 429 with the wait), and log lines carry no addresses.

  • Faster newsletter editing on the Mailing List page (#504). The page half of the editing API change further down. A draft on the Newsletter or Drafts tab now opens with the subject, a Suggest subjects button whose three to five AI suggestions fill the subject when clicked, a Preview text field with a 150-character counter, and the note. The action bar holds Save changes, Reset (discards local edits), Regenerate intro, Send test to me (one [TEST] copy to the reply-to in settings, then a 60-second countdown) and Keep in Drafts or the approval box. Below it, a section editor sits beside the email preview on wide screens: sections collapse, and sections and items move up and down or are removed, with the last item protected as the server requires. Section edits count as unsaved, so Keep and Approve stay disabled until they are saved, and Regenerate intro and the test send wait for a save too, since both act on the stored issue. After a test send the page holds the issue's new etag, so the next save is not refused as stale. Frontend only; nothing on the page reaches subscribers except the unchanged Approve. IssueDetail, SectionEditor, ApprovalBox and issueFormat are split out of NewsletterIssues.jsx.

  • Faster newsletter editing in the API: preview text, section trimming, AI intro and subject help, and a test send (#504). The owner found editing a weekly issue "still very stiff"; this is the API half, the page follows. PATCH /api/cms/newsletters/{id} now also takes preheader (the inbox preview text, at most 150 characters, rendered as the standard hidden block at the top of the email) and sections, which may only remove or reorder items and sections: every item is matched to a stored one by section and URL, the stored items are written back, and anything added, moved or edited is a 400. POST .../intro regenerates a draft's intro with the builder's own drafter and instruction (a failure stores nothing, 502); POST .../subjects returns three to five subject-line suggestions and writes nothing. POST .../test (publisher) emails one [TEST] copy to the reply-to address in Newsletter settings, never an address from the request, at most once a minute per issue. Nothing here can reach subscribers: the test uses Resend's single-email endpoint, which is why it works while sending is switched off. Approve is unchanged apart from now rendering the preheader. Also corrects timer-catalogue-sync.test.js, which placed local.timer_catalogue in infra/main.tf; it is in infra/functionapp.tf.

  • The weekly newsletter builds itself into Drafts every Monday (#504). A new timer, buildWeeklyNewsletter, runs Monday 13:00 UTC (08:00 CDT / 07:00 CST), builds the last seven days into an issue saved straight to the Drafts tab, and sends a Telegram message with the subject, the item count and a link to /admin/mailing-list?tab=drafts. A week with nothing new, or a day whose issue is already kept in Drafts or approved, writes nothing and says so; a missing AI intro is called out. It never sends: approval stays a manual press on the Drafts tab. It ships disarmed behind FEATURE_FLAG_BUILD_WEEKLY_NEWSLETTER — add BUILD_WEEKLY_NEWSLETTER to enabled_timers in the HCP Terraform workspace to arm it. The issue builder gained a keep option for this, and the build-newsletter-issue job now passes only days through from its payload.

  • A Settings tab on the Mailing List page (#504). Newsletter settings no longer sit under the issues, and the Connection tab is gone: both are on Settings, the last tab. A bookmark to the old Connection tab opens Settings.

  • Newsletter, Drafts and Published tabs on the Mailing List page (#504). The owner's layout of 2026-09-13. Newsletter builds this week's issue for review; each issue card has a red X that deletes it at once, and Keep in Drafts moves an issue worth sending to Drafts, which is the only place an issue is approved. Published is a month calendar of what was sent (filled count) or scheduled (outlined count) on each local day; choosing a day shows that day's email in a sandboxed preview. Reject is now only Clear stuck send, for an issue left mid-send.

  • Delete, save to Drafts, and a month of published issues in the newsletter API (#504). The API half of the Mailing List page's Review, Drafts and Published tabs. DELETE /api/cms/newsletters/{id} deletes a draft or rejected issue (never one that was approved); it is a conditional write to deleted, so it cannot race an approval, and a deleted issue reads as not found and is rebuilt from scratch. POST /api/cms/newsletters/{id}/save keeps a draft, and the builder never rebuilds over a saved one. GET /api/cms/newsletters?month=YYYY-MM lists what was sent or scheduled in a month for the calendar. List rows now carry etag and savedAt.

  • Approve the weekly newsletter from the Mailing List page (#504, ADR 0030 §2a). POST /api/cms/newsletters/{id}/approve routes the approval logic added just before this, and the Newsletter tab gains Approve and schedule, then Yes, send it. It sends the version on screen, will not approve unsaved edits, re-reads the issue after any answer that is not a clean success, and offers only reject for an issue stuck in sending. While newsletter_sending_enabled is false in Terraform the page says sending is switched off and shows no Approve button, and the route refuses with 503.

  • The approval logic for the weekly newsletter (#504, ADR 0030 §2a). approve in lib/newsletter/admin-handlers.js and createBroadcast in the Resend client. It landed before its route; the route and the Approve button are the entry above. Approval stays refused until newsletter_sending_enabled is true in Terraform.

At most once, and only as the approver saw it. Approval requires the issue's etag, so an issue edited after the approver opened it is refused. The send slot and the email are worked out first, then the issue is claimed draft → sending with an ETag-conditional write before Resend is called, so two approvals at once create one broadcast. The outcome is recorded only while the issue is still that claim: a reject that lands mid-send stands. A refused broadcast returns the issue to draft with Resend's reason. No answer from Resend at all leaves sending, because a broadcast may exist. Reject may now clear an issue in sending. Approval logs carry the invocation id only.

  • A Terraform switch for newsletter sending, off by default (#504, ADR 0030 §2a). New variable newsletter_sending_enabled (default false) sets the app setting NEWSLETTER_SENDING_ENABLED. The issue API reports it as sendingEnabled, and the Mailing List page says when sending is switched off. Nothing sends yet: approval arrives next, and will be refused while this is off, so turning sending on is the owner's confirmed Terraform run.

  • Build and review the weekly newsletter from the Mailing List page (#504, ADR 0030 §2a). Third of the small changes. The Newsletter tab builds this week's issue, lists recent ones, shows the email exactly as subscribers would receive it, and lets an editor change the subject, add a note, or reject the draft. Newsletter settings — postal address, reply-to, send day, time and zone — are on the same tab, and the issue view says what a send still needs. Nothing on the page sends; approval is the next change.

The old digest is gone. generate-weekly-digest and lib/content/digest.js are replaced by build-newsletter-issue, which runs the structured issue builder. The old job's drafts had no links and nothing read them.

Edits carry the version they were made against. Every save and reject sends the issue's etag; if someone else changed the issue first, the page shows the server's sentence and reloads the stored version instead of overwriting it. The email preview renders in an iframe with an empty sandbox.

  • Admin API for weekly newsletter drafts, and the newsletter settings (#504, ADR 0030 §2a). Second of the small changes building the weekly newsletter. Editors can list issues, read one rendered exactly as it would send, edit a draft's subject and note, and reject it: GET /api/cms/newsletters, GET|PATCH /api/cms/newsletters/{id} and POST /api/cms/newsletters/{id}/reject. Nothing in this API sends.

cms/platform-settings/newsletter-settings stores what a send will need: the postal address, a reply-to inbox (one on news.hybridcloudworks.com, which receives no mail, is refused), and the send slot as weekday, time and IANA zone. A partial save is allowed; the issue view reports what is still missing and when an issue approved now would go out. The audit row records whether the address and reply-to are set, never their values.

Edits are ETag-conditional, so a concurrent change is a 409 rather than an overwrite, and an id that is not issue-YYYY-MM-DD is refused before Cosmos is read. The page that uses this is the next change.

  • The weekly newsletter's core library: what goes in an issue, how it is built, rendered and scheduled (#504, ADR 0030 §2a). No route, job or page uses it yet — the admin surface and the job swap are the next two changes, split so each is small enough to review on its own.

Sections are a registry. lib/newsletter/sections.js collects new articles (linked to the URL publishing stored, never a guessed one), Microsoft certification news with exam codes, and approved study and podcast episodes. A section is one entry; one that throws costs its heading, not the issue.

An issue is a structured draft. issue.js stores the sections' items, a plain-text intro and subject from the existing drafter, and room for the owner's note; a week with nothing new builds nothing and asks no model, and with the AI off the issue still builds. One issue per day, and an issue that has left draft/rejected is never rebuilt.

The email escapes everything and re-checks every link. render.js escapes each stored value once and drops any item whose link is not https or a site path, however it got there; the footer carries the postal address and Resend's per-recipient unsubscribe link.

The send slot survives daylight saving. schedule.js turns a weekday, a wall-clock time and an IANA zone into a UTC instant per send, so Tuesday 09:00 Central stays 09:00 in March and November. The email-address check moved to email.js so settings validation will not load the signup handlers.

  • The newsletter signup works, with double opt-in (#504, ADR 0030 §3a). The form on every page posted to a route that never existed. It now posts to public/newsletter/subscribe, which emails a confirmation link from newsletter@news.hybridcloudworks.com and adds nobody to the list; the link opens /newsletter/confirm, and only pressing Confirm there writes the contact to Resend's Newsletter segment. This is the second of three steps; sending the weekly digest is the third.

Confirm does not trust the write. ADR 0030 named resend/resend-node#458, a contact created unsubscribed: false coming back unsubscribed — which would tell someone they are in and then skip them on every broadcast. So after writing, confirm reads the contact and its segments back and answers success only when both say subscribed, repairing once if they do not. Resend does not document what creating an existing contact does either; the same read-back covers it, and someone who had unsubscribed and confirms again is resubscribed, because confirming is that consent.

Signup is a way to make this site email a stranger, so it is limited twice: five attempts per Cloudflare-verified caller per hour, and two confirmation emails per address per hour whoever asks. It never reads the list, so it answers identically whether or not an address is already subscribed. A filled honeypot is answered like a real signup and sends nothing.

Three choices keep the address out of places it does not belong. The token rides in the URL fragment, which no server sees, so the address inside it never reaches a request log. Loading the confirm page confirms nothing — mail scanners open every link before the person does — and only the button POSTs. No log line in either handler contains an address. The token's HMAC key is HKDF-derived from RESEND_API_KEY, so there is no second secret to seed: whoever holds that key can add contacts directly, and forging a confirmation grants them nothing.

  • Resend replaces Klaviyo: its key has a row on the API-keys page and a test that refuses the wrong kind of key (#504, ADR 0030). This is the first of three steps. It swaps the plumbing; the signup route and sending follow.

What went. klaviyoProxy, both Klaviyo Key Vault references and their catalogue rows, the Klaviyo card on the Integrations page, the Lists and Campaigns tabs on the Mailing List page, and lib/klaviyo.js. Nothing wrote to Klaviyo, so nothing that stored data is lost; the weekly-digest buttons, the one part of that page that worked, stay.

What came. RESEND_API_KEY, a Key Vault reference and a catalogue row, and a resend entry in connectionProbe that calls GET /domains on the server. That call was chosen because it answers the question worth asking before anything is built on the key: Resend refuses a key minted with sending access only, and that key could never manage contacts or broadcasts. A pass also reports how many sending domains exist, and zero gets its own sentence — the key works and nothing can be sent yet. It records no key verdict, because a refusal here does not always mean "remint".

Seed before apply. The API-keys page writes to Key Vault by catalogue name and does not need the app setting to exist, so the key can be seeded once this deploys and before the Terraform run that adds its reference. Done in that order, monitor-unresolved-secrets.yml never sees RESEND_API_KEY unresolved.

Also corrected on the way through: docs/standards/variables-and-secrets.md said infra/main.tf declares 21 Key Vault references. They are in infra/functionapp.tf, and there are 25 after this change.

  • ADR 0030 records the newsletter provider, and corrects what #504 said was broken. The evaluation #504 asked for is decided: Resend, with the cadence staying in the Function App's timer rather than moving into a vendor's scheduler.

The send allowance decides it, not the feature lists. Klaviyo's free plan is 500 emails a month. A weekly newsletter is 4.33 sends per contact per month, so that plan runs out at 115 subscribers — and at its own 250-contact ceiling the most it can deliver is fortnightly. Resend's marketing meter bills on contacts stored rather than emails sent, so 1,000 contacts receive a weekly newsletter for nothing. The first paid step is USD 40 a month at 5,000 contacts, against roughly USD 100 for Klaviyo at the same size; the record states plainly that this is a cliff and not a ramp, and sets a revisit trigger at 800 contacts so the jump is chosen rather than billed.

#504's description of the failure was wrong and the ADR says so. The issue called the signup form "a control that looks like it worked" and compared it to the Publer button that reported success for a 403. NewsletterSignup.jsx:46 checks res.ok and throws, so the 404 renders a visible "Subscription failed." Nothing is silently discarded and nobody is told they subscribed when they did not. What is live is a visibly broken control on every page — embarrassing rather than lossy, and not a reason to rush the provider choice. Recording the correction matters more than the verdict: the next session to read #504 would otherwise act on an urgency that is not there.

The one lossy failure this could still produce is named as a risk instead: resend/resend-node#458 reports a contact created with unsubscribed: false coming back unsubscribed, which would capture a subscriber and then skip them on every broadcast. The first write asserts the stored state rather than trusting the call, and a test pins it.

Also decided: the public route contract names no vendor, so a provider change is a handler change and the frontend does not move; Resend holds the list and Cosmos does not mirror it, for the reason ADR 0029 keeps the feed as the one boundary — two stores of the same truth disagree, and the unsubscribe is where that becomes mailing someone who opted out; and new code targets segments.* rather than the deprecated audiences.*.

  • A documented, mostly-automated rollback for admin sign-in. The Entra split left one recovery path — set VITE_ENTRA_CLIENT_ID back and redeploy — living only in a pull request description and somebody's memory. It also stopped working the moment the API registration's redirect URIs were cleared, because a frontend pointed at a registration with no redirect URIs fails at the redirect.

docs/runbooks/admin-signin-rollback.md now carries it, and leads with how to tell whether a rollback is even the right answer — a table mapping what Admin → Health shows to whether this page applies. A removed App Role and a wrong audience look identical from outside the portal and want opposite responses.

deploy-azure-frontend.yml takes an optional entra_client_id_override, so the rollback deploy is one dispatch and needs no repository-variable write. Extending the real deploy rather than adding a second workflow is the point: an emergency path that has drifted from the path it is meant to replace is worse than no emergency path, and a copy of 340 lines of build-and-upload would drift. Blank is falsy in a GitHub expression, so an ordinary run is unchanged, and the run logs which id it built with so the record is in the run rather than in a memory.

What stays manual, and why that is deliberate. Restoring the redirect URIs needs Entra directory rights, and the deploy identity has none — infra/oidc.tf explains that Azure Owner does not grant them. Granting CI Application Administrator so a rollback is one click would hand every future workflow run the power to rewrite app registrations, permanently, against an incident that may never happen. scripts/rollback/restore-admin-signin.ps1 does that half, reads the registration back rather than trusting the PATCH, and prints the client id for the workflow input.

The identifiers live in the script because the two house rules meet here and appear to conflict: the docs redaction gate rejects real GUIDs under docs/, and the working agreement rejects placeholders in a line meant to be pasted. A runbook full of 00000000-… would satisfy the gate and fail the operator. scripts/ is not scanned, so the runbook says "run this" and the values it needs are concrete. A test asserts the runbook contains neither a GUID nor a placeholder, and that the runbook, the script and the workflow all name the same input.

  • Sign-in lands on a page built for it, not on the public home page (#520). msalConfig.js set redirectUri: window.location.origin, so Entra returned the browser to / — a page that had never touched MSAL, because AdminAuthGuard is lazy-loaded and mounted only under /admin. Nothing consumed the authorization code. The fix at the time was useAuthRedirectLanding, mounted from App.jsx on every route in the application — including every anonymous visit to a provider news page — so that a regex could watch window.location.hash for a fragment that appears on one.

/auth/callback is that one page. initializeAuth() consumes the fragment and MSAL's navigateToLoginRequestUrl — which defaults to true — returns the user to wherever sign-in started, the mechanism the bare origin was working around rather than using. The 2026-08-23 incident that prompted the hook is carried into the new page's header so it is not lost with it.

The hook was kept for one more release, deliberately. A browser mid-redirect at cutover, or a bookmarked stale fragment, still landed on /; removing both halves at once would have turned those into a home page with a dead #code= in the address bar that reproduces on every reload — the exact symptom the hook was written for. It was removed in #531 once the API registration's bare-origin redirect URIs were cleared, which is what closed that window; see the Removed section.

A test asserts the path inside redirectUri is a route App.jsx declares. That agreement is between a config string and a route table, so nothing that renders a component would notice it breaking — and when it breaks, sign-in fails in production and nowhere else.

Also here: MSAL now has loggerOptions at Warning in production, dropping anything MSAL flags as containing PII. There were none at all, and all three failures this codebase's comments are built around surface at Warning, so Error alone would have shown nothing new on any of the days they cost. /auth/* is noindex, nofollow — an indexed callback page would appear in search results holding somebody's authorization code.

And the unreachable popup branch in entraAuth.js is gone, along with the prefersRedirect() that returned an unconditional true. The reasoning is kept and moved onto signIn; the code was four MSAL calls, and while it existed a test had to mock loginPopup to reject in order to assert nothing called it. A test arranging behaviour for unreachable code is a trap for the next reader.

  • The Entra configuration is reviewable in the admin GUI (#519). It was visible nowhere. The tenant, audience, App Roles, scope and token version are plain app settings rather than Key Vault references, so they never appeared in secret-catalog.js and never reached the Integrations page — and the SPA's own half (VITE_ENTRA_CLIENT_ID, VITE_ENTRA_TENANT_ID, VITE_ENTRA_API_SCOPE) was rendered nowhere at all. A scope that disagreed with the audience, or a SPA pointed at a different tenant from the API, was invisible until every call started returning 401.

Integrations gains a Microsoft Entra ID panel in the existing "Site platform" group, whose own blurb already described exactly this — values the site runs on, each one says what changing it breaks. Two columns: what this browser was built with, and what the API says it enforces, the second from getAuthExpectations. A page that compared the frontend against itself would prove only that it agrees with itself. The group now survives an empty credential list, because the API failing to answer is itself a configuration symptom and hiding the panel then would be exactly backwards.

Health gains azp, scp and ver rows and three verdicts: the delegated scope is present, the token version matches, and — the one that matters structurally — azp differs from aud. azp is the client that asked for the token and aud is the API it is for; one registration serving both makes them the same GUID, which is the condition DECISION 3 warns about and #522 exists to end. When they match, the card says so and explains why the delegated-scope check keeps it safe meanwhile.

Identifiers only. No credential, no token, no oid, no email — the rule the Integrations page states in its own header, and there is a test asserting the panel renders nothing that looks like a JWT or a secret.

getAuthExpectations now also reports requiredScope, requiredTokenVersion and labAgentAppRole, each read from the constant the guard actually enforces rather than a copy, with a test tying the two together. '2.0' became an exported constant for the same reason it was worth doing for the App Role names: a second spelling is a silent drift.

  • The Monday cron now refreshes the Azure study-guide outlines too, not just the catalogue (#499). The outlines shipped with #498 and refreshed only by hand, so between hand runs the detail pages could show an outline older than the exam list beside it.

This workflow was deliberately scoped to one file: its git add, its artifact upload and its pull request body all named frontend/src/data/azure/certifications.js alone, and that scope is the argument for what the App installation token is allowed to reach. Widening it to a second output was therefore its own reviewable change rather than a line inside #498.

Six touch points, all shaped like the existing ones: the update job now installs the functions tree as well — update-study-guides.mjs imports the Listen & Learn parser from functions/src/lib/listen-and-learn, so node resolves cheerio out of functions/node_modules and without it the import fails at resolution before a guide is fetched; the study-guide script runs after the catalogue script, because it reads eligibleCerts() out of certifications.js and running it first would refresh the outlines against last week's exam list; study-guides.test.js runs beside certifications.test.js, separately because it asserts a different contract (no outline is orphaned from a live exam); and the change check, artifact, download, copy, git add, commit message and pull request body all gain the second path.

Two things #499 did not list. setup-node was keying its cache on the frontend lockfile alone while this job now installs two trees, so the cache would have gone on reporting a hit while functions/ was fetched from the registry every Monday. And the staging guard went from "stages the catalogue" to "stages exactly these two paths and nothing else" — the one-file scope was the security argument, so widening it to two deserves an enumerated boundary rather than a second presence check. A third output is now a deliberate edit to that test.

The failure policy is stated where it will be read. A partial failure is not a red run, and the script already implemented that: it refuses to write only when nothing parses, and keeps an exam's previous outline when that exam's own fetch fails. What was missing was visibility, so the pull request body now carries the study-guide summary — which already names every failed exam with its code, key and reason — plus a sentence explaining that a non-zero "Failed" count means the page still renders an outline, but one older than the date the file claims.

  • Every live Azure certification page now carries the official study guide's "Skills measured" outline, area by area, each deep-linked to its heading on Microsoft Learn (#498). The owner's ask, from the AB-650 page: "Topics Covered" named a handful of themes, and what a learner needs is the table of contents Microsoft actually examines against, in Microsoft's own words, linked to where it lives. Azure only, as asked.

The data already existed in the pipeline and was being thrown away. functions/src/lib/listen-and-learn/studyguide.js parses these guides into weighted areas, sub-headings and objectives to script Listen & Learn episodes — and the generator used the outline to write a prompt and discarded it. But the generator only runs when the owner generates an exam, and on 2026-09-11 exactly one Azure certification had ever been generated. An outline that existed only after a generation run would have been missing from nearly every page. So the outline ships as data, like the catalogue: frontend/scripts/update-study-guides.mjs reuses that parser across packages, and src/data/azure/study-guides.js carries 56 of 56 eligible guides, 234 areas, 961 sections, zero failures, stamped DATA_AS_OF and held by study-guides.test.js to the same rules as the catalogue: every live exam with a guide has an outline, no outline is an orphan, the join is the guide URL's tail because applied-skills entries share site-assigned codes and a URL tail can only name one guide.

Each area's anchor is read off the Learn page, never derived. Learn builds its ids from the whole heading — "(20–25%)" becomes "2025" — so the parser's slugify(name) cannot reproduce them, and a guessed fragment would scroll the reader to the top of the page while looking like a deep link. The parser now captures the <h3>'s own id, null where the page carries none, and the page renders a plain link in that case rather than a broken one. Verified in the built output: the prerendered AZ-104 page carries #manage-azure-identities-and-governance-2025.

It is in the HTML. Because the data ships with the site, the outline is in every prerendered Azure detail page for crawlers and for readers with JavaScript off; the average prerendered page grew from 27 kB to 32 kB.

The cost, stated rather than hidden: 85 KB gzipped on every Azure detail page, against 3 KB for the AWS one, because the whole outline module is bundled into the Azure detail chunk although each page needs one fifty-sixth of it and already has that in its HTML. Not fixed here, on purpose: splitting it per exam means a lazy import inside a Suspense boundary so the hydrating render keeps the server HTML, which is a second change to the hydration story useToday was built around and wants its own review. Filed as a follow-up, with import.meta.glob — already used in provider-coverage.test.js — as the pattern.

Not on the Monday cron yet, and why. update-learn-catalogue.yml is deliberately scoped so its App token may touch one file; widening it to a second output is its own reviewable change and is filed. Until then the outlines refresh by npm run data:update:study-guides, which refuses to write if nothing parses and keeps an exam's previous outline if its fetch fails, so one bad page cannot erase a good outline.

  • The Azure detail hero carries the study podcast as one player with a chapter list (#498). ListenAndLearn below the fold renders a full card per episode; this is the "press play" surface, so a learner who has already decided to listen does not scroll past the topics to find the button. One native <audio> element and the chapters in study-guide order, switching chapter by swapping src — one set of controls, one position, one thing a screen reader announces. Renders nothing unless an approved episode has audio: on 2026-09-11 that is AB-100 and no other Azure exam, so the slot is present on every page and visible on one. Named "Study podcast", not "Listen & Learn" again, because the full section already carries that heading and a screen reader would announce the feature twice with nothing to tell them apart; its <audio> is labelled "Play chapter:" for the same reason.

  • Real brand marks on the Social Hub account chips, and a Threads entry that did not exist (#498). lucide-react v1 dropped its brand icons and the chips had been standing in with a chain link for LinkedIn, an @ for X, two heads for Facebook and a camera for Instagram — and nothing at all for Threads, which joined the workspace after the map was written. The owner's own report from the live page: "I cannot tell which account is which." components/shared/BrandIcon.jsx carries the networks' simple marks as inline SVG, currentColor so each chip's colour token applies, and aria-hidden because every chip renders the account name beside it.

  • /education, a front door to the Learn section that did not exist. routes.education(provider) appeared only inside a provider's own header, so a learner had to have already chosen a cloud to find the section at all, and the home page linked three of the eight — on hover. Two people were served by nothing: the beginner who does not yet know which cloud to learn, and the practitioner moving from one to another.

Eight provider tiles, each with the exams bookable today, the entry-level one to start with, and that catalogue's own freshness line through the existing CatalogueFreshness. Then a level table: rows are tiers, columns are providers, so AZ-104 and SOA-C03 land in the same row and the reader draws the line themselves. No cert-to-cert mapping is asserted anywhere — the table folds each catalogue's own level field, and a test walks all eight catalogues so a new level word fails the suite rather than silently dropping a row.

Built entirely from data that already exists, and the page adds none of its own. Prerendered like the other standalone routes. Its canonical comes from socialTags in prerender.mjs, which derives one for every route and never overwrites a page that sets its own — so this page sets a title and a description and lets the prerenderer do the rest, exactly as /about does. Confirmed in the built output rather than assumed.

"Bookable" includes expiring, deliberately. A published retirement date does not close an exam, and excluding those would have hidden seven sittable exams including AWS DVA-C02 and SAP-C02 from the one page meant to help someone choose. Non-active rows carry a word — retiring, beta — never a colour alone.

Twelve vendor level words fold into four tiers, which is the only hand-written part and is tested. Foundational, Fundamentals, Foundations and Practitioner all mean "start here" at AWS, Azure, GitHub and the FinOps Foundation respectively. Two folds are lossy and are recorded rather than hidden: VMware's Professional (VCP) and Advanced (VCAP) are genuinely different rungs that share a cell, and AWS's Business is one exam and a role cut rather than a tier.

The page surfaced two data problems it did not cause, both now filed: the same GitHub exam is carried in two catalogues at contradicting levels, and the Azure catalogue is the only one of eight with no DATA_SOURCE.

  • The Plaud tab says when the 12-hour token refresh last ran, so the rotation finally has a witness (#358). refreshPlaudToken rotates the Plaud MCP OAuth pair every 12 hours and writes lastTokenRefresh, oauthExpiresAt and lastTokenRefreshError to mcp_servers/plaud. Nothing rendered any of them, so from every surface a person can reach, "the timer is armed" and "the timer has actually run" looked identical.

The log was no help either, and that is the part worth recording. The timer reports success through log.log?.(), which is Information, and T-719 cut host.json's Function category to Warning — so the trace is not ingested. This is the T-766 witness gap in a different timer, found the same way: by asking what a person would actually read to confirm the thing happened.

The fix was a rendering gap, not an API one. stripOAuthToken replaces only the two token VALUES with booleans and spreads the rest of the document, so all three fields were already being returned. Recording Hub → Plaud → Connect now shows when auto-refresh last ran, when the access token expires, and the last failure if there was one.

Three distinctions the panel keeps, each of which would otherwise mislead:

  • Not yet read is not "never". refreshState is null until the first read answers, and the panel renders nothing for it. Printing "never" for an unanswered read is a claim rather than a measurement — the same distinction this page already draws between unknown and disconnected.
  • Connected is not refreshing. A document can hold an access token and no refresh token, which the banner already said; now it also says whether anything has actually rotated since.
  • A failed rotation is shown, not swallowed. lastTokenRefreshError renders in its own line, so a revoked refresh token is legible before the access token quietly expires.

fmtWhen takes both shapes the document stores, because the timer writes lastTokenRefresh as an ISO string and oauthExpiresAt as epoch milliseconds. It returns empty for anything unparseable: a token page is the wrong place to make someone wonder whether the date or the token is broken.

Review caught two more places the panel would have claimed more than it knew, which is the defect this whole change exists to remove. A thrown read set the banner to unknown and left the rotation panel showing the last successful timestamp as though it were current, so one half was confident about a read the other half called unknown; refreshState is now cleared with the status. And an unparseable value formatted to empty while its row still rendered, producing a label followed by nothing - which reads as a broken page rather than a missing value. The expiry row is gated on the formatted string, and a lastTokenRefresh that is present but unreadable now says so rather than reporting that the timer never ran, because those are different facts.

Confirmed live 2026-09-11, deployed with b70cd21f. The panel renders on the deployed site and the owner confirmed it was working.

The deploy was also confirmed without a credential, by a second technique beside the route probe #489 recorded — one for Functions, one for the frontend. The admin BUNDLE is public even though the PAGE behind it is not, so the chunk was fetched from the CDN and searched for a string that exists only in this change. That reaches inside a page an unauthenticated session cannot open.

The chunk is RecordingHubPage-<hash>.js, and the hash is written here as a placeholder on purpose: it changes with every build, so quoting the one from this deploy would hand the next reader a 404 and a reason to distrust the method. Find the current name by fetching / for the entry chunk, then reading the lazy-chunk filenames it references.

The observation followed on 2026-09-11: the timestamp advanced, which is the full cycle #358 asks for, and that row is now ticked. The two were kept apart in this entry while they were apart, because seeing the panel is the precondition for the read and not the read itself — easy to conflate precisely because the panel is the only place the rotation was ever visible, so both arrive through the same screen.

Recorded as owner-reported rather than measured. The Connect tab is behind admin sign-in, so no automated check could read that timestamp — not the probes that verified these deploys, not a monitor, not an agent session. It was read by the operator in their own browser, and the issue comment says the same thing in the same words. "Observed" and "reported as observed" are different evidence, and an estate that has twice been misled by a confident report of something nobody measured should keep saying which one it has.

A third followed: the auto-refresh row rendered even with no refresh token stored, where the banner already says the access token expires on its own. "Last ran: not since this token was stored" beside that implies a refresh token exists and has merely not fired yet. The row is now shown only where auto-refresh can run. The expiry row is kept in that case and matters more there than anywhere, because it is when the connection stops working.

  • The API-keys page refuses a credential paste that carries an invisible character, a curly quote, or an Authorization header (#484). rejectSecretValue already refused leading or trailing whitespace, which is the visible half of a bad paste. String.prototype.trim() is no defence against the other half: it removes U+00A0 and U+FEFF and only at the ends, it does not touch U+200B or the text-direction marks even there, and nothing it does reaches the middle of a string. A value carrying one of those stores clean, resolves clean, and is refused upstream forever as "the key is wrong" - the exact sentence that cost #358 several days on a credential where nothing could see the stored string.

Six checks now run after the existing ones, deliberately last so a placeholder still reads as a placeholder rather than as a value with a space in it: invisible and zero-width characters, control characters, a matched wrapping pair of quotes, curly quotes, an auth-scheme prefix, and interior whitespace. Each names what was found, because "invalid" here would reproduce one layer earlier the same uselessness the feature exists to end.

Naming what was found turned out to be harder than writing the checks, and review corrected the sentence twice. The interior-whitespace message said "a space" while the check is /\s/ - and the wide part is exactly what survives to it, since every ASCII whitespace character is a control character refused earlier, so what actually reaches it is U+00A0, U+2028, U+3000 and their neighbours. The label message said "part of the Authorization header", which is true of Bearer and false of api_key:, a YAML key far more often than a header. Both now say the accurate thing, and the tests assert the quoted prefix rather than the prose around it.

Ordered most specific first, which is not cosmetic. A pasted Authorization: Bearer <key> trips three rules at once, and the useful sentence is the one about the header rather than the one about a space. The scheme alternation is longest-first for the same reason: with bearer ahead of bearer-api, a Publer value would match the short one and the message would quote a scheme the operator never pasted.

A quote INSIDE a value is still allowed - only a matched wrapping pair is refused - and a regression test carries four real credential shapes from this estate's own catalogue so hyphens, underscores, dots, colons and mixed case keep working.

The module and its tests name every suspect code point as a number, never as the character. A file about invisible characters that contains invisible characters reads as binary to git and to grep, and the one byte nobody can review is the one nobody can see; integrations/rest-proxy.js recorded the same trap against its own control-character check. Both files are asserted clean of the characters they describe.

Not built, and that is the decision rather than an omission: the "shape" readout (length, alphabet, flags) from the deleted fix/358-secret-shape-diagnostic branch. It answered "is the stored string the one I copied", which #358 answered another way, and it would put a description of a secret on a page whose stated promise is that it never reads one back.

Confirmed live 2026-09-10, deployed with 5123493c. Three malformed values were pasted at the API-keys page and each was refused by name with nothing written: a value prefixed with an auth scheme, a value still wearing its quotes, and a value with a space in the middle.

The confirmation was run FROM A PHONE, which is the authentic instrument rather than a convenience. The curly-quote case exists because a phone keyboard substitutes them, so a desk browser can only simulate the error this check was written for. The quoted value is refused either as wrapped in quotes or as containing a curly quote depending on the keyboard, and both are correct - which is worth writing down, because a reader expecting one exact sentence would read the other as a failure.

  • Telegram, RSS.com and YouTube can be tested from the Integrations page, without their credentials ever reaching a browser (#483). The three held keys that are only ever read on the server, so their cards had a globe and no beaker: #481 pointed each globe at the page where its credential is minted, which is the best a card with no test can do, but "test everything possible with a simple GET" was not actually satisfied. connectionProbe is that test. lib/integrations/connection-probe.js runs one hard-coded read-only call per service — Telegram's getMe, RSS.com's GET /v4/podcasts, and a one-result YouTube search.list — behind the same editor guard the REST proxies use.

The caller sends a NAME, not a path. rest-proxy.js takes { path, method, body } and spends its header explaining why that is a confused deputy, with assertSafePath as the boundary that makes it safe. That boundary holds, but it is a denylist. Nothing here needs the latitude: the page asks one question per service and there is one call that answers it, so { probe: 'telegram' } is the whole request and the URL, method and headers are built server-side from a frozen table. The property is stronger than a validated path — the caller never supplies one. Object.hasOwn rather than a bare lookup, so constructor and __proto__ are refused like any other unknown name.

Two of the three authenticate in the URL, which made redaction a security property rather than tidiness. Telegram's token is the path and YouTube's key is a query parameter, so the ordinary habit of quoting a failed URL back into an error is a credential leak — the kind that ends up in a screenshot or in Application Insights forever. The built URL is never returned and never logged, and the upstream response is redacted before it is parsed, which is one choke point rather than four. That ordering was not the first draft: it redacted the error string and the log and passed the parsed body through untouched, and its own test caught the token sitting inside a provider error that echoed the request line back. The test asserts the absence on all three paths that can produce a string — a rejection, an unparseable body, and a transport error whose message quotes its target.

Only Telegram reports a key verdict, and the other two are a decision rather than an omission. getMe takes the bot token and nothing else and answers 401 when it is wrong, so a rejection has exactly one setting to blame — and TELEGRAM_CHAT_ID stays unprobed for the same reason, because a success says nothing about it. RSS.com's 401/403 split has not been measured here, and the YouTube Data API answers 403 for a disabled API, an exceeded quota and a referrer-restricted key alike; a light that reddens on a quota day would send the owner to remint a key that is fine, which is exactly what #358 cost two days. The beaker still reports those failures to the operator who pressed it, which is the point of the button.

The Telegram token keeps its literal colon, and that trade has a second half, and review found a gap in it. encodeURIComponent would send %3A, which Telegram would have to decode before routing; if it ever did not, the probe would answer 404 and the page would tell the owner their token is bad — a wrong answer to the one question the button exists to ask. So the token goes in unencoded and assertUrlSafe refuses any value carrying whitespace, a slash, a question mark, a hash or a backslash, naming the SETTING rather than quoting the value. It is not a defence against a caller, who never supplies this; it is a defence against a mis-pasted secret quietly changing which URL is called, of which a trailing newline is the common case. Redaction covers the percent-encoded form as well as the raw one for the same reason: an upstream that quotes its request target back shows the encoded bytes, and matching only the raw string would sail straight past it.

% joined that refusal list on review, and it is the least obvious member. Because the token goes in unencoded, a value carrying its own %2F is decoded by the far end into the delimiter the character class had just refused — so blocking the literal slash while allowing the escape that becomes one was a gap rather than a boundary. No real Telegram token contains a percent sign. The test fixtures were rewritten in the same pass: they read 123456789:AAH… and AIzaSy…, the genuine Telegram and Google key shapes, which a secret scanner cannot tell from live values and whose false positive costs a real rotation. They now say not-a-real-… and keep only the properties the tests need. Both found by Copilot review of 89288f8a.

Telegram answers { description } where the other providers answer message or detail, so upstream-error.js and its browser mirror proxyEnvelope.js learned that field — appended last, so nothing Publer, Klaviyo or Linkie resolve to can change. Without it the probe would print a bare HTTP 401, which is the thing #463 item 4 exists to prevent.

The YouTube probe costs about 100 of 10,000 daily quota units per press, because the Data API prices search.list per call and not per result. That is why it is a button, why the success message says the cost out loud, and why it must never become a timer or a page-load check.

One test changed rather than being added, and the direction matters. The page asserted that a credentialed service with no test points at the page where its credential is managed, guarded by expect(untestable.length) .toBeGreaterThan(0) so the rule could not go vacuous. These three were the last three, so that guard now fails for the right reason — which makes it the wrong guard. It is replaced by an assertion that the set is empty, which pins today's fact and hands the rule back the moment anything joins.

Confirmed live 2026-09-10, deployed with 5123493c. All three beakers were pressed: Telegram and YouTube reported connected, and RSS.com reported that RSSCOM_API_KEY is not set.

That third result is the feature working, not failing, and the distinction is the one this repository keeps paying for. RSS.com's key has never been seeded, so "not configured" is the only honest answer a probe can give - and it is a different answer from "the credential was rejected", which is precisely the confusion #358 spent days inside. A card that said "could not connect" for both would have been useless.

The deployment itself was confirmed without a credential, by a method worth keeping: POST /api/connectionProbe answered 401 while an invented route name answered 404, on a route that did not exist in the previous package. A guard rejection proves the route is registered; the 404 control proves the 401 was not the edge answering for everything.

The health endpoint cannot answer this, and the reason is a trap worth naming. It does report a generation, so it looks like a deploy identifier - but that value is RUNTIME_CONFIG_GENERATION, written by var.config_generation on a Terraform apply (T-513). It answers "which configuration generation is this worker consuming", which is a different question, and it does not move when a Functions package is deployed: it read gate2-a8fd3c8 both before and after this deploy. Reading it as a build stamp would report a stale package as a fresh one. Everything that would identify the build - runtime version, site name, feature flags - was stripped from that endpoint by T-402, because an unauthenticated inventory is what host enumeration looks for. So the route probe with a control is the available method, not a preference.

  • The Azure Learn catalogue refreshes itself every Monday, and the Friday Skills Hub scrape finally has a reader (#461 items 3 and 4). frontend/scripts/update-applied-skills.mjs had only ever been run by hand — the file said "Last manual sync: 2026-04-16" until #464 — so it now follows three rules: every run stamps today's date into DATA_AS_OF and the header line; a source that cannot be fetched or parses to zero items ends the run with exit code 1 and one sentence, and the file is not written (the title overrides no longer turn an empty poster into five entries); and lifecycle only moves forward (beta → active → expiring → retired), so a poster that still lists a retired exam, or an API that still says "(beta)" for one #464 made active, keeps the file's value and prints the disagreement — and an entry the sources no longer list is kept, never deleted, because dropping AI-102 would break its replacedBy link and the retired cards. A poster entry that neither the Learn API nor the file knows is reported and not added: the first live run parsed the PDF's "Full certification title" template label as exam MB-300. The regenerated blocks go through Prettier so a run that changes nothing but the date is a one-line diff. .github/workflows/update-learn-catalogue.yml runs it Mondays 06:30 UTC (and by hand) in two jobs modelled on the manifest workflow — npm ci --ignore-scripts and the updater with contents: read and no Azure identity, then a branch pushed with the manifest App's token and a pull request opened ready for review whose body lists every entry added, changed, removed, kept or disputed; scripts/learn-catalogue- workflow.test.mjs pins the single-file git add, the per-attempt branch name, the masked token and the absence of --draft. Run live on 2026-09-09 it added four applied skills the browse API lists (Build an agent in Microsoft Copilot Studio, Develop an agent with integrated tools, Manage GitHub secret scanning by using GitHub Copilot, Protect information in Microsoft 365 Copilot by using Microsoft Purview), renamed one (Build a generative AI chat app → Develop a Generative AI Chat App Using the Microsoft Foundry SDK), kept all twelve retired exams and five retired applied skills, and changed no status. GET /api/public/cert-events? platform= reads the certEvents container scrapeSkillsHubRss writes (lib/timers/skills-hub.js: id from the feed guid, type, certCodes, title, summary, link, pubDate stamped on every row, mentionedDates, source: 'skills-hub-rss', deduplicated by id) — platform validated against the provider keys, answered by scraper source since the documents carry no provider (only azure has one), projected to a listing allowlist, ORDER BY c.pubDate DESC, capped at 100 — registered in the route inventory and .azure/api-surface.json. The Azure education page merges the result over its static timelineEvents by id (lib/certEvents.js: the earliest date a post mentions on or after its publication day is the event's day) in an effect, so the pre-rendered HTML and the hydrating render carry the static entries alone and the static entries remain when the list is empty or unreachable; the data file's comment that claimed the array was scraper-sourced now says which side is which. The first real scrape is Friday 2026-09-11 09:00 UTC; until then the route answers an empty list.
  • The owner's button: Listen & Learn reads with Gemini "Best" or "Economy", set as a default and overridable per run (#436, #432). Owner instruction 2026-09-09: "have a button to go from gemini 2.5 to 3.1 and back — for newer technologies, better is best, for older, cheaper is best." Two Gemini TTS models are offered, both priced in COST_TABLE: Best — newest voice, about twice the cost (gemini-3.1-flash-tts-preview) and Economy — cheaper (gemini-2.5-flash-preview-tts). The stored default is a new admin_config/listen_and_learn_speech document, { geminiModel }, edited from a Listen & Learn voice card on /admin/platform as a two-option control with each model's per-episode ceiling beside it; the route accepts only the two ids and answers 400 by sentence for anything else, following the podcast-feeds card's validation and write pattern. The generation form gains a Voice model picker with the same two options, defaulting to the stored value and carried in the job payload as ttsModel, which the job refuses by sentence for any other id. Precedence in the job: the run's choice → the stored default → LISTEN_AND_LEARN_TTS_MODEL → the module default (lib/listen-and-learn/speech-settings.js; a settings read failure fails the run rather than reading in a voice nobody chose). The 202 and the queued toast name the chosen model and its cost, and the episode's speechModel and its usage row record which one read it. The rule of thumb on both controls — newer certifications: Best; older ones: Economy — is guidance for the person choosing, not automation; choosing by certification age is a follow-up.

  • One Recording Hub at /admin/recording-hub — podcast transcripts and Plaud in two tabs (#442; wires #434, presents #435). /admin/recordings redirects there and RecordingsPage.jsx is gone. The Podcast tab lists every podcast_transcripts document with its source chip (an article, linked to the editor and the public page; or a recording with its title), status, truncated and audioError badges and a player; opens the review view with the transcript beside its source — key takeaways for an article, the recording id and the generator's attributionLeaks as a warning for a recording; approves or returns to draft through the review route; renders the host.rsscom line (publishing… while approval's publish job is pending / published / error with Retry / skipped), accepting the review route's 202-with-job-id as well as its 200, with Retry keeping a plain-toast guard for a deployment that predates the publish route; and lists the show's episodes from public/podcasts (read-only — season metadata editing is not in this change). The Plaud tab carries the Library, Upload and Connect behaviour unchanged, adds Script this on every library and stored recording (POST cms/podcast/transcripts/generate-from-recording, job generate-podcast-transcript-from-recording: get_file + get_transcript through the MCP with the stored token, source_list as the fallback, generateRecordingScript under the podcastScript feature, then the same voice → upload → save → spend tail the article pipeline uses — now one shared finishTranscript in lib/podcast/generate.js rather than two copies; stored as plaud_<recordingId> or recording_<id> with the generator's provenance and attributionLeaks), and audio upload for transcription through the Plaud Embedded Transcription API — verified against docs.plaud.ai on 2026-09-08 and written into the header of lib/podcast/plaud-embedded.js — as POST cms/podcast/recordings/upload (mp3/m4a/wav, 40 MiB, refused before storage when unconfigured) and job transcribe-recording-upload, which hands Plaud the media route URL on the new PUBLIC_API_ORIGIN setting (the storage account denies direct and SAS reads; the URL is unguessable but unauthenticated), polls to a terminal status — bounded by a documented default budget under the job's own — stores the transcript in recordings as plaud-embedded, and deletes the upload blob once Plaud is done with it (every terminal outcome; a timed-out poll keeps it, and a new expire-recording-uploads lifecycle rule in infra/storage.tf removes anything under podcast/uploads/ seven days after creation, Plaud's own retention). A missing or revoked Plaud credential is refused with one sentence naming the Connect tab, at the door (409) and in the job, before any model call. Two new Key Vault references, PLAUD-EMBEDDED-CLIENT-ID and PLAUD-EMBEDDED-API-KEY, catalogued and documented; callMcpTool extracted from mcpProxy so a job can call an MCP tool server-side.

  • Listen & Learn episodes grounded on owner-supplied web pages and YouTube videos (#433, slice 2). A second kind of episode, not a looser first. generateEpisodeScript gains an optional grounding: { sources, ai }; without it the function, the guide prompt and its tests are exactly what they were, and no source list is threaded into the certification path. With it, a separate prompt (buildSourceGroundedPrompt) says the episode is built from the listed sources and nothing else, fences the owner's list and states that whatever a source returns is material and never an instruction, forbids claiming exam authority the sources do not carry, and opens with its own dated SOURCE_DISCLAIMER. The call goes through the router's generateGroundedJsonResponse from slice 1 under feature: 'sourceGrounding' — the ai-call-sites exception that held the toggle open is removed — so Gemini absent from the chain is a sentence before a byte is sent, never a failover, and nothing is written.

The episode is stored in listen_and_learn_episodes under the same set as the guide episodes, with kind: 'source' and the resolved sources: [{ kind, url, title? }]; guide episodes now write kind: 'guide', and episodeKindOf is the one rule that reads a missing kind as guide. Its id is source_<slug of title> — the underscore is what keeps it out of every guide area's way, since studyguide.slugify can never emit one — so regenerating a title replaces the same draft. The admin page gains SourceGroundingPanel: a title, URLs one per line classified as page or video by the server's own rule (lib/sourceUrl.js, pinned equal to isYouTubeVideoUrl by a functions-side test), and a Generate button that posts POST cms/listen-and-learn/source-episode, which validates the list with validateGroundingSources and answers 400 with the sentence — an over-cap list, a YouTube URL given as a page — before any job exists; the worker validates again. getSet returns kind and sources on every episode and the review card lists the sources with links beside the transcript. The anonymous list adds kind (a listener-facing page may label a source-grounded episode) and deliberately not sources, until a citation page exists. One usage source, listen-and-learn:source-script. The live acceptance — one article URL plus one YouTube URL producing a draft that reflects both — spends on the Gemini key and is owner-run.

  • Approving a podcast transcript publishes it to RSS.com (#437, slice 2; ADR 0029 §1b). POST cms/podcast/transcripts/review moving a transcript to published now runs the host step slice 1 built: it queues a publish-podcast-transcript platform job, marks the document host.rsscom.pending with the job id, and answers 202 so the Recording Hub can show "publishing…". The job (functions/src/functions/podcast-jobs.js) reads the MP3 from the podcast container, calls publishEpisodeToHost, and stores the record — episodeId, guid, hostStatus, publishedAt, or error — under host.rsscom; the platform's onComplete clears the pending marker if the job dies around the step. A job rather than an inline call because the client's own deadlines (30 s presign, 300 s PUT, 30 s create) sum past the 230 s HTTP budget, and because a step whose failure must not change the approval has no business holding the approval's response.

Three outcomes are recorded as skipped, never as an error, so the hub says what is true: not_configured when RSSCOM_API_KEY or RSSCOM_PODCAST_ID is not seeded (the reason is isConfigured's own sentence), no_audio when the transcript generated without audio (nothing is uploaded without audio), and not_published when a transcript was withdrawn to draft between enqueue and run. A publish failure never un-approves: status, approvedAt and approvedBy are written by the review and by nothing in the host path, and the tests assert the host patch carries only host.

Retry is a route, not a regeneration. POST cms/podcast/transcripts/{id}/publish (publisher) refuses anything not published with 409, refuses to double-queue while a publish job is still in flight, and otherwise runs the same step — which PATCHes the host episode the document already names rather than creating a second one. custom_link carries the article's public URL when the article document has one (publicUrlOf, the Social Hub's precedence). The feed remains the ingest boundary: nothing here writes podcasts, and the site learns of the episode when timers/podcasts.js reads the show's feed. Listen & Learn episodes still do not publish, pending the fourth decision on #349. Nothing has been sent to the live API yet — the two secrets are unseeded — so the first real publish is the owner's, with the key. lib/podcast/publish-transcript.js + tests; handlers.js, podcast-http.js, podcast-jobs.js; .azure/api-surface.json.

  • The AI router can ground a generation on web pages and YouTube videos, and can no longer lose a prompt part on the way to a model (#433, slice 1). generateGroundedJsonResponse({ prompt, sources, feature, … }) takes { kind: 'page' | 'video', url } entries and sends them to Gemini's Interactions endpoint — the same /v1beta/interactions the Listen & Learn speech side has called since 2026-08-24 — as url_context for pages and { type: 'video', uri } input for videos. Nothing calls it yet; the source-grounded episode kind is slice 2.

It fails rather than fails over. The call resolves the provider chain exactly as every other call does — portal order, disabled providers, CONTENTFORGE_AI_PROVIDER — and then requires Gemini to be in it, because Gemini is the only provider that can read either source type. If it is not, the call throws AI_NOT_CONFIGURED with the reason: the key is not set, it is disabled in the portal, or the pin names another provider. It never hands on to OpenAI or Anthropic, which would answer confidently from the prompt alone. Sources are validated before anything is resolved — http(s) only, YouTube URLs are video and only YouTube URLs are, duplicates dropped, at most 20 pages (Google's url_context limit) and 10 videos — and an over-cap list is refused in a sentence rather than truncated. The prompt tells the model that whatever a source returns is data, never instruction, and every URL passes through the same fenceArticleText the article episodes use, so a source cannot close the caller's fence.

A dropped part is now impossible. toGeminiParts, toOpenAiContent and toAnthropicContent used to return null for any part shape they did not recognise and filter it out; a fileData part — the shape the issue's first comment assumed a YouTube URL would take — reached the model as nothing, and the model answered without it. Each converter now throws a 400 for a shape it cannot carry, which neither retries nor fails over. The one production caller passing parts (inspect.js alt text) sends only text and image/* inline data and is unaffected. The Source grounding feature toggle joins the AI Engine catalogue; ai-call-sites.test.js scans the new entry point and carries a checked, self-expiring exception for the toggle until slice 2 gives it a call site.

  • Any published article can become a podcast transcript (#435, slice 2 of #432). The generator #439 left unwired is now wired: a per-item Podcast transcript action on /admin/published — the end of the ContentForge workflow, on the already-live list only, because "published" is what makes an article final enough to read aloud — confirms, then POST /api/cms/podcast/transcripts/generate enqueues the new generate-podcast-transcript job. The job loads the article, refuses unless isPublicDocument says it is published (before the model is called, so a refusal costs nothing and writes nothing), scripts it through generateArticleScript, synthesises the audio, uploads it and saves the transcript as a draft. Reading and approving are GET /api/cms/podcast/transcripts, GET …/{id} and POST …/review — the last at publisher, because approval is what #437 will hang the RSS.com publish on.

Its own container, by owner ruling. The earlier analysis on #435 had article episodes borrowing listen_and_learn_episodes under a reserved setId; the owner ruled on 2026-09-08 that Listen & Learn is the Learn section's product and content transcripts are the podcast's, so they are two containers: podcast_transcripts (Cosmos, /id, ids article_<slug> with plaud_<recordingId> reserved for #434) and podcast (blob, article/<slug>.mp3). Both are declared in Terraform and wait on an owner-confirmed apply. Until it runs, the job fails with a sentence naming the Cosmos container and the apply rather than a bare Resource Not Found, and a missing blob container is recorded in audioError with the transcript still saved — every audio-stage failure degrades that way here, because the transcript is the artefact and #436 re-voices it. The container is class A in the export (44 authored, 61 full, 54 delta), the audio is served through the media route, spend is recorded under podcast:script and podcast:audio, and the model call answers to its own podcastScript portal toggle rather than Listen & Learn's. Section counts and the content manifest do not count transcripts yet: nothing public reads them until the article-side player lands, and nothing in the portal lists them until the Recording Hub (#442) ships — the confirmation says so.

  • Groundwork for publishing episodes to RSS.com: the API client, the idempotent host-publish step, and the two Key Vault references (#437, slice 1; ADR 0029 §1b). Nothing calls the new step yet. The owner approved RSS.com Max on 2026-09-08 so that, eventually, approving an episode uploads it to the show rather than leaving the owner to upload it by hand; the approval hook that makes that happen, and the admin retry, are the next slice. What this slice settles is how the host is spoken to and what a publish writes on the document, so that the hook is a wiring change rather than a design one. The issue's first task was to confirm whether the API had left beta; it has not. Checked 2026-09-09: RSS.com's help article says the API "was introduced in December 2025 and is currently in beta" and that "small changes to the documentation or endpoints may occur". So the client is written to the issue's beta-tolerant design rather than waiting for a date nobody has given.

functions/src/lib/podcast/rsscom.js is the API, read from https://api.rss.com/v4/openapi.json on that date and recorded in the module header: X-Api-Key; GET /v4/podcasts; a presigned upload (POST …/assets/presigned-uploads, then the bytes PUT to the returned URL with no key attached, because it is a storage host); POST and PATCH …/episodes; GET one or many; PUT …/transcript. It has timeouts, a typed RssComError carrying the upstream status and detail, and no retries — retrying a POST …/episodes inside a client is how a duplicate lands on a public feed. isConfigured(env) answers with a sentence naming which of RSSCOM_API_KEY and RSSCOM_PODCAST_ID is missing, and an unresolved Key Vault reference counts as missing, the same rule every other key here follows.

functions/src/lib/podcast/host-publish.js is the step, and it is idempotent on the document: host.rsscom.episodeId present means PATCH that episode, with audio re-uploaded only when audioPath changed since the last publish; absent means upload, create, and record the id. It returns the patch to store and never throws — a failure lands as host.rsscom.error = { status, code, message, retryable } beside the previous host record, and status, approvedAt and approvedBy are never in the patch, so a failed publish cannot un-approve anything. The host is told the truth: ai_content: true on every episode, plain-text notes built from the summary and key takeaways, and schedule_datetime set on create because the spec documents that as "auto-publishing as soon as transcode completes" — without it the episode would sit as a draft on the host and the feed would never carry it. One gap is recorded rather than hidden: a create whose response is lost before the id is stored would be repeated, because the spec offers no client-chosen key.

The two settings are Key Vault references in infra/functionapp.tf (RSSCOM-API-KEY, RSSCOM-PODCAST-ID), catalogued on the API-keys page under Social & audience, and listed in Required-Inputs §4.6. Not in this slice, by design: the approval hook, the admin retry, and any ingest change — the feed stays the way an episode reaches podcasts.

  • A script generator for recording transcripts, sibling to the article one (#434, generator only; #446). generateRecordingScript in functions/src/lib/listen-and-learn/recording-script.js takes a Plaud transcript and returns the same script shape generateEpisodeScript and generateArticleScript return, through the same validation, byte fitting and prompt-injection fence — imported from script.js and ai/prompt-fence.js, with tests that swap the shared fence for a marker and prove the marker reaches the prompt, so a copy cannot drift. Nothing calls it yet: the admin surface, the stored episode and the USAGE_SOURCES entry are the next slice, and the returned source object carries what that slice needs — kind: 'plaud', the recording id, title, time, duration, segment counts and whether the input was cut.

It is a retelling, not a transcript. The issue's rule is that the two hosts must not attribute lines to the real people in the recording, and a diarised Plaud transcript invites exactly that: every segment carries Speaker 1, Speaker 2 or a name the owner typed in, and the obvious script maps those onto Maya and Elena. So the labels never reach the model. renderTranscriptForPrompt writes each segment as one line and a paragraph break where the speaker changes — the shape of the conversation, with nothing that could name or number a person. A persistent neutral marker ([A], [B]) was rejected because it is an alias, and a model that can say "the second participant argued" is one sentence away from "Speaker 2 said". The prompt forbids naming or quoting anyone who spoke, the disclaimer says out loud that the voices are not the participants, and the result reports any transcript label the finished dialogue repeated verbatim, for the review surface rather than as a refusal.

Empty or trivially short transcripts are refused before any spend, for the reason resolveArticleBody throws: a prompt with nothing in it produces a plausible episode about the title. Measured live on 2026-09-08, a 28-second device check has an empty source_list, which is the file this catches. The input ceiling is 120 kB — about two hours of speech — and the cut lands at a segment boundary, because a half sentence at the end of the fence reads to the model as a transcription error to smooth over rather than an edge to respect. The prompt is told, and so is the result.

normalizePlaudTranscript accepts the measured get_transcript shape (segments[] of start_time/end_time/content/speaker in ms), the get_file shape with its source_list, a bare utterance array, a pasted plain-text transcript, and the { transcript } document POST cms/recordings already stores. One thing is inferred rather than measured and the header says so: the key that holds the utterances inside a data_type: "transaction" entry, because both files sampled had an empty source_list; the normaliser finds the array by shape.

  • ElevenLabs reads Listen & Learn, and the run says what it will cost before it starts (#436, ADR 0029 §2a). The owner approved a paid ElevenLabs plan on 2026-09-08, which met the revisit trigger ADR 0029 §2 wrote for itself; §2a records the amendment and this is the code for it. speech/elevenlabs.js is a third entry in the provider switch, first in preference order when ELEVENLABS_API_KEY resolves, under the same contract as its siblings: MP3 out (mp3_44100_64, the 64 kbps the Gemini path already encodes to), so the blob path, the stored contentType and the player are untouched. Every turn is an inputs[] entry with its own voice_id — Sarah for Maya, Aria for Elena, both premade, overridable per host — so a turn whose speaker has no voice is a hard error before a byte is uploaded, not after. Dialogue is chunked at the endpoint's documented 2,000-character ceiling and the parts concatenated, as the Azure path already does.

The test that said elevenlabs was not a provider is gone, on purpose. It was the record of the deferral; speech/index.test.js now records the order instead, and the assertion it replaced is described in that file's header rather than deleted quietly.

Out of credit is a state a paid provider has and a free one does not. The API answers it with 401 quota_exceeded; that is surfaced once, with its status and a code, is never retried, and — unless a pin says otherwise — moves the switch on to Gemini, which stays as the fallback for exactly this. Any other failure still fails the area, because a fault hidden behind a different voice is worse than a gap.

Cost is recorded and stated. A character is the output unit: usage rows carry the billed count from the API's character-cost header (our own count, flagged estimated, when the header is absent), and COST_TABLE.elevenlabs prices it at USD 100 per million — USD 0.10 per 1,000 — so getCostEstimate and the usage page work unchanged. The job's 202 now carries speech: { provider, estimatedCostUsd, … }, a ceiling computed against the script byte limit per episode, and the admin page prints it the moment the run is accepted: "Queued — speech by ElevenLabs, up to $7.20 (8 episodes × $0.90)". ELEVENLABS-API-KEY is declared in infra/functionapp.tf and the secret catalogue; seeding it is the owner's step, and until then the switch moves on to Gemini as before.

Incidentally fixed in the same page: useAuthReady() returns authReady, and the page destructured ready, so the sets list never loaded on mount.

  • scripts/check-workflow-health.mjs — the guard that would have caught the three weeks. Split out of #426, which retired the workflow that prompted it on 2026-09-08. validate-deployed.yml explained in its own header why it could not pass from a GitHub-hosted runner, and still sat in .github/workflows/ from 2026-08-18 to 2026-09-08: the knowledge was not missing, nothing was reading it. This reads run history from the Actions API and names any workflow with no success among its sampled runs, at least one failure among them, and no attempt of any kind newer than ten days. Run against the live repository on 2026-09-08 it named validate-deployed.yml and nothing else — no false positives across the other eighteen.

It measures rather than asserts. There is no list of workflows in it to keep in step with the directory — a hand-maintained list would have needed somebody to notice the thing it exists to notice. A workflow added tomorrow is covered tomorrow.

Three verdicts, kept apart. healthy, broken, and unproven, the last being "this tool is making no claim" and reached three ways: nothing finished to read (never ran, or still running); everything that finished stopped short of a verdict (cancelled, neutral, action_required, skipped, stale); or a real failure still inside the staleness window, which is somebody mid-debug rather than an abandonment. Counting a cancellation as a failure would have flagged deploy-azure-frontend.yml, which was used successfully three times the same morning. Staleness is part of the failure condition rather than a separate finding, because a detector that fires on ordinary work stops being read — the failure mode that allowed the three weeks.

broken means "nothing succeeded and something failed", not "every run failed", and the message says which. Review asked for the stricter rule, with a mixed sample demoted to unproven; it was declined on the evidence. A workflow that failed a month ago and had a run cancelled a month ago has failed, has never worked and has been left — and under the stricter rule it would be printed under a line reading "none of which reached a verdict", about a run that reached a verdict of failure. One stray cancellation would also have been enough to hide validate-deployed.yml itself. What keeps the looser rule honest is that the verdict prints its own arithmetic: "N of M sampled run(s) failed, none succeeded".

A trap found while building it, recorded in the file. The first draft called /actions/runs?workflow_id={id}. That form returns HTTP 200 and IGNORES the filter, handing back the repository's newest runs across every workflow — so all nineteen came back "healthy", including the one whose entire history is two failures. It is /actions/workflows/{id}/runs. The script now also asserts that every returned run belongs to the workflow it asked about, and exits 2 rather than 0 if that does not hold, because a silently unfiltered response is indistinguishable from good news.

It runs as a second JOB in monitor-deploy-drift.yml, not a new file. The owner's instruction on 2026-09-08 was that there are too many workflows; that file already holds actions: read, already runs Node without npm ci, already runs on a schedule, and is already the delivery-health monitor (renamed "Monitor delivery health"). A separate job rather than a second step, because "the frontend is four days behind" and "a workflow has been failing since August" are different findings, and one job conclusion would merge them into a red X that says neither.

The job captures stderr, and that is load-bearing. Every exit-2 path in the script writes with console.error and nothing to stdout, while the job captured the report with report=$(node ...). Measured: exit 2, zero bytes captured — a red job with a blank step summary, in the one case an operator most needs to read why. The invocation now redirects with 2>&1, and check-workflow-health.invocation.test.mjs pins both that and the capture-then-read-$? shape, because node ... | tee would report tee's status and never fail.

Every refusal path exits 2, including the ones that used to throw. An uncaught throw from the awaited main() exits 1 — measured — which in this tool means "a workflow is broken", so a response the script could not read was reported as a repository containing a broken workflow. Two places did it. main filtered the listing on w.path and asked for runs by w.id, and an entry carrying neither threw at w.path.startsWith(...); listingRefusal now checks entry shape alongside pagination. And the run listing was read as got.workflow_runs || [], where a missing array became "no runs" — a silent mis-measurement reported as unproven — while a truthy non-array threw at runs.filter(...); that guard is now runsRefusal, pure, exported and tested, and it carries the workflow_id filter check that used to sit untested inside main. The requests also pin X-GitHub-Api-Version: 2022-11-28, matching check-deploy-drift.mjs (the other job in the same workflow file), github-app-token.mjs and open-manifest-pr.mjs.

Covered by scripts/check-workflow-health.test.mjs (42 tests, real run histories as fixtures), check-workflow-health.invocation.test.mjs (4), and one case in entrypoint-guards.test.mjs; the suite goes 295 → 342, main having dropped from 300 to 295 when #426 deleted powershell-hygiene.test.mjs with the script it read.

Every one proven load-bearing by mutation, each reverted afterwards. Implementing the stricter broken rule failed four tests, two of them written for earlier review rounds, with expected 'unproven' to be 'broken' and the demoted workflow printing 2 run(s), none of which reached a verdict. Making the broken message claim "every sampled run failed" failed the arithmetic test. Removing 2>&1 from the workflow invocation, and replacing the capture with a | tee pipeline, each failed their own assertion in the invocation test. Removing the entry-shape guard from listingRefusal failed with listingRefusal accepted {}: expected null not to be null. Restoring got.workflow_runs || [] failed the shape test with TypeError: Cannot read properties of null — the defect demonstrating itself — and putting the leading truthy check back into the workflow_id comparison failed fails CLOSED on a row it cannot identify at all. And reverting the entry-point guard to the file:// template form failed with expected +0 to be 2, the script exiting 0 on Windows having run nothing — the same bug that once shipped in smoke-deployed.mjs.

  • The Plaud Connect tab takes the refresh token too (#342, T-518 Wave 5). It stored only the access token, while refreshPlaudToken rotates the pair with the refresh token and Plaud's access token lasts about a day — so the Library lapsed daily and the timer, once armed, would have marked the document disconnected at its first firing. The tab now has a second field for refresh_token (both come from ~/.plaud/tokens-mcp.json), the API stores it write-only beside the access token — never returned, carried through PUT round trips, reported as hasOauthRefreshToken — and the banner says whether auto-refresh is armed. Reviewed against Plaud's documentation on 2026-09-05: the quickstart's client id and secret belong to Plaud Embedded, a different product (device SDK and transcription API) that nothing here uses; the MCP tokens are the two values that matter.

  • cleanupSoftDeletedContent is dry-run until CONTENT_HARD_DELETE=true, never deletes a document whose deletion mark has no recorded origin, and reports every run (#334). Before this the hard reaper deleted anything carrying softDeletedAt older than seven days, and three different writers put that mark on a document: the admin soft-delete route, which also records deletionRequestedBy; and the two rejected-content agers, which record softDeletedReason: 'rejected_aged_out'. A migrated document with an old mark from the Firebase days carries neither, and it would have gone at the first firing. Now the pin is the T-302 rule applied to the one timer that deletes documents rather than blobs — infra/functionapp.tf seeds it "false" beside the two blob pins — and origin is the rule the pin does not lift: a mark with no deletionRequestedBy and no softDeletedReason is counted, logged at Warning as a count, recorded by id in the audit entry that every armed run writes once it has examined anything (a dry run writes no audit document), and left for a human in the admin content queue's soft_deleted filter.

Every run now logs one summary line, idle runs included: in dry-run, what it would delete by origin; when armed, what it did. Before this an idle run wrote nothing, not even its audit record, which is why Wave 3a had no witness at all. Traces carry counts only; ids go to the audit document.

  • T-518 Wave 2 armed and observed: the two feed-ingesting timers now run, and podcasts has its first production write (#332). SYNC_RSS_FEEDS and FETCH_PODCAST_FEEDS were added to enabled_timers in the hcw-azure workspace in one apply on 2026-09-03, ahead of the 00:00 Chicago window. Same class, so they grouped: both ingest feeds and create content documents, neither deletes anything, and re-delivery is safe by construction — cacheFeed upserts rss_cache per feed, and processFeed reads each episode by id and preserves createdAt before upserting.

Observed through the durable side effect, not the host trace — the first wave closed on the #328 witness. The Executed rows the runbook's gate 4 used to read have not existed since #321 (T-766), so the evidence is what each timer leaves behind, read through the public API by scripts/verify-timer-witness.mjs:

  • syncRssFeeds — PASS on the first read: every rss_cache document re-stamped, newest refreshedAt 2026-09-03T07:00:16Z, which is the 02:00 Chicago boundary of its every-two-hours schedule to the second. Nothing else writes that stamp at that minute — the admin fetch-rss-feeds job had last run the previous evening — so this is the timer.
  • fetchPodcastFeeds — FAIL on the first read (nothing newer than the May migration after the 05:30Z firing), then PASS at the 13:30Z firing on 2026-09-04, the first after the #330 deploy restarted the host at 13:24Z. podcasts.updatedAt is re-stamped on every episode each run, so a fresh stamp is the handler completing an upsert, not merely starting.

What the record cannot say. The 05:30Z miss is not explained. That firing predates #330, so whatever happened — a feed that did not answer, an empty feed, or a handler that did not run — was logged at Information and dropped by the Warning gate; and the firings between the two reads were not read. The next miss will be readable: since #330 a failed feed writes an Error row and an empty feed a Warning row in the Function.fetchPodcastFeeds category, with the query in the #330 entry below. This is the collision T-766 describes, met on the first wave to arm after it.

T-764 does not close on this entry. Its writer is observed; the closing condition also asks for the azure podcast page to render an episode, and that read is still owed.

  • The arming gate reads the timer's durable side effect through the public API, and needs nothing else (#328). scripts/verify-timer-witness.mjs takes a timer name and an ISO --since, fetches that timer's witness from the Cloudflare-fronted API — rss_cache.refreshedAt for syncRssFeeds, podcasts.updatedAt for fetchPodcastFeeds, content.publishedAt for publishScheduledContent — and says whether the newest stamp is at or after the moment named. No az, no workspace, no extension, no telemetry plane: three fewer places for an observation to be lost. A malformed stamp never counts as evidence; an unparseable --since refuses rather than comparing against NaN. There is deliberately no cron parser — the schedule is something the operator already knows, and a parser is a second thing to be wrong about.

Fifteen of the eighteen timers have no public witness, and the script says so with exit 2 rather than a pass or a fail, because "cannot evaluate" and "evaluated and failed" are different findings. The runbook's witness table now covers all eighteen with the reason for each no, and the script's test asserts that table names exactly the timers schedulers.js and jobs-sweeper.js register — a timer added without a row fails CI instead of arriving at a cutover with no gate. Thirteen tests; the drift guard was mutation-checked by deleting a row and watching it fail.

Also records what #327 deferred: the log-analytics extension is now Step 0's sixth item in the Cutover-Runbook, and that step's heading no longer carries a count of its own items.

  • Every provider page is live: the last 24 shipped-dark pages came out from behind ComingSoonPage (#325). finops, gcp, github and terraform now serve their full sections alongside aws, azure, vmware and ansible. Owner decision 2026-09-02, taken as one release rather than section by section.

The pages were never unfinished. Each is structurally identical to its live AWS or Azure counterpart — the same shared component with a different provider prop — so the guard was two lines and nothing else: ComingSoonPage's import and an early return above the real one. The routes were already public; visitors have been landing on these URLs and reading "coming soon". What changed is what those URLs answer with.

GUARDED_FILES in frontend/scripts/validate-provider-pages.js is now empty, and the array and its checker deliberately stay. The mechanism is what makes shipping a page dark reviewable: a page behind ComingSoonPage with no entry there is an accident, and an entry there with its markers gone is a page that went live without anyone deciding. Deleting the empty array would delete that check for the next page that needs it. The validator reports 55 live files, 0 guarded.

Eighteen hardcoded date: 'Updated <month> 2026' stamps were removed rather than refreshed, across terraform/ToolsPage, terraform/ModulesPage and finops/FocusPage. Nothing maintains them: they were written months ago and would have gone live in September still claiming February. A date no process updates is a stale claim on a page that will never change, and removing it is the fix that does not need doing again — the same reasoning applied to the timer counts in TODO.md the same day.

  • T-518 Wave 1 armed and observed: the platform's two safe timers now run (#323). PLATFORM_JOB_SWEEPER and MONITOR_PUBLISHING_PIPELINE were added to enabled_timers in the hcw-azure workspace in one apply, taking the estate from three armed timers of eighteen to five.

Two timers in one apply is a departure from Cutover-Runbook step 5, and it was an owner decision rather than a shortcut. Step 5 reads one timer per apply, observed before the next. Taken literally across the fifteen that remain that is fifteen applies over roughly five weeks — three of the fifteen fire weekly, so their observation windows alone are three weeks — and every one of those applies restarts the Function App through the workspace's permanent azapi diff. What the rule exists to prove is that arming works at all, and that was settled before this wave: the mechanism had been observed three times, once across the apply boundary itself (publishScheduledContent, four skipped invocations then four ran, with the flag as the only variable). What remains is per-handler behaviour, which groups by risk. The precedent is CHECK_AGENT_HEALTH and CLEANUP_TEMP_STORAGE, armed together on 2026-08-30 and recorded as a departure with its justification.

What still does not group, and this is the limit of the concession: the two timers that delete documents with no dry-run pin — CLEANUP_SOFT_DELETED_CONTENT and CLEANUP_REJECTED_CONTENT — stay one per apply. Grouping is a concession to calendar arithmetic, not to destructive operations.

These two were the right first wave because neither can damage data. platformJobSweeper re-enqueues jobs that have sat queued past STALE_QUEUED_MS, closing the gap in functions/src/lib/jobs.js where the job document is written before the queue output binding sends the message — so a binding failure used to leave a job queued forever. Re-delivery is safe by construction: the worker claims with an etag-conditioned replace, so a duplicate message for a job that did start is skipped rather than double-processed. monitorPublishingPipeline is a read-only watchdog.

Observed, not merely applied. The standard is step 5's fourth gate, and all four were read from the workspace with scripts/cutover/05-verify-timer.ps1 on 2026-09-02:

  • Deployment: both flags read true from a live az functionapp config appsettings list against func-site-prod-cus-01, with the armed set printing as CHECK_AGENT_HEALTH, CLEANUP_TEMP_STORAGE, MONITOR_PUBLISHING_PIPELINE, PLATFORM_JOB_SWEEPER, PUBLISH_SCHEDULED_CONTENT.
  • Runtime: platformJobSweeper registered with schedule 0 */15 * * * *, read back from the host rather than from source.
  • Behaviour: the sweeper's own re-enqueued … stale job(s), reaped … abandoned running job(s) line present in the trace stream, which is the handler reporting its work rather than the host reporting an invocation.
  • Invocation: both timers observed firing inside their windows.

The clock half of the gate applies to only one of the two, and saying so matters more than the counts do. platformJobSweeper fires every fifteen minutes, so it has no local-time dependency and the WEBSITE_TIME_ZONE trap cannot express itself in its history — for it the gate is the count and the handler's own line, and a timestamp check would prove nothing. monitorPublishingPipeline runs 0 0 */6 * * *, which is 00:00, 06:00, 12:00 and 18:00 Chicago, so it is the one that can fire five hours early and still pass a naive "did it run" check. Its ScheduleStatus.Last landed on the intended local hour. A wave that reported one number for both timers would have proven the weaker thing twice.

Arming two timers made four counts elsewhere in TODO.md wrong at once, and they were removed rather than corrected. The section heading, the master-table row, the attack-sequence phase and an owner-decisions row each said "the remaining 15 timers" — accurate when three of eighteen were armed, wrong the moment five were, and wrong again after every wave that follows. That is the T-722 defect, in the file whose own wave table already carries the rule ("this table is the plan, and no count above it is"), so the fix is the rule rather than an arithmetic update: the counts are gone and the enumerated sentence that lists the armed timers by name — which cannot drift from itself — is what remains. The one remaining count in that section was found by review: enabled_timers was still described as "holding those three".

The per-run figures are not restated here. They were read by the operator at their own prompt and the evidence standard is the observation, not a number copied into a second document where it can drift from the first — the same reason infra/frontend.tf states its ingestion reduction as a floor rather than a measurement.

  • The runbook now covers signing in, because five separate az failures cost round trips in two days and none of them named its own cause (#323). wiki/Cutover-Runbook.md gains Step 0, ahead of the Entra step: the estate's single tenant and its four subscriptions in one table, the scoped device-code login, the subscription pin, and the az account show line that says whether any of it worked.

The five, each recorded with what it actually was rather than what it looked like: a plain az login handing off to the Windows WAM broker and spinning with nothing to read; AADSTS700082, which reads as a permissions problem and is a cached token past its 90-day inactivity window; a cached tenant that is not the estate tenant — the expensive one, because reads then succeed and return nothing, and an empty result from the wrong estate is indistinguishable from a real absence; a tenant-wide login that enumerates everything and leaves scripted reads apparently stuck, with the az logout → az account clear → scoped-login reset written out; and AADSTS50076 MFA warnings for two tenants that are not this estate — one of them named hybridcloudworks.com, which is the trap — printed several loud lines above the results table, so a command that fully succeeded reads at a glance as one that failed.

The tenant id is written out rather than referenced. It is not a secret — Entra publishes it for any domain at that domain's OpenID configuration endpoint — and the alternative is a placeholder in a command meant to be pasted, which .claude/CLAUDE.md prohibits, having twice paid for one. Subscription ids were already written out in this file and in scripts/cutover/05-verify-timer.ps1; no credential, key or token appears.

Also recorded there: sub-plat-mgmt-prod-cus and rg-mgmt-plat-prod-cus swap their two middle tokens, which is easy to write backwards and fails as a not-found against a name that looks correct.

  • TODO.md's handling rule is now a merge gate, not prose. scripts/check-todo-changelog-movement.mjs, run by the Repository Policy workflow on every pull request, fails when a T-identifier leaves TODO.md without CHANGELOG.md carrying it — the "completed items are removed after the corresponding entry is present" rule, enforced at the same head. Renumbers and moves within TODO.md pass (the id still exists); earlier changelog entries pass (the changelog only grows); and a base that cannot be read exits loudly rather than passing, because a gate that cannot evaluate has not evaluated. Unit-tested for all four shapes and mutation-tested against a fabricated removal.

  • The repository gains its own code-review skill (#316). .claude/skills/hcw-code-review/ teaches an agent to review a diff the way this repository's CI and reviewers do: SKILL.md scopes the change, routes each touched path to a per-component checklist (frontend, Functions, infra, scripts/workflows, VPS agent and edge probe), and applies the cross-cutting checks the PR template and Repository Policy already enforce — secrets, content-free telemetry, pinning, the Markdown allowlist, TODO/CHANGELOG movement, and the owner-facing instruction rules in .claude/CLAUDE.md. Each reference file carries the component's real verification commands — the ones CI runs plus the review-time validators CI does not — so a review's Verification section reports what was actually executed rather than what was assumed. The directory lives under .claude/, already allowlisted as a harness directory in scripts/validate-repository-structure.ps1.

  • T-519 closed: the reachability alert is armed, and the estate's one outage-surviving signal finally pages (#315). The blocker was what wrangler.toml predicted: the Worker's secret held the Instrumentation Key, not the connection string, so parseConnectionString threw on every */5 invocation and availabilityResults stayed empty. Fixed 2026-09-01 with the piped command in edge/availability-probe/wrangler.toml — the value never touched a screen or clipboard — and verified as a chain, not a deploy: wrangler tail showed a clean Ok invocation; twelve success == 1 / HTTP 200 rows landed on the 5-minute cadence (18:25–18:45 UTC and onward), twice the six a full PT30M window needs, so the rule's first evaluation ran against a populated window; availability_probe_alert_enabled = true was applied in the hcw-azure workspace; alert-api-reachability-prod-cus now lists among the four scheduled-query rules in rg-web-site-prod-cus; and the registered function count read 122 both before and after the apply's expected Function App restart. Justification unchanged from the 2026-08-31 measurement: the GitHub-scheduled half of the detection pair delivers 22% of its hourly runs with a 12.7-hour worst-case blind window, and the probe runs on a scheduler GitHub cannot drop. Removed from TODO.md, whose open list drops to four; the standard Azure web test stays disarmed in Terraform (Bot Fight Mode, ADR 0024) with the Worker as the approved path around it.

  • TODO.md gains one working order across everything open (#314). A repo-wide sweep (dedicated tracker, wiki backlogs, inline markers, notImplemented contract) confirmed the five-item table is complete, then added an "attack sequence" section that sequences all of it — the five items, the settings sweep, the optional seeds, the live confirmations — into seven dependency-ordered phases. It adds no items and restates no procedures: each phase links to the one section carrying the commands and success criteria, so the sequence cannot drift from the sections (the T-722 lesson, applied in advance). The 24 dark provider pages gain a row in "Owner decisions" — previously that gate lived only in the inline // TODO: remove to re-enable markers and frontend/scripts/validate-provider-pages.js, outside the tracker, and the row records that re-enabling a page is two edits (markers deleted AND the path moved GUARDED_FILES → LIVE_FILES) because the validator asserts both directions.

  • insertModuleIntoMarkdown honours its position parameter (#314). The last genuine inline code TODO: both branches appended to the end, so a caller passing a real index got a silent no-op. It now string-splices the serialized module directly ahead of the position-th <module> tag, leaving every other byte of the document — trailing prose included — where it was; -1 or an index past the last module appends, byte-identical to the old behaviour. Deliberately NOT parse → splice → rebuild: an insert makes the module list outnumber the placeholders, and rebuildMarkdownWithModules appends the surplus at the document end, so a middle insert would move the last existing module past any trailing prose — the first draft did exactly that, and review caught it. Four new test cases pin the contract as exact document bytes.

  • The pre-rendered DOM is hydrated instead of discarded (T-714, #296). main.jsx used createRoot, so 120 pre-rendered documents were built, shipped and thrown away at boot. It now calls hydrateRoot when the mount point's data-prerendered-route stamp matches the live path, seeded from data-prerendered-seed on that same element, and client-renders exactly as before when it does not.

The stamp is the load-bearing half, not defensive programming. staticwebapp.config.json's navigationFallback serves /index.html — the home page's markup — for any path without a file of its own, at HTTP 200. Every /admin route arrives that way. Hydrating on "the mount point has children" would have mismatched on the busiest pages in the app.

Five Playwright tests drive a real browser, including a node-identity probe that proves the server DOM is reused rather than coincidentally identical, and both guards are mutation-tested. onRecoverableError reports a mismatch, which was silent in a production build — the failure mode that would have let this regress to wasting every document with nobody noticing.

  • check-tfc-plan.mjs --commit <sha> (T-724, #298). The tool resolved the workspace's LATEST run, which is why tfc-plan-check.yml refused to run per-pull-request: the check would have been green, or red, about a run nobody asked about. --commit resolves the run HCP Terraform planned for a given commit, through the configuration version's ingress attributes, and the workflow takes a matching commit dispatch input.

"No run for this commit" returns 2, not 0 — it is not "the plan is boring". An absent relationship is skipped (CLI-driven runs carry no configuration version); a dangling one throws, so an ignored include= cannot masquerade as "no run for this commit". That distinction was found in review: the first draft documented throwing on unreadable shapes and then skipped exactly those cases.

  • scripts/workflow-write-permissions.test.mjs (T-726, #298). A ruleset bypass is granted to the Actions token, not to a workflow, so every workflow holding contents: write can push to main past all twelve required contexts. This pins that set to a reviewed two, each with a written justification. It bounds the exposure; it does not close it. Mutation-tested: granting the permission to iac-validate.yml fails the guard.

  • Deployment drift is measured now, instead of being discovered by accident (2026-09-01). deploy-functions.yml and deploy-azure-frontend.yml are both workflow_dispatch only, by a recorded decision — enabling a workflow and enabling auto-deploy-on-merge are separate choices, and only the first was made. So merging deploys nothing, and nothing said so.

That one gap produced both of 2026-08-31's incidents, hours apart, each found by accident rather than by a check:

  • The manifest route merged at 2026-08-30 02:45 UTC against a Function App last deployed at 01:21. publish-content-manifest.yml then failed with a 404 for two nights, and its own error message blamed the app — which was healthy, reporting 121 registered functions throughout.
  • The frontend was 35 commits behind, including a hydration change and a sanitizer hardening, found only because someone deployed for another reason.

AGE, NOT COMMIT COUNT, and that is the whole design. The first incident was ONE commit behind; the second was THIRTY-FIVE. No count threshold separates them — catch the one and you fire on every ordinary merge, tolerate ordinary merges and you miss the outage. What they share is that both sat undeployed for days. Age tolerates the normal merge-then-deploy gap, which is the entire point of dispatch-only releases, and still catches a change merged and forgotten.

monitor-deploy-drift.yml runs every four hours and fails when a service has been behind for 24 hours or more, emailing the owner through GitHub's own notifications — the mechanism monitor-functions-registered.yml already uses, needing no action group and no dependency on the subscription being watched. It reads only the GitHub API: the last successful run of each deploy workflow, and the commits on main touching that service's paths since. No Azure, no OIDC, no environment:, and contents: read plus actions: read.

Drift is measured per service over the paths that service ships, so thirty commits touching only wiki/ leave the Function App exactly as correct as it was.

On running this from a schedule GitHub delivers 22% of the time, said plainly rather than assumed away: for an outage detector that is a real problem, and here it is not. This watches a condition measured in days against a threshold measured in hours, so a check landing every 4.6 hours on average has ample margin against 24. The workflow header records that the reasoning stops holding if the threshold ever drops near the delivery gap.

36 tests, and six load-bearing decisions are mutation-verified: reading only the first page of commits, dropping the de-duplication, defaulting an unreadable commits payload to an empty page, taking the newest commit instead of the oldest, > instead of >= on the threshold, and leaving a raw | in a table cell. All six fail the suite when introduced.

All five fail in the same direction — reporting a service as healthier than it is — and that direction is the actual finding. Every defect review caught in this file, and every one caught while writing it, made a stale service look current. None made a current service look stale. A monitor whose bugs all point at "everything is fine" fails the one way it must not, so the mutation set is exactly those.

The sixth is a legibility failure rather than a correctness one, and belongs with them anyway: commit subjects and error text are not this script's to constrain, and a | in either ends its Markdown cell early — the row grows a column, everything after it shifts, and the table stops rendering as a table. That table IS the report, read to decide whether to deploy, at the hour a monitor tends to fire. Newlines are folded for the same reason.

Two defects were caught before merge and are worth recording:

  • oldestCommit originally took the last element, on the true-but-narrow grounds that GitHub returns commits newest first. That holds for one page of one path and nothing else — the merged list is newest-first only within each per-path segment, so with two paths the last element is the second path's oldest, which can be far newer. Found in review. It now takes the minimum by date and assumes no ordering at all, which removes the class rather than the instance. The same review found the missing pagination and de-duplication.

  • The setup-node pin was a SHA appearing in no other workflow here — invented rather than copied, which is the exact defect pinning by SHA exists to prevent. It is now the pin the other eight workflows use, and both action SHAs were verified against .github/workflows before commit.

  • The three repository-resolvable test-coverage follow-ups. API base resolution — the original line asked for api.js with VITE_BACKEND_PROVIDER=azure, a switch that no longer exists because the GCP backend is gone and the Azure base is the only one. functionsBase.test.js now pins what replaced it: getEndpoint composes an authenticated route onto the configured base in both topologies and throws naming the route when it is unset, an anonymous publicApi read goes to the same base, and a source scan fails if VITE_BACKEND_PROVIDER or any VITE_GCP_* variable reappears — a reintroduced switch would mean a second resolution path, which is the defect that file exists to prevent. Public content limits — limit and offset come straight off an anonymous query string, so public-reads.test.js now covers non-numeric, empty, zero, negative, fractional, oversized, Infinity and at-the-ceiling values on both listContent and listPodcasts, including that a negative limit clamps up to one item rather than producing an empty slice, that an offset past the end is an empty page with an honest total, and that limit=0 reads as unset. Partial configuration updates — admin-integrations.test.js now pins that a PATCH omitting oauthToken never sends the key (so the merge cannot clear it), that the untouched token stays out of the response built from the merged document, that the read artefact is dropped, that a revoke remains an explicit empty-string write, and that an ai-providers patch touches only the fields it names. ai_providers documents hold apiKeyEnvVar, the name of a server-side setting, never a key — oauthToken on mcp_servers is the only secret value either collection stores.

  • The visitor-facing upstream delta (T-409). From Site-Main 088f458, with their tests: RichTextBody (architecture and framework overviews render markdown as markdown, HTML as sanitised HTML), CoderCornerSnippet

  • CodeBlock (the snippet, language and repository link the coder_corner contract requires now render; fenced code gets highlighting and a copy button), WafAssessment + the vendor Well-Architected pillar sets (a Well-Architected tab when an architecture carries waf), FeaturedArchitecture
  • colorClasses (the AWS/Azure galleries' featured panel is data-driven), and the Ansible and VMware education data, rendered through a new EducationTracks component with level filter, learning paths and resources.
  • The eleven Firestore triggers as six change-feed functions (T-324). functions/src/functions/change-feed.js registers one app.cosmosDB function per watched container on the identity-based binding (COSMOS_CONNECTION__accountEndpoint + __credential = managedidentity, never a connection string), each with its own leases prefix. The before-image substitutes are ported from Site-Main lib/triggers/: value markers (image mirrors, Publer push), the rising-edge claim on an etag-conditioned replace (AI cover, slug page), the activation stamp (Telegram alerts) and content_stats_markers (dashboard counters, idempotent). Image mirroring keeps the {docId}/images/… blob scheme and serves through the media route; the template cover is stored as SVG; the AI cover calls Replicate over REST. The three deletes the feed cannot see: DELETE /api/cms/content/{id} and deleteContentItem move the counters, DELETE /api/cms/social-posts/{id} un-publishes on Publer first, and DELETE /api/cms/blogs/{id} is new (publisher, audited). lib/notify.js is the Telegram notifier with its per-source cooldown; resolve/reopen clear activationNotifiedAt.
  • The external-ingestion timers (T-323, closed). syncSocialCalendarScheduled reconciles social_posts with Publer (matched posts take Publer's state, vanished ones are marked deleted, unmatched Publer posts become publer_<id>); fetchBlogListings scrapes eleven non-RSS listing pages through Firecrawl's v1 REST structured extraction into content drafts in the RSS shape; fetchPodcastFeeds upserts PodBean episodes into podcasts. Each skips itself while its key is a Key Vault stub. Three more FEATURE_FLAG_* settings, all "false"; SYNC_SOCIAL_CALENDAR stays off until the cutover delta import (D12). Fifteen of sixteen timers are now registered; refreshToolServiceCacheScheduled stays demoted with Cloud Tools.
  • Twelve of the sixteen timers (T-323). functions/src/lib/timers/ carries generateReviewerDigest, cleanupRejectedContent (soft), cleanupSoftDeletedContent (hard, with linked blogs and content_versions rows), monitorPublishingPipeline, checkLiveLinks, reVerifyCertifications (republishes the certifications snapshot), cleanupUnusedCertImages, scrapeSkillsHubRss, refreshPlaudToken, forgeScheduled, and the two stubs — cleanupTempStorage (prefix + age, not an orphan sweep: T-302) and checkAgentHealth (T-401) — each a factory with injected store/fetch/storage, registered in schedulers.js through one flag-gated timer() helper with the §4.2 NCRONTAB. The two blob-deleting timers are dry-run until TEMP_STORAGE_CLEANUP_DELETE / CERT_IMAGE_CLEANUP_DELETE. Digest, alert and system-audit records go through lib/timers/workflow-records.js. Ten new FEATURE_FLAG_* app settings, all "false".
  • forge-article and generate-weekly-digest platform jobs; the stale-job sweeper. ContentForge's pipeline is ported whole (functions/src/lib/content/forge*.js, drafting.js): dedupe against the published corpus, admin-editable profile and prompts from admin_config with code defaults, format rotation, the forge module instruction and word soup, deterministic dash scrub / banned-phrase scan / module repair, the best-fit weighted grader with its keyword prescreen, forge_ready vs editing routing, version + audit + forge_stats writes. The weekly digest drafts from the last N days of live content into newsletters (dryRun previews); the Mailing List page gained the preview and draft buttons. platformJobSweeper (every 15 min, FEATURE_FLAG_PLATFORM_JOB_SWEEPER) re-enqueues jobs left queued by a failed output binding — the gap lib/jobs.js documented. generate-listen-and-learn is deferred to T-411 (three Google services, no frontend here), closing T-322.
  • batch-inspect platform job — the article inspector, ported. functions/src/lib/content/ carries Site-Main's scrapeArticle (fetch + cheerio + turndown; strict TLS; reader and headless fallbacks only behind CONTENTFORGE_SCRAPE_FALLBACK_ENABLED / CONTENTFORGE_HEADLESS_FALLBACK_*), extractPublishedDate, the voice / format-rotation block (pickNextFormat off scrapedAt in Cosmos, fails open), the verbatim analysis system prompt, the critique gate with one automatic revision, and buildInspectionUpdateData with its upstream tests. The job (inspect-jobs.js) selects up to 25 ingested documents — inspectTrigger: true first, then unflagged ones that have not failed — inspects each 4 s apart, records inspectError on failure and keeps going; results are counts and ids only. OpsHealthPage "Batch Inspect" now runs runJob('batch-inspect', { limit: 10 }); batchInspect left the RPC contract. Not ported: the architecture-diagram (multimodal) path — such documents record an inspectError saying so — and cover-on-inspect.
  • AI router (functions/src/lib/ai/router.js, T-322 §4.4) — ported from Site-Main's ai-model-router.js with the provider model the owner chose on 2026-08-21: a provider is on when its key is present. Anthropic, OpenAI and Gemini (public API by key; Vertex dropped — ADC is a GCP identity the app cannot hold), resolved in that order or pinned by CONTENTFORGE_AI_PROVIDER; an unresolved Key Vault reference counts as no key; no key → AI_NOT_CONFIGURED with a sentence naming the three secrets. fetch instead of axios; purpose → model table, JSON repair round trip, retry on 408/429/5xx, usage capture with cost estimates and the Anthropic prompt-cache marker all kept. 15 tests, none touching the network; the upstream cost tests came across.
  • fetch-rss-feeds — the first real platform job (T-322), ported from Site-Main's processRssFeeds: 20 feeds across 8 providers through rss-parser, one rss_cache document per feed with items[] capped at 20 on write (T-319's write-time cap), new content drafts through the existing four-stage dedup (≤ 10 per feed), and the homepage_feeds/latest round-robin aggregate. The admin "RSS Fetch" button enqueues it via runJob() instead of calling fetchRssFeedsManual (which never existed here); the syncRssFeeds timer stub now runs the same ingest every two hours behind its flag. TLS failures skip the feed with the reason recorded; one feed failing never abandons the sweep. 17 new tests. Not ported: the Telegram alert on feed errors — errors are in the job result.
  • Platform jobs — the pattern for every handler over Flex Consumption's 230 s HTTP cap (T-322 scaffold). functions/src/lib/jobs.js: a job-type registry, POST /api/enqueueJob (editor; type allowlist, per-type payload cap; 202 + jobId; message to Storage Queue platform-jobs through an output binding on the identity-based host connection), GET|POST /api/getJob (viewer), and a queue-triggered worker that claims with an etag-conditioned replace — at-least-once delivery never runs a job twice — and records succeeded / failed / timeout without rethrowing into the queue. New jobs container (30-day TTL, indexed like lab_jobs). Client: frontend/src/lib/jobs.js runJob() enqueues and polls with the Labs backoff. Built-in type noop. 14 new functions tests, 5 frontend tests; the route inventory now asserts the worker is the only queue trigger.
  • infra/scratch.tf — the migration rehearsal estate. cosmos-site-sbx-cus (serverless, keys off, the same firewall shape, the same hcw database and the same 72 containers from the same generated spec) and stsitesbxcus01 (the five content containers plus a private migration-reports) in their own resource group rg-db-site-sbx-cus, created only while cosmos_scratch_enabled / storage_scratch_enabled are true and destroyed when they are not. Mirrors production's posture on purpose: a key-authenticated rehearsal against an open account passes while proving nothing about the DefaultAzureCredential + RBAC path production takes. Outputs via one(); set-github-variables.ps1 wave 2 seeds COSMOS_SCRATCH_ENDPOINT, STORAGE_SCRATCH_ACCOUNT and SCRATCH_RESOURCE_GROUP from them, and leaves them alone while null.
  • scripts/migration-probe.mjs. One SELECT VALUE COUNT(1) that runs before the export and classifies a Cosmos 403 as firewall or rbac — two unrelated causes the SDK error does not distinguish, and which would otherwise surface only on the first upsert after a full export.
  • scripts/migrate-storage-to-blob.mjs + scripts/lib/storage-manifest.mjs. Manifest-driven GCS → Blob --inventory | --copy [--dry-run] [--overwrite] | --verify on @google-cloud/storage + @azure/storage-blob, idempotent by gcsmd5 metadata, carrying contentType / cacheControl, with a verify that compares counts, bytes, every object's MD5 and a deterministic byte-for-byte sample. --inventory exits 2 on an unmanifested prefix, mirroring the Firestore preflight. A vitest suite asserts every target container is one of the five Terraform names. Replaces migrate-storage-to-blob.sh.
  • Wiki pages Migration-Runbook and Phase-4-Data-Migration. Referenced from eleven places (README, the plan, _Sidebar, the workflow, the manifest header); neither existed. The runbook is the twelve-step operator sequence with the evidence each step produces; the Phase-4 page is the decision log.
  • WEBSITE_TIME_ZONE = "America/Chicago" on the Function App. Eight of Site-Main's sixteen schedules are declared in that zone; NCRONTAB on Linux evaluates in UTC unless told otherwise.
  • storage_resource_group output, pairing with storage_account the way web_resource_group pairs with functions_storage_account — what migrate-data.yml scopes its per-run firewall window to.
  • The HCP Terraform → Azure bootstrap, which existed nowhere. infra/providers.tf declares the azurerm provider with no credential — correct, because runs execute under HCP Terraform dynamic provider credentials — but the identity those credentials assume has to exist first, and nothing in this repository could create it. infra/oidc.tf creates the GitHub Actions identity, which only exists after a successful apply. Terraform cannot create the credential Terraform authenticates with. A repository-wide grep for ARM_CLIENT_ID, TFC_AZURE_* and app.terraform.io across .tf, .yml and .md returned nothing: the first apply had no documented path to authenticate, and the gap was invisible to file-by-file review because every individual file was correct and only the join between them was missing.

scripts/bootstrap-terraform-oidc.ps1 closes it. It creates rg-hcw-bootstrap, the id-hcw-terraform user-assigned managed identity, two federated credentials against https://app.terraform.io — one per run phase, because Entra matches token subjects exactly and case-sensitively with no wildcards, so a single credential leaves every apply failing at authentication while every plan succeeds — and Contributor plus Role Based Access Control Administrator at subscription scope (Contributor cannot create the role assignments infra/ declares; RBAC Administrator cannot grant Owner, so the identity cannot escalate itself).

A managed identity rather than an app registration, for the reason infra/oidc.tf already documents: app registrations need Application Administrator in Entra, which Azure Owner does not grant. The identity is deliberately outside Terraform state, in its own resource group — Terraform managing the credential it authenticates with means a destroy or a bad plan locks the workspace out of the subscription with no way back.

The script is idempotent and preflights before it proposes anything: CLI present, signed in, tenant matches, subscription visible, role-assignment rights held, Microsoft.ManagedIdentity registered. Sign-in is performed by the script rather than demanded of the operator — being signed in to a different directory is the normal state for anyone working across tenants, so it runs az login --tenant itself and re-reads the account afterwards, because a directory switch also changes which subscriptions are visible. -DeviceCode covers sessions with no browser of their own (SSH, containers, Cloud Shell) and the case where the browser keeps reusing the wrong cached account; the script falls back to it automatically when the interactive flow fails, since that failure is environmental — no display, no loopback — more often than it is a credential problem. It handles the fresh-tenant case explicitly — a Global Administrator holds no Azure RBAC by default, which produces errors that suggest the wrong fix, so -ElevateAccess takes the documented one-time root-scope elevation, grants Owner on the target subscription, and removes the root grant again.

Documented in Deployment Runbook §0 (which now tables the two OIDC handshakes side by side — confusing them strands the operator hunting for a CLIENT_ID that does not exist until after the first apply), CHECKLIST §8 (the four workspace environment variables, contractual and exempt from the 2-word rule), and REVIEW §4.0. The iac-repo-standardizer agent and the IaC Repository Standard both gained a bootstrap identity section making this the first thing audited on any repository, since the failure generalizes to every credential-free IaC repo.

  • Free-tier disposition recorded on the Cost-Analysis wiki page (now wiki-as-code, staged in .github/wiki/). Decisions from the workload owner's free-services meter review: runner image stays on Docker Hub (ACR rejected — month-13 cost for a failover-only image); Cosmos free tier is unusable by design (serverless); Service Bus / VM / SQL / LB 12-month meters rejected as expiring traps; blob + egress discounts are automatic. Adds the standing AI options reference for the future AI RPCs: always-free F0 SKUs per task (Translator, Language, Vision, Content Safety, Document Intelligence, Speech) with the mechanics that make them budget-safe (throttle-not-bill on quota, create directly with F0 — Foundry-provisioned resources default to S0, keyless applies) and the explicit note that generative drafting/image work has no free Azure tier — that is Azure OpenAI or the SaaS keys. (PR #116)

  • CodeQL actions language added to the advanced matrix — the retired Default setup had been scanning workflow files (language:actions); the advanced setup now owns that coverage across the repository's 12 workflows. Context: the Tool status page's erroring language:go / language:java-kotlin entries are stale Default-setup configurations auto-created ~3 weeks ago from stray Go/Java snippet files inside the vendored .claude/ harness — the exact paths the advanced config excludes. Those languages are deliberately NOT added to the matrix; the stale configurations are removed operator-side from the Tool status page's ⋯ menu. (PR #115)

  • Variable naming standard (workload owner directive, 2026-08-18) — operator-set configuration names are UPPER_SNAKE_CASE, maximum 2 words (3 only to break a real collision), with no provider prefixes: CLIENT_ID, TENANT_ID, SUBSCRIPTION_ID, RESOURCE_GROUP, APP_HOSTNAME. Contractual names (VITE_*, GITHUB_TOKEN) are exempt. Applied immediately to every workflow-consumed repository variable — all were still unset, so the renames are free: AZURE_CLIENT_ID→ CLIENT_ID, AZURE_TENANT_ID→TENANT_ID, AZURE_SUBSCRIPTION_ID→ SUBSCRIPTION_ID, AZURE_RESOURCE_GROUP→RESOURCE_GROUP, FUNCTION_APP_HOSTNAME→APP_HOSTNAME (FUNCTIONS_STORAGE_ACCOUNT keeps its third word to avoid colliding with the content account). The standard is codified in the iac-repo-standardizer agent — which now sweeps vars.*/secrets.* on every standardization run — and in the Wiki IaC-Repository-Standard page; CHECKLIST §7 carries the rule and an APP_HOSTNAME row. (PR #114)

  • Apply verification for the T-503–T-506 hardening (2026-08-18) — the operator applied the full set in HCP Terraform; cold start passed, verifying the T-503 VNet runtime/package-pull path directly. A post-apply validate-deployed run is byte-identical to the pre-apply baseline (no external regression), and Repository Policy / IaC Validation / CI / CodeQL are all green on main. One verification remains blocked: heal-computed-properties — the probe for T-504's 0.0.0.0 Azure-datacenter sentinel — fails at Azure login because the AZURE_CLIENT_ID / AZURE_TENANT_ID / AZURE_SUBSCRIPTION_ID repository variables were never set (a pre-existing gap, failing on every run before the hardening too; now recorded in CHECKLIST §7). Evidence table published as an addendum to the Wiki Resource-Validation-Report; plan v0.2 dispositions moved to APPLIED. (PR #112)

  • T-503 — Functions host storage network-restricted (apply pending in HCP Terraform; the last item of the T-50x hardening series). The host storage account moves to default-Deny with three deliberate survivors: the Flex app's runtime/package-pull path (VNet integration + new Microsoft.Storage service endpoint on the integration subnet — which also makes the content account's existing subnet rule provably non-inert), a per-run firewall window in deploy-functions.yml (add runner IP → deploy → always-run remove) under a new Storage Account Contributor grant scoped to exactly this account, and operator windows via functions_storage_admin_ip_rules. Rollback is one variable: functions_storage_network_default_action = "Allow". The #trivy:ignore:AVD-AZU-0012 suppression is deleted — the CI gate now enforces the control it previously excused. New required inputs AZURE_RESOURCE_GROUP and FUNCTIONS_STORAGE_ACCOUNT recorded in CHECKLIST §7. Verify after apply with a functions deploy and a cold-start invocation. (PR #111)

  • T-504/T-505/T-506 — the security and observability remediation ADR-0018 refused to ratify, now implemented in Terraform (apply pending in HCP Terraform). Cosmos hardening (T-504): VNet service firewall allowing the Functions integration subnet (new Microsoft.AzureCosmosDB service endpoint), the 0.0.0.0 Azure-datacenter sentinel so heal-computed-properties keeps working from GitHub-hosted runners (variable-gated to drop later), operator-window cosmos_admin_ip_rules, local_authentication_disabled (variable, default true), and continuous backup (free 7-day tier). Observability layer (T-505): infra/observability.tf adds the ag-hcw-ops-prod action group and diagnostic settings for Key Vault, Cosmos (the plan's four categories), the content blob service and Azure OpenAI; the budget gains the approved 50/75/90/100 ladder plus a Forecasted-at-100 alert routed through the group; Log Analytics gets the 0.25 GB/day cap. Keyless OpenAI (T-506): custom subdomain (planned replacement of the stateless account + both deployments — openai-client.js has zero importers, so nothing breaks), local_auth_enabled = false, Cognitive Services OpenAI User for the Function App identity, the primary-key output deleted, and an AZURE_OPENAI_ENDPOINT app setting for future keyless wiring. Plan v0.2-as-built dispositions updated to "resolved in code, closes on apply". (PR #108)

  • Infrastructure plan v0.2-as-built and ADRs 0018–0021 — implements the REVIEW §8.2 decision (workload owner, 2026-08-18) to supersede plan v0.1 with a plan that describes the real system. .azure/infrastructure-plan.json is now version 0.2-as-built: every implemented resource with its as-built properties, each deviation from v0.1 dispositioned as either a ratified decision or explicitly-unratified remediation debt (T-503–T-506, purge protection). Four ADRs staged to the Wiki: 0018 (umbrella supersede + disposition table), 0019 (single Function App — supersedes ADR-0004), 0020 (flat native Terraform root module — supersedes ADR-0005's AVM clause, resolves TODO T-502, and rewrites the README AVM guardrail to "pinned versions, stable addresses"), 0021 (Container Apps CI runner ratified as failover-only). ADR register updated; ADR-0004 marked superseded. New TODO T-506 (keyless Azure OpenAI: RBAC grant, delete the key output, disable local auth). (PR #107)

  • Resource validation pass, first execution (2026-08-18) — results published as the Wiki Resource-Validation-Report page (staged in .github/wiki/, linked from Home and the sidebar). External surface: edge live, TLS healthy to 2026-09-28, www/api-azure NXDOMAIN (consistent with same-origin), but Cloudflare bot challenge blocks all datacenter-IP validation of the origin. Plan-vs-code parity: ~40% of the approved plan's resources implemented, with material security-posture deviations (Cosmos open to the internet with key auth on, LRS vs ZRS, purge protection defaulted off, ungated keyed OpenAI) and material never-planned resources (CI runner, 71 containers, model deployments). Follow-ups filed: TODO T-504 (Cosmos hardening), T-505 (observability control layer); human decisions REVIEW §8.1 (Cloudflare synthetic-access rule) and §8.2 (reconcile implementation to plan, or supersede the plan as-built). (PR #106)

  • validate-deployed.yml — on-demand deployed-surface validation — a workflow_dispatch workflow running the externally observable half of the Deployment Runbook's §4 verification from a GitHub-hosted runner: DNS for the apex/www/api-azure names, TLS certificate inspection, frontend status + security headers, and scripts/smoke-deployed.mjs tier 1 (anonymous, no side effects) against a dispatch-time base URL (default https://hybridcloudworks.com/api). No secrets or cloud credentials — same doctrine as ci.yml; smoke tiers 2–3 remain operator-run. Results land in the job summary. The staged Deployment Runbook §4 references it. (PR #105)

  • IaC repository standardization — the repository now carries the baseline governance surface expected of a permanent infrastructure repo: .github/CONTRIBUTING.md, .github/SECURITY.md, .github/CODEOWNERS, a pull-request template with a Terraform-plan gate, issue templates (including an infrastructure change request with blast-radius and rollback prompts), a root .editorconfig, and infra/README.md documenting layout, working rules, guardrails and the ALZ-absorption posture. The repository policy script allowlists exactly these files; narrative documentation still belongs in the Wiki. A new iac-repo-standardizer agent (.claude/agents/) encodes the standard so future repositories can be brought to the same baseline.

  • IaC validation gate — .github/workflows/iac-validate.yml runs terraform fmt, terraform validate (via init -backend=false, so no credentials or state access), tflint (infra/.tflint.hcl) and a Trivy IaC misconfiguration scan on every pull request touching infra/**. Until now nothing validated Terraform changes at all while the prototype delivery workflow stayed disabled.
  • prevent_destroy guards on stateful resources — the Cosmos DB account, both storage accounts and the Key Vault now refuse plans that would replace them; removing a guard is itself a reviewed change. Applied together with the terraform fmt drift that had accumulated in main.tf.
  • Deployment Runbook and IaC Repository Standard as wiki-as-code — the day-1 apply procedure, day-2 operations, ALZ-absorption sequence, and the standard this repository now conforms to, staged under .github/wiki/ (with updated Home and _Sidebar) and published to the GitHub Wiki by the new sync-wiki.yml workflow on merge to main. The workflow overlays staged pages only — unstaged wiki pages remain UI-editable — and uses the built-in GITHUB_TOKEN, so no PAT or additional GitHub App is required. Staged pages become repository-owned: they get PR review like the code they describe. The sidebar's repository links now point at the HybridCloudWorks org instead of the pre-move personal fork.

Changed

  • Azure certification detail pages download only their own study-guide outline (#500). frontend/src/data/azure/study-guides.js (491 KB source) is now one module per guide plus index.js under frontend/src/data/azure/study-guides/. CertDetailPage loads the exam's module through import.meta.glob and use() inside a memoised Suspense boundary, so hydration keeps the prerendered outline until the module lands. The Azure detail chunk drops from 74.33 kB to 6.36 kB gzipped (AWS: 2.75 kB), plus about 1.8 kB for the one guide. Prerendered HTML is unchanged: AZ-104 still carries its outline and all five deep links, and all 50 guides render. update-study-guides.mjs writes the directory through scripts/study-guide-files.mjs. Guide modules carry no date, so an unchanged guide rewrites byte-identical. The weekly update-learn-catalogue.yml stages and replaces the directory, and detects new guides with git status --porcelain, since a new file is untracked.

  • The admin sidebar stays put, reads cleanly, and calls the newsletter page Newsletter Hub (#566). The admin shell is now exactly one viewport tall: the sidebar is pinned, its menu scrolls on its own when it outgrows the screen, and only the content column (#admin-main) scrolls, so the ContentForge brand never leaves view on a long page such as New Content. Pinning with sticky alone was measured and does not work on this site: html { overflow-y: scroll } with body { overflow-x: hidden } makes the body a scroll container that never scrolls, so a sticky aside pins to it rather than to the window. ScrollToTop now also resets #admin-main, so a page opened from the menu starts at the top. The brand block trades its fixed h-14 and leading-none for a minimum height with padding, which is what clipped "ContentForge" at 125% and 150% zoom. The nested <main> is gone: #admin-main is a div inside App's <main id="main-content">, so an admin page has one main landmark. The product tags beside menu items (New, Podcast, Publer, Linkie, Resend, VPS) are removed; the Review Queue and Editor count badges stay. "Mailing List" becomes "Newsletter Hub" in the menu, the page header, the Telegram message when the Monday build fails, the builder's "review and approve" message, and the Resend error fallbacks on the page and the API. The route /admin/mailing-list and the cms/mailing-list/* API paths are unchanged, so bookmarks and links keep working.

  • The SPA and the API are two app registrations again, and localhost is off the production one (#522, #521). This ADR-0006 decision — "one Entra SPA registration, one API registration" — was accepted, written into verify-token.js DECISION 3 and infra/variables.tf, and then not what the tenant did: the cutover put an SPA platform on the API's own registration. A codebase describing a tenant it does not have is worse than either choice made deliberately.

The divergence was reasoned, not accidental. One registration means the SPA requests a scope on its own app, which consents automatically and removes the risk of a client id and an audience that disagree — and this project has paid for that class of failure twice. That risk is now caught before it ships: assertDeployConfig refuses a deploy build whose Entra ids are not GUIDs (#516), which is why this landed after it and not before. The rationale is kept struck through in 02-entra-spa-client.ps1 rather than deleted, because the argument was sound and somebody will otherwise re-derive it.

Microsoft's stated reason for separating them is permission inheritance: "if the web API has a higher set of permissions, then the client app doesn't inherit them." One registration is one service principal, so any credential or Graph permission ever added for the API is simultaneously available to a browser-delivered public client.

Exactly one value changes: VITE_ENTRA_CLIENT_ID. ENTRA_API_AUDIENCE and VITE_ENTRA_API_SCOPE do not — the split changes the client, not the resource. App-role assignments do not need redoing either: they live on the API's service principal, and the roles claim rides in the access token whichever client asked. Both facts are now stated in the places most likely to be edited by someone assuming otherwise.

01-entra-spa.ps1 is replaced by 01-entra-api.ps1 (the resource, its preconditions, and the Admin assignment), 02-entra-spa-client.ps1 (the client) and 03-entra-dev-client.ps1. The client scripts take -Prune, which replaces the redirect URI list rather than unioning it, and print removals as loudly as additions — the unioning predecessor could do neither, which is how http://localhost:5173 survived unnoticed on a production registration.

Localhost now lives on a dev-only registration. cors.js already argued this case for origins — "any page running on a victim's machine … can make cross-origin calls to production carrying the victim's token" — and it applies with more force to a redirect URI, where a process on 127.0.0.1 can complete an authorization-code redirect for the production client id. CORS gates its localhost entry on NODE_ENV; an app registration has no such gate, so the only control is not listing it. The fix had been applied in one place and left open in the other.

The Static Web App origin stays, and its comment now says why instead of claiming to be temporary. It said "REMOVE IT when DNS moves"; DNS moved weeks ago and it is still there, because deploy-azure-frontend.yml documents the production break-glass as re-pointing the custom domain — which lands on exactly that hostname. Removing it would take admin sign-in out of the only documented recovery path, discovered during an incident. The revisit trigger is now a real one: when the Static Web App is deleted.

A vitest file reads the PowerShell as text and asserts the production scripts list no localhost and nothing plain-http. PowerShell is otherwise untested here, and these are the files that rewrite the tenant.

  • MFA is enforced by security defaults, not Conditional Access (#514). Checked against the live tenant on 2026-09-12: security defaults are enabled in tenant properties, and Conditional Access is unavailable because the tenant is not licensed for Entra ID P1. AdminAuthGuard.jsx, entraAuth.js and ADR 0006 all stated Conditional Access, which would have sent anyone looking for a policy to an empty blade.

Two consequences are now written down rather than left to be rediscovered: security defaults cannot be scoped or excepted, so a break-glass path has to survive MFA rather than bypass it; and Entra disables security defaults automatically the moment any Conditional Access policy is created — so licensing P1 later is not a free upgrade, it is a change that silently replaces the MFA control.

lib/auth/roles.js has said since it was written that Continuous Access Evaluation "does not cover us", and now carries the citation for it: CAE requires that both the client and the resource be CAE-enabled, and CAE-enabled resources are Microsoft first-party services. Declaring CP1 would take on the obligation to handle claims challenges while no resource in the call graph can issue one.

  • A deploy build with the Entra variables unset shipped a sign-in pointed at every tenant (#516). msalConfig.js defaulted the authority to common when VITE_ENTRA_TENANT_ID was empty, and vite.config.js validated exactly one variable — VITE_AZURE_FUNCTIONS_URL — so nothing stopped the build. deploy-azure-frontend.yml even carried a comment acknowledging it.

What that actually produced is worth stating, because it is not what it sounds like. common was never a route to backend access: the API pins one tenant by issuer, so a token from another directory could never have been authorized. What it produced was a SPA that accepted a sign-in from any Entra tenant or personal Microsoft account, stored that identity in localStorage, rendered signed-in UI, and then 401'd on every call — a confusing partial success where an unambiguous failure belonged.

The backend has had this instinct since it was written: verify-token.js refuses to start without its two settings, and infra/variables.tf rejects an empty audience because an empty one silently disables audience validation. The client half never got it, and now has it.

Shape, not presence. A presence check would have caught neither realistic mistake, because .env.example recommended common for the tenant and /.default for the scope — both non-empty, both wrong. A GUID test rejects empty, common, organizations and consumers in one rule, and the error lists every problem at once so three missing variables do not cost three deploy attempts. /.default is deliberately still allowed: it is a real delegated request shape, just not this registration's, and the lever for that is documentation rather than the build.

REQUIRE_API_BASE keeps its name even though it now gates more than the API base. Renaming it is fail-open — the deploy workflow is what sets it, so a rename landing without the matching workflow edit would silently disable every check with nothing going red.

msalConfig.js now falls back to an all-zero GUID rather than throwing. A module-scope throw would take out the several test files that import it transitively without mocking, and would convert a build-time problem into a runtime crash on a path that already has a working failure mode: MSAL fails at authority resolution, onAuthStateChanged reports null, and the sign-in card renders.

Also corrected: .env.example no longer suggests common or /.default, and functions/local.settings.json.example gained ENTRA_TENANT_ID and ENTRA_API_AUDIENCE and lost FIREBASE_PROJECT_ID — without the two Entra settings a local func start could not reach getDefaultGuard() at all, because createTokenVerifier throws on both.

  • The API accepted an ID token as an access token (#515). functions/src/lib/auth/verify-token.js has always opened with DECISION 3, which names the hazard precisely: "With a single registration, an ID token minted for the SPA carries aud = <client-id> — indistinguishable from an access token for the API." The tenant runs one registration — scripts/cutover/01-entra-spa.ps1 put an SPA platform on the API's own registration — so the hazard was live rather than hypothetical. An ID token carried the same aud, the same v2 issuer and the same signing key as an access token, and because Entra emits assigned app roles in both token types it carried roles: ['Admin'] too. Every check the guard made, it passed.

Defence in depth, not an escalation. Both tokens sit in the same browser cache for the same user, so holding one implies holding the other and there is no cross-user path. But it was this codebase's own documented invariant, violated in production, and the second half of DECISION 3 — "reject anything without a scp/roles claim" — had never been implemented. scp appeared exactly once in functions/src, in that sentence.

That proposed remedy would not have worked either. roles is in ID tokens, so a roles-presence check separates nothing. scp is the claim Entra puts in delegated access tokens and never in ID tokens, and it is now required in authenticate() — so requireUser endpoints inherit it, which matters because bootstrapCurrentUserAdmin is reached that way. Agents are unaffected: require-agent.js runs its own client-credentials path, and now refuses a token bearing scp so the two guards are disjoint by construction rather than by the two role names happening to differ.

Alongside it, the verifier now asserts what Microsoft's API-protection guidance asks for and jsonwebtoken does not check: tid is a GUID and matches the configured tenant, iss agrees with that tid, and ver is 2.0. The v1 sts.windows.net issuer is gone — with a bare-GUID audience it could never pass, but infra/variables.tf documents the api:// audience as a supported alternative, and the day someone took that option the v1 issuer would have gone live with no version gate behind it. The assertions run inside the jwt.verify callback, so every claim they read has already had its signature checked and tid reaching an audit row is not attacker-controlled.

Before deploying, confirm scp is in the live token. Admin → Health lists the claims present in the caller's own token; scp must be among them. If it is not, every admin call returns 401 with a missing-scope row in admin_audit_logs — a clean, attributable failure, and one revert away.

  • An expired token told the owner he was not authorized, and offered to re-provision him (#503). Opening /admin/integrations on 2026-09-11 showed Access Denied — spatino@hybridcloudworks.com is not authorized with a Bootstrap My Admin Access button. Signing out and back in "fixed" it, which is the tell: the problem was never authorization. The admins/{oid} record was correct the whole time.

Three outcomes were returning one value. fetchAdminStatusFromBackend answered { isAdmin: false } for a rejected token, for a call that never completed, and for a server that genuinely said no. Only the third is an authorization answer; the first two are a check that could not run, reported as a check that ran and refused. The warning string it logged — 'Not an admin or token expired' — admitted on the line above that it could not tell which. That is the same defect class as unknown versus disconnected on the Recording Hub and the 401-versus-403 saga that cost #358 two days.

The hook now returns a discriminated accessState — authorized, unauthorized, unknown — and unknown is never cached, so a transient failure does not sit in front of the portal for five minutes. authedFetch keeps the HTTP status on the error it throws and acquireApiToken tags every failure with the one action that recovers it, which is what makes the three distinguishable at all: requireUser on the server denies only with 401, so a non-200 from this route is never a verdict about the caller.

What the owner sees now. An expired session redirects to Entra and comes back signed in, without a denial and without anything to click — once per tab, because a second automatic attempt against a genuinely broken configuration is an infinite redirect loop with no screen to read. After that it is an honest "Could not verify your access" card with a Sign in again button; an unreachable API gets the same card with Try again. Access Denied is now reached only by a real 200 saying so.

And the bootstrap button is no longer guessed at. getCurrentAdminStatus returns canBootstrap, the server running the same three-way gate the POST enforces, so the button appears only when pressing it would succeed — never for an account that already holds a registry row, active or deactivated. Offering a deactivated admin a self-promotion to super_admin is the opposite of what the deactivation meant.

  • The Monday refresh put the six GitHub exams straight back into the Azure catalogue, hours after #507 removed them (#496). Found by dispatching the workflow #499 had just widened — run 34627119281 on 2026-09-11 opened a pull request re-adding GH-100, GH-200, GH-300, GH-500, GH-600 and GH-900, with their six study-guide outlines behind them, +1167 lines of them.

Nothing was broken; the decision was simply in a place no script reads. GitHub certification moved onto Microsoft Learn, so all six sit on the Microsoft Certifications poster and in the credentials browse API — both of which are this updater's sources. It had always been offering them, and before #496 the catalogue accepted them. #496 recorded the removal in the catalogue's file header and in a cross-catalogue test, and update-applied-skills.mjs reads neither. So the rule now lives where the automation meets it: the script refuses any exam another catalogue owns, before building the row.

A prefix, not the six codes. GitHub adding a GH-700 would recreate the duplication exactly, and it belongs to /github/education for the same reason the six do — the Azure LEVEL_META has no Professional rung, so a GitHub exam cannot be stated truthfully there whatever is typed. Refusals are reported in the pull request body under their own heading rather than dropped silently, and deliberately not in the "not added" bucket, which means "nobody could confirm this exam" and would send a reviewer hunting a source problem that does not exist.

The workflow also checked the wrong thing. Its test step ran azure/certifications.test.js alone, which by construction cannot see a contradiction with another catalogue — so the run went green and left the defect for the pull request's CI to catch. It now runs education-catalogues.test.js too, where the cross-catalogue guard lives, so the workflow refuses to open that pull request itself.

Worth recording that the guard worked: one exam, one catalogue (#496) is what failed on the bad pull request, on the same day it was written. The regression was caught by CI, not by a reader — but by the pull request's CI rather than the workflow's, which is the gap closed here.

  • The three dated catalogue rows were settled without waiting three weeks for them (#494). #494 held four obligations that all read as "come back after 2026-09-30". Three of them did not need the wait, and answering them now turned up two things the issue had wrong.

AZ-800 was missing from the issue's own table. It carries the same expiryDate: '2026-09-30' and the same replacedBy: 'az-802' as AZ-801, so it trips in the same breath — four rows go stale on 2026-10-01 across three catalogues, not the one the issue expected a person to fix. A test now pins the exact list, so the next dated row is a named expectation rather than a surprise.

The pages were never going to lie, which changes what the alarm is for. deriveStatus reads the dates at render time, so on 2026-10-01 AZ-800 and AZ-801 render retired and PAA renders active with nobody touching a file. No visitor is shown a retired exam as testable. What goes wrong on that day is narrower and worth stating precisely: a stored status its own dates have overtaken is a claim nobody has rechecked against the vendor. Pinned at a fixed future date so it is known rather than discovered.

Whether the Monday run self-heals AZ-801 was answerable today, and the answer is "in one of three cases". #494 recorded it as "an expectation, not a guarantee — nobody has watched that workflow correct a row yet", but reconcileLifecycle never reads the calendar: it compares the file's status against the source's by LIFECYCLE_RANK, so what the 2026-10-05 run will do is decided entirely by what Microsoft reports, and every branch was exercisable immediately. It retires the row when Microsoft reports retired on the same date. It does not when Microsoft reports a different retirement date — the date conflict is caught before rank and the file wins, leaving the row stale with one report line as the only clue. It does not when Microsoft is slow and still reports expiring, and that case reports nothing at all. All three are rule 3 working as designed, and none of them was covered: no test anywhere fed the reconciler a source saying retired, so the forward move the issue was waiting on had never been exercised.

AZ-802 closes the Azure half outright (owner, 2026-09-11): it becomes the single test replacing both AZ-800 and AZ-801. That is what makes those two rows need no vendor round-trip on 2026-10-01 — the successor is already known and already in the file, which certifications.test.js has asserted since #464. The gap was that nothing said the successor is itself undated: AZ-802 is active and carries no dated field at all, so findStaleStatuses can never name it and it is not a third row to revisit later. Now pinned, and it fails the day Microsoft puts a date on AZ-802 — which is the right moment to look, rather than meeting a dated successor as a red build months later with no record of why anyone expected otherwise.

MLA-C02 confirmed a false positive, as #494 argued: dateProblems reads only betaEndDate and gaDate for a beta row, never betaStartDate. Pinned both ways — silent on 2026-10-01, and firing on 2027-01-15 once its gaDate is reached — so the reasoning is enforced rather than restated in prose the next time somebody sorts by date and panics.

PAA was re-read against Google on 2026-09-11 and nothing had moved: still "open until September 30", three hours, ~80 questions, $120 against $200 retail, English, one-year validity, still no GA date. DATA_AS_OF is deliberately not bumped — one row was re-read, not all fifteen credentials, and that field is a claim about the whole file.

The one thing no day before 2026-10-01 can supply is what the credential becomes, because the post-beta state does not exist yet. That obligation is carried by the alarm rather than by the board: on 2026-10-01 the suite fails naming PAA and pointing at DATA_SOURCE, which is the instruction the ticket held, delivered on the day it becomes actionable.

  • Every certification catalogue re-verified against its vendor, ahead of the 2026-09-29 deadline (#469). Seven catalogues were checked row by row against the official source and DATA_AS_OF moved to 2026-09-10. Azure is excluded because update-learn-catalogue.yml refreshes it on a Monday cron, which was confirmed sound rather than assumed.

One real error, nine months old. The AWS row called itself SCS-C02 while already linking SCS-C03's exam guide, and nothing had ever compared the two. SCS-C02 stopped being the current exam on 2025-12-01. The row is now SCS-C03 with previousSlugs: ['scs-c02'] so minted links still resolve, following the SOA-C02 precedent, and the three places that hard-coded the old code outside the catalogue were updated with it — aws/LandingPage.jsx and two entries in MicrocredentialDetailPage.jsx, which would otherwise have displayed a superseded exam beside a corrected one.

Three more corrections, each from the vendor's own page. VMware's VCTA is gone from Broadcom's index entirely and is now retired — with no retiredDate and no replacement, because Broadcom publishes neither and inventing them is the failure this repository keeps legislating against. Red Hat renamed RHCE to "Red Hat Certified Engineer in Ansible", and the ex374- URL the file carried now 404s. The FinOps Professional exam requires three prerequisite certifications rather than two, confirmed in two places on the Foundation's own site.

GCP, GitHub and HashiCorp needed no data changes at all, and two near-misses are worth more than the diff would suggest. A cached copy of GH-100's page still says "(beta)"; the live page does not, so trusting the cache would have flipped a good row to beta. And GH-500 is absent from learn.github.com/credentials while its own page is live and its study guide took a revision in July 2026 — an omission from a marketing page is not a retirement. Both are now recorded in the file headers as sources NOT to trust, which is the more durable half of the work.

What was deliberately not done. Broadcom lists thirteen VCF certifications against the six this catalogue carries. The titles and exam codes are recorded in the file header, but no rows were added: hours and prepTime are this site's own study estimates, and fabricating fourteen of them is exactly the confidently-wrong entry the rule exists to prevent. AWS's successRate percentages have the same problem and are flagged for a decision — every other field in these catalogues is sourced; that one never was.

A clock trap worth recording. todayIso() builds its string from getFullYear/getMonth/getDate, so the freshness check runs on the LOCAL calendar. On a machine in CDT at 23:11, DATA_AS_OF = '2026-09-11' — the correct UTC date — fails six tests as "in the future", while a UTC CI runner would accept it. The date used is the one the repository's own clock agrees with.

  • Every timer now runs on UTC; nine live jobs moved five hours earlier (#416). Owner decision: all times in this app are UTC. WEBSITE_TIME_ZONE = America/Chicago was set app-wide on the Function App, so a bare NCRONTAB hour meant Central — and six places in infra/observability.tf, infra/variables.tf, infra/functionapp.tf and ADR 0028 said the Cosmos exporter ran at "03:00 UTC" when it ran at 08:00 UTC in summer and 09:00 in winter. The most expensive of the six is the alert-cosmos-export-daily description, which is the sentence that arrives in the alert mail: an operator reading it at 05:00 UTC would have gone looking for a run that was not due for another three hours. The issue offered correcting the prose or correcting the clock; the owner chose the clock, so those six sentences became true rather than being rewritten.

The setting was removed, not set to "UTC". UTC is already the platform default — "The default time zone used with the CRON expressions is Coordinated Universal Time (UTC)" — and Microsoft documents both WEBSITE_TIME_ZONE and TZ as unsupported on the plan this app runs on: "aren't currently supported when running on Linux in a Flex Consumption or Consumption plan … can create SSL-related issues and cause metrics to stop working for your app" (Learn, Timer trigger for Azure Functions, retrieved 2026-09-07). Setting it to "UTC" would have kept an unsupported setting on an unsupported plan in exchange for nothing. Nothing in functions/src formats a local time for a human — every date it derives comes from toISOString() or a getUTC* method — so the app clock reached NCRONTAB and the host's own log stamps and nothing else.

scrapeSkillsHubRss is the one schedule that changed, so that it would NOT move. Its upstream schedule was every friday 09:00 in UTC and it was ported as 0 0 4 * * 5 because 04:00 Central is 09:00 UTC. Left alone it would have retimed the one job whose intent was already UTC; it is now 0 0 9 * * 5, the same instant, and no longer an hour late every winter. The nine fixed-hour jobs — cleanupTempStorage, cosmosExportScheduler, forgeScheduled, cleanupRejectedContent, cleanupUnusedCertImages, generateReviewerDigest, checkLiveLinks, reVerifyCertifications — all moved five hours earlier in absolute time (six once CST starts), and the six hour-interval timers kept their cadence and shifted phase. The four minute-interval timers did not move at all. Migration-Plan §4.2 carries the per-timer table of what moved, appended rather than rewritten.

functions/src/functions/timer-schedules-utc.test.js is the new guard. It reads the real app.timer() registrations rather than the source text, fails if WEBSITE_TIME_ZONE or TZ reappears in infra/*.tf, computes which schedules depend on the app clock instead of trusting a list, and requires each of them to carry the UTC time it is meant to mean. scripts/cutover/05-verify-timer.ps1 printed "04:00:00+00:00 means WEBSITE_TIME_ZONE is NOT being applied", which is now exactly backwards; it prints the inverse, and says so.

  • A publisher can give a published article the right slug, from the row where they already look at its URL (#400). Publisher, not editor: this changes the URL a live article is served at, which is the same authority publishContent and the #374 re-host backfill require, and the reason updateContentItem — an editor route — no longer accepts a slug at all.

#403 made new collisions impossible and made a republish move an article off a slug another document holds. It could not give an article the RIGHT slug, and three articles need exactly that: they share one title, so they were published onto one URL and two of them are unreachable. Nothing in the admin UI edited a slug, so the only fix was a script that would leave nothing behind for the next time.

The generic route was checked first, and it is what made a dedicated one necessary. updateContentItem accepted a slug — slug was on no denylist and matched none of its normalizers, so it fell through to the generic string branch and was stored EXACTLY as sent: { slug: ' Hello World!! ' } round-tripped with its spaces, its capitals and its punctuation intact, and nothing asked whether another document already held the value. Both halves of that are how three articles ended up on one URL, so the route is fixed rather than worked around: slug and Slug join FORBIDDEN_CONTENT_UPDATE_KEYS, and a site URL now has exactly one writer.

That writer is POST /api/cms/content/slug (publisher), a second narrow reason on the publish pipeline beside the #374 re-host — the extension point REHOST_IMAGES_REASON's own comment anticipated. It slugifys the input so an operator cannot type a value the pipeline would never produce, probes c.slug OR c.Slug through the same never-throwing slugHolders wrapper an ordinary publish uses, and REFUSES a slug another document holds rather than suffixing it — the one place a set-slug has to be stricter than a publish, because silently giving an operator a different URL than the one they asked for is how a URL correction becomes the next URL surprise. An unanswered probe is likewise a refusal — not because it probes differently, but because evaluateSlugChange reads that wrapper's null as a refusal where resolveSlug reads it as "not established". Same probe, opposite reading, and for the same reason read the other way round: a publish must not lose an article's URL to a transient query error, but a NEW slug assigned on an unverified probe is how one URL gets two documents.

It is narrow because a full republish does not merely do more than these articles need — it fails. They arrived from Site-Main already published and have never been through this pipeline's gates: the one still serving the contested URL scores 68 against a threshold of 82 (238 words, no inline modules) and carries no hero image, so the quality gate refuses it and the image gate refuses it after that, and a set-slug built on the full path would write a failed quality report onto the article and leave its URL exactly as broken as it found it. So the branch writes the cased slug pair and curatedSubpagePath / slugPageUrl / publishedUrl / publicUrl — derived through resolveCuratedSubpagePath and toPublicUrl, so a corrected URL and a published URL can never be built by different rules — in ONE patch conditioned on the read's ETag, and nothing else: no gates, no cover or social trigger, no dates, no forge stats, no Live rewrite, no status change. One patch rather than two steps so there is no window in which an article holds a new slug at its old URL.

It refuses rather than half-completing, in two places. A slug another document holds is refused rather than suffixed — and so is an article whose published path cannot be computed at all, meaning it has neither a stored curatedSubpagePath nor a provider recognisable in any of the five fields resolvePublishContext reads. Writing the slug for such an article and no URLs would leave the panel reporting a move beside a link still pointing at the old URL, which is precisely the state this route was built to remove. Both are a 409 naming what would fix them. That diverges from the ordinary publish write, which spreads the path conditionally, and the divergence is the point: a publish is best-effort on a document being published, where a missing path is one incomplete field among many, while a set-slug is a surgical correction to a URL, and one that cannot compute the URL should stop and say so.

Both cased fields are written, to one value. The probe reads c.slug OR c.Slug, so a document holds every distinct value across the pair: writing only slug would leave the article still holding its old URL in Slug, blocking any other article that legitimately wants it, and would leave the pair divergent — the exact shape, on ten of the twenty-two published articles, that hid #400.

The control is on Publish, inside the row of the already-live article being fixed, beside the "View Live" link it changes. It offers the article's own Slug field as the suggested value, because for this class of article that field is the source publisher's slug and therefore the only surviving record of what the article actually is. Normalisation stays server-side — slugify lives in functions/ and there is no package shared with frontend/, so a live client-side preview would mean a second copy of the one function whose output IS the URL — and the response names requested beside slug, which the panel shows whenever they differ. Nothing can silently no-op, and the panel distinguishes four outcomes rather than collapsing them: a move (the URL is a different one now), a repair (the slug stayed, but Slug was brought into line with it or URL fields that had drifted were re-derived — a real write, and one of the cases this exists for), a truthful no-op, and a refusal that wrote nothing. The response carries moved and the list of fields the patch touched so the panel is told which happened rather than guessing, and a refusal names the document holding the URL. - The site has a show of its own, and every audio page leads with it (#349). admin_config/podcast_feeds could only describe feeds that belong to a provider — every row is { provider, url } — so the RSS.com show the owner created, Hybrid Cloud Insights, had nowhere to go: seeding it under one provider would have hidden it from the other seven pages and mislabelled it on the eighth. The document grows one field, mainFeedUrl, above the provider rows and visibly the primary one on Admin → Platform settings; the provider rows stay as optional extras. A document stored today normalizes unchanged and a Functions revision predating the field ignores it, so the merge and the deploy can happen in either order.

Inside a run the show is an ordinary feed entry under the reserved provider main, returned first by resolvePodcastFeeds, so the ingest loop, the dedupe, the 410 handling and the summary line work on it unchanged and the episodes it writes carry provider: 'main'. A reserved value rather than an absent one because every consumer of podcasts already keys on that field — the container's composite index, the public list's SQL filter, the section counts — and a row with no provider would be fetched by no query and counted by nothing.

GET /api/public/podcasts?provider= now filters c.provider IN (@provider, @mainProvider): one query, so a show episode arrives once, appears on every provider's page and is duplicated on none. The page orders the show first as a group, so the featured player on every provider page plays the show's newest episode; the source filter is built from the sources a page actually has rather than a fixed pair; and the RSS subscribe button falls back from the provider's feed to the show, since a reader subscribing from a page with no provider feed is looking at the show. Two writes are refused rather than resolved — main as a provider row, and a provider row repeating the main feed's URL, which would have both ingests build the same episode ids and flip an episode's provider every two hours.

The sitemap follows. countPodcastDocs counts a show episode for every provider's audio and audio-architecture, by name rather than through _unattributed — that bucket means "we do not know which pages this reaches", and here we do. So the sixteen audio URLs #404 dropped return the moment one show episode is ingested and the manifest is refreshed; a page with content that is not advertised is the mirror image of the bug #373 fixed. Recorded in ADR 0029 §1a, which the hosting decision it amends already owned.

  • Thirty-three empty section pages leave the sitemap, and the counts that decide it now come from every container a section page reads (#373). Part 2 could speak for one section: frameworks reads the content container and the manifest builder was handed that corpus. Every other section reads something the builder cannot see — blog, coder-corner and code fall back to the legacy blogs container when content has nothing for a provider, which is precisely the case being declared empty, and audio and audio-architecture read podcasts and listen_and_learn_episodes, which are not content at all. So the counting moved into the app, where the Cosmos access already is: GET /api/public/content-manifest returns a sections map beside items, computed over all four containers under the same public filter GET /api/public/content applies, and the builder prefers it — falling back to its own frameworks-only count when the deployed revision predates the field, so merging this and deploying Functions can happen in either order without the five frameworks pages returning to the sitemap in between. sitemapRoutes in the pre-render is unchanged: it never named a section, so it drops the new ones the moment the counts arrive. Measured against the live containers and the 2026-09-07 crawl, the sitemap goes from 113 URLs to 80, and all thirty-three that leave are pages the crawl calls empty; nothing the crawl calls works is dropped. Two rules are the reason a count is trusted: only a section whose page infers a provider from titles and URLs (blog, frameworks) lets an unattributable item block its zeros — the rest match a stored provider field in SQL, so such an item is fetched by no page and blocking sixteen audio pages on behalf of two provider-less podcast rows would be wrong — and a read that fills its window returns null rather than a truncated count, because the rows past the window are indistinguishable from rows that do not exist. architecture-designs is deliberately not counted: its pages merge staticBlueprints hardcoded in each ArchitecturePage.jsx, so four of the five render fine on a corpus holding one architecture document and an API zero would drop working URLs. Every dropped route still renders and still serves its empty state; this changes what is advertised, not what exists.
  • Every publish now checks whether another article already holds its URL, and the collisions already in the corpus have a report of their own (#400). One case is deliberately left open and is named below: two first-publishes racing on the same title can still both probe clean and take the same bare slug. Closing it would mean suffixing every new URL with a document id, which is not worth doing to a site's whole URL space for a collision that is rare, recoverable and now reported. Three published articles share enable-ai-powered-discovery-of-azure-updates-with-microsoft-release-communicatio, so two of them are published with no URL at all and the third's identity changes with the order Cosmos returns rows — visible in #399's diff, where the body under that key moved from 7ZCmiOEIlWMg99xQHY0A to 1k5ayjbEdYdo7NzvXIWW. None of the three was ever assigned that slug by this pipeline: processPublishContent writes slug and Slug to the same value in one patch, and on all three — and on ten of the twenty-two published articles — the two differ. They arrived from Site-Main already contentStatus: published, which takes the branch that reuses the stored slug, and that branch never probed. uniqueSlug becomes pure resolveSlug plus a slugHolders probe, keeping its shape — bare slug when nothing else holds it, suffixed with the document id when something does, and a lookup failure never blocking a publish — with both gaps closed: the probe now reads c.slug OR c.Slug, because the two fields carry different values on ten of the twenty-two published articles and the manifest routes on slug || Slug; and every publish probes, a republish included, moving off a slug another document holds instead of re-asserting it. A probe that throws still yields the always-unique suffixed slug on a first publish and now leaves a live URL untouched on a republish. What a read cannot establish — two publishers racing on one title — is documented at resolveSlug and caught by the new scripts/report-slug-collisions.mjs, which prints the contested URLs from the committed manifest with no credential. curatedSubpagePath moves with the slug now (resolveCuratedSubpagePath, found by Copilot's review): it was taken from the stored value unconditionally, so an article leaving a contested slug would have kept advertising the contested URL in slugPageUrl, publishedUrl and publicUrl while slug said otherwise, and it is normalised to a leading slash because three frontend hooks read the provider out of curatedSubpagePath.split('/')[1]. The slug is normalised once before the probe for the same reason the URL is: resolveSlug trimmed while slugHolders probed the raw value, so a stored ' shared-slug ' — migrated or hand-edited, which is this corpus — matched no holder and the trimmed form was then written onto a URL another article held.
  • The Social Hub lists Publer accounts again, and when it cannot it says which of three things is wrong (#397). Both of its call sites tested the publerProxy response with Array.isArray, and the proxy answers with an envelope — { ok, status, data }, or { ok: false, code: 'INTEGRATION_NOT_CONFIGURED' } when no key is seeded — which is never an array. So the check was always false and a fully connected workspace rendered exactly like an empty one, on the Compose tab and on Connection Settings both. The reader the Platform settings page grew for the same bug (#391) has moved to frontend/src/lib/publerAccounts.js with its unit tests, and both pages import it rather than keeping two copies to drift apart. The three outcomes are now distinct where they used to be one empty list: an unseeded integration says so and names the Connection Settings tab, a failed call says the call failed and carries the upstream message, and only a proxy that answered with no accounts still reads as an empty workspace. The reader names that third case explicitly, because the proxy answers HTTP 200 whatever happens: Publer refusing the key comes back as a resolved { ok: false, status: 401 }, which no .catch() will ever see, so both pages showed an authentication failure as an empty workspace until unwrapPublerAccounts grew a failed outcome carrying Publer's own status. The unconfigured copy no longer blames the API key either — one code covers a missing PUBLER_API_KEY and a missing PUBLER_WORKSPACE_ID, so the page repeats whichever the server named rather than guessing. Reaching Connection Settings had also been throwing a ReferenceError before it could paint — its two credential tiles called publerKey() and publerWsId(), which are defined nowhere in the bundle and cannot be, because FINDING-04 moved both values into Key Vault. One tile replaces them and reports the only thing the browser can observe: whether the proxy could use the credential.
  • The audit stops calling a cancelled request a failure, the Azure architecture page names its provider, and the 2026-09-07 crawl is on the record (#361, #371, #373, #374). The run against the deployed batch reads 70 works / 40 empty / 3 defect over 113 sitemap URLs, from 68 / 48 / 2 over 118: the eight empty frameworks pages left the sitemap and the repeated blog URL appears once. Of the three defects one was the crawler's own: Chromium reports a request the browser cancelled (net::ERR_ABORTED) exactly as it reports one the network failed, which made the home page a defect for a health route that answered 200 when asked directly. /azure/architecture-designs had kept the title it was copied with and never said "Azure". The third is real and stays open on #374: the article's <img> points at an .mp4, which re-hosting will store but cannot make render.
  • The content-manifest job can actually open its pull request (found by the first run that had a change to push). actions/checkout persists the job's GITHUB_TOKEN as an http.extraheader in .git/config, and that header outranks a token embedded in the remote URL, so the App installation token the job mints was never the credential git used. The push authenticated as github-actions[bot], which holds contents: read in that job by design, and the remote answered 403. persist-credentials: false on the checkout leaves the App token as the only credential. The job had reported success on every run since it was rewritten to open a pull request, because the published set had not changed since 2026-08-23 and the push step never ran.
  • The podcast page is the one audio surface, and the media route serves byte ranges (#349, ADR 0029). A provider's podcast page now lists the host's ingested episodes and the published Listen & Learn episodes for that provider in one date-sorted list, each row labelled with its source, played by one component with a keyboard-operable seek slider. The aws, azure and gcp copies of the page (the shared page with colours inlined) are gone; every provider routes to the shared page and the coverage test demands a metadata row for each. New GET /api/public/listen-and-learn/ episodes?platform= returns every approved episode with audio across a provider's certifications, projected to a listing allowlist (no transcript) and joined to its certification; GET /api/public/podcasts now carries feedUrl from admin_config/podcast_feeds — the document the ingest timer reads — so the RSS subscribe button and the ingest cannot name two different feeds. GET|HEAD /api/public/media/… honours a single Range: bytes=… with 206 and Content-Range, reading only the requested bytes through the SDK's ranged download; a start past the end answers 416; every success carries Accept-Ranges: bytes; caching is unchanged. Seeking into a Listen & Learn episode no longer downloads the whole file, and the range support is what a self-hosted feed would need from an enclosure host. ADR 0029 records the owner's no-paid-services decisions: RSS.com Free with manual upload and feed ingest, Gemini then Azure AI Speech with ElevenLabs deferred, StreamYard as a manual studio with the feed as the boundary. Nothing named after the previous host remains in frontend/.
  • The two live checks that need an admin's token run from Admin → Diagnostics (#355, #356). Both issues had the owner copying a bearer token out of developer tools and decoding it by hand. /admin/diagnostics does the checks in the signed-in session: it acquires the access token the way every API call does, decodes it locally, and keeps only claim names plus aud, roles and exp — the token, oid, sub and email never reach React state, the DOM or the clipboard, and the registry comparison is a boolean computed in the browser. The values it compares against come from a new GET /api/getAuthExpectations (admin-identity-http.js), which returns the audience and tenant the verifier is configured with and the App Role value the guard checks, so the page tests the API as deployed rather than a constant built into the frontend; it sits behind requireUser rather than requireRole because the caller being diagnosed may hold the role and be missing from the registry, and it discloses nothing a valid token does not already carry. The Labs panel submits one shell-echo job exactly as the Labs console does, reads the lab_jobs document back and cancels it so nothing is left queued for the next agent, then judges the result against the documented { jobId, type, status: 'queued' } response; a second button repeats the request through a plain fetch with no Authorization header and passes only on 401/403. Copy report puts a Markdown summary — names, booleans, statuses and job ids — on the clipboard for the closing comments on #355 and #356. Registered in .azure/api-surface.json; handler and page tests beside each module.
  • Three hand-seeded admin_config documents get an admin screen (#351, #352, and the feed list from #348/#349). Admin → Platform Settings (/admin/platform) reads and writes default_heroes (the per-provider cover a post gets when AI generation is off or fails), social_autopost (the enabled switch, the Publer accounts and the undo-window delay the publish trigger schedules against) and podcast_feeds (one RSS URL per provider for the two-hourly timer). Until now each could only be written by an operator holding a Cosmos data-plane role, typing the JSON by hand. The new GET|PUT /api/cms/platform-settings/{setting} (editor) normalizes every save to exactly the shape its consumer reads — hero keys resolve to the eight canonical providers and values must be a same-origin path or https URL, enabled must be a real boolean and the delay a whole number of minutes, podcast rows pass the timer's own isValidFeedEntry — and refuses unknown keys rather than dropping them, so the document can no longer drift from the code. A document seeded by hand that does not validate is shown empty with the reason, and saving replaces it. "Use bundled defaults" fills the covers with /images/default-heroes/<provider>.png; the Social card offers the accounts the Social Hub already lists from Publer, with a free-text id as the fallback. Every save is audited as platform_setting_updated with counts only: the route answers with the normalized document, as the page needs it to, but no log line or audit row ever carries document contents.
  • The sixteen already-published articles can be re-hosted from the Publish page (#374, the backfill). Admin → Publish gains an Images: re-host hotlinked action: it lists every published article whose body fields still reference a third-party image — counts, hosts and the last run, never a body — with each one selected, and "Re-host selected" runs them through the publish pipeline in batches of five, tabulating rewritten and failed counts and the failed hosts per article. Two routes back it, GET|POST /api/cms/content/rehost-images (editor to list, publisher to run, up to 25 ids a request). The write is processPublishContent under a new reason: 'rehost-images', which a plain republish of a live article would not have been safe to stand in for: that path re-runs the gates, re-arms cover generation on an article without a cover and the social caption on one that never posted, and rewrites Live and the publish dates. Under the reason the patch is the rewritten body field(s), the inlineImages summary and updatedAt, conditioned on the ETag, with a rehost-images version row — and nothing else. Pinned by tests that put the same live article through both paths.
  • Cosmos exporter infrastructure, inert until armed (#231, ADR 0028). The Azure half of the out-of-account export: a private cosmos-export container on the RA-GRS content account, a lifecycle rule that deletes its full/ and delta/ run blobs 35 days after creation (five weekly fulls plus their deltas, about 12 GB; the state/ change-feed tokens are never under it), a FEATURE_FLAG_COSMOS_EXPORT app setting on the Function App, and two scheduled-query alert rules that fire on the absence of a cosmosExportCompleted custom event — one daily for any completed run in the trailing two days, one on Mondays for the Sunday full. All of it hangs on one new workspace variable, cosmos_export_enabled, default false: the merge applies an empty container, a lifecycle rule and a "false" setting and creates no alert rule, so nothing runs and nothing can page until the owner sets the variable in HCP Terraform and confirms that second run. The rules are shaped around two Azure limits met while writing them: a log alert's query time range is capped at two days, so "no full in eight days" became "no full on the Sunday just gone", checked each Monday; and a rule evaluated daily cannot be stateful, so both are stateless and their descriptions say what that means. No new role assignment — the Function App identity already holds Storage Blob Data Contributor on the account and Cosmos Built-in Data Contributor on the database account.
  • Landing hero rotations and default covers are generated art (#371, #351). The four landing pages whose hero sets never existed (/gcp, /github, /terraform, /finops, twenty 404s per the audit) get their five images each under frontend/public/images/<provider>-hero/, and the eight default social covers #351 needs land at frontend/public/images/default-heroes/{azure,aws,gcp,github,terraform,ansible,vmware,multi}.png (1200×630). Nothing was bought and nothing is stock: a new dev script, frontend/scripts/generate-brand-art.mjs (npm run art:generate), draws each file from an SVG template — five distinct compositions per set (diagonal bands, concentric rings, isometric cubes, a node graph, layered waves) in the provider palette from src/index.css, with the provider name as low-contrast typography in the site's own Goldman face — and rasterizes it with @resvg/resvg-js, a devDependency only. Rendering is seeded and uses no system font, so the same script yields byte-identical PNGs on any machine (--check proves it against the committed files). Every PNG is 1155×924 8-bit RGBA like azure-hero/ and under 240 KB. The emptied hero arrays from PR #380 are restored; hero-assets-exist.test.js stays green, and generate-brand-art.test.js pins each committed file's dimensions, colour type and size. The owner can replace any file one-for-one with real artwork; nothing keys on it being generated, and the generator is not part of the build.
  • The Cosmos out-of-account exporter, as ADR 0028 designed it (issue #231, application half). A daily timer, cosmosExportScheduler, decides full (Sunday) or delta from the UTC date and enqueues one cosmos-export-container job per exported container onto the existing platform jobs queue; the worker pages SELECT * FROM c (full) or reads the change feed from the stored continuation (delta, latest-version mode, whole container) and streams the documents as gzip NDJSON into the private cosmos-export blob container at the Cool tier, writing the continuation only after the blob is safely written and a full run's checkpoint taken before it starts paging. Each job ends with a marker; the worker that finds every marker writes the run manifest once and emits the cosmosExportCompleted custom event the missing-run alerts count. The classification is code — functions/src/lib/backup/export-classification.js, the ADR's six classes — and a test holds it to infra/cosmos-containers.json: every one of the 72 provisioned containers exactly once, nothing unprovisioned, 60 on a full and 53 on a delta. Logs and telemetry carry container names, counts and durations, never a document. Inert until FEATURE_FLAG_COSMOS_EXPORT is true (or 1), which Terraform writes from cosmos_export_enabled; the container, lifecycle rule, alert rules and setting are the Terraform half. The app gains its first custom-event emitter (lib/telemetry.js, the edge probe's envelope shape, no SDK), and uploadBlobFromStream accepts an access tier and metadata. A restore runbook (docs/runbooks/cosmos-restore.md) gives the quarterly drill from ADR 0028 §6 as paste-ready PowerShell, and scripts/restore-cosmos-export.mjs reads the latest complete full plus its deltas from the container and upserts them into a target account, with --dry-run that only counts and --verify that counts the target back.
  • Article bodies stop hotlinking upstream images (#374). At publish time every external <img> or ![](https://raw.githubusercontent.com/saulpatinojr/HCW-HybridCloudWorks/main/…) URL in any body field (blogDraft, Content, content — the audited article kept an RSS stub in one and its eleven images in another) is fetched through the guarded fetcher the cover mirror already uses, stored under the article id in the public covers container and rewritten to the site's /api/public/media/… path; the version snapshot carries the rewritten body. A URL that cannot be fetched is left exactly as it was and named on the document in a new inlineImages summary (fields, rewritten, failed, failedUrls, at), and no failure of this step can fail a publish. Wired into the HTTP publish, the Telegram approve job and the scheduled publisher; a republish re-hosts an already-published article. scripts/scan-inline-images.mjs reads the committed content manifest and lists the published articles still carrying third-party image URLs, so the size of the class is a command away.
  • Section pages with nothing published leave the sitemap (#373, part 2). The content manifest now carries type (the field the section pages select on, already public through the content list) and a sections map counting framework items per provider, with an _unattributed row for items whose provider the page would have to infer. The pre-render omits /<provider>/frameworks from sitemap.xml when its count is exactly zero and nothing is unattributed; the page is still rendered and served with its empty state. A manifest without sections — the deployed route predates the field until the next Functions deploy — drops nothing. Coder-corner and code pages are deliberately not counted: they fall back to the legacy blogs container, which the manifest does not read, so their zero would not mean an empty page.
  • ADR 0028 designs the Cosmos out-of-account export (issue #231). Weekly full export plus daily change-feed deltas, run as a timer that fans one queue message per container onto the existing platform jobs worker, written gzip-NDJSON to a private cosmos-export container on the RA-GRS content account at Cool tier with a 35-day lifecycle, monitored by an alert on the missing run. The 72 provisioned containers are classified: 43 authored and 10 configuration containers get fulls and deltas, 7 operational-record containers get fulls only, 12 regenerable, seed and transient containers are excluded, and a test will hold the classification to the provisioned list. Deletes reach the copy through the weekly full, accepted and written down. Status is Proposed: RPO 24 h / RTO 8 h, the class C policy, the same- subscription copy and the LRS host storage acceptance are the owner's four decisions, listed on #231.
  • Cost Analysis costs Cosmos recovery against the measured account size (#385, issue #231). The owner measured DataUsage on 2026-09-06: 2.39 GB. The page gains a section pricing what recovery already costs (continuous 30-day backup about $0.48 a month, a point-in-time restore $0.36) and the two export shapes from #231 per week and per month: daily full exports at about $18 a month in request units versus weekly full plus change-feed deltas at $3–4, both with 30-day blob retention on the RA-GRS content account. The numbers settle the design — weekly full plus deltas, Cool tier, about $5 a month for the whole recovery posture — and the measurement plan gains a line to re-read the size when the exporter is armed. Prices are Microsoft Learn list prices for one Central US region, with the calculator named as the place to re-verify.
  • One sitemap entry per URL, and listing pages that say "nothing published yet" when that is the truth (#373, part 1). The content manifest carried one slug three times, so the sitemap advertised the same blog URL three times and the pre-render wrote the file three times; buildManifest now keeps the first occurrence and names the rest in skipped, and the pre-render dedupes its route list as a second guard. ContentListingTemplate distinguished nothing from filtered-to-nothing only by accident: with zero items it still said "found matching your criteria" and offered a Clear Filters button. It now says "No {type}s published yet." and the published pages audit recognises that copy (and the frameworks page's) as an empty state rather than a defect. Part 2 — dropping zero-item section pages from the sitemap — stays on #373.
  • The audit report records the post-deploy re-run, and empty-state copy no longer marks a full page empty (#384). After #376, #379 and #380 were deployed on 2026-09-06 the crawl went from 0 works / 11 empty / 107 defect to 68 / 48 / 2; the dated report carries both columns. Three of the 48 were landings marked empty because their podcast widget says "No episodes available yet." beside two thousand characters of real content, so the crawler now counts empty-state copy only on a page under 800 characters and notes it otherwise. The rest of the 48 are real: about thirty section pages with nothing published behind them, now the stated scope of #373.
  • Copilot code review MCP configuration applied (#378; issue #369, closed 2026-09-06). The owner completed every step the runbook laid out, in order: the hcw-azure apply that created github_copilot_review and its seven resources; COPILOT_REVIEW_CLIENT_ID seeded from the new output; the setup workflow dispatched by hand and green on its second run (the first, fifty-eight seconds after the merge, ran before the apply and failed on an empty client id — the runbook now says steps 1 to 3 wait for the merge and what that failure looks like); the read-only GitHub App HCW Copilot Review Reader registered, installed on this repository only, its App ID and private key stored; and the five-server configuration pasted into the repository's Copilot settings. The runbook status and the required-inputs rows record the values as set. The first real review, of #378 itself, then showed two things the documentation had left open. Copilot code review does run copilot-setup-steps.yml, but in Copilot's own runner, which resolves no vars.*: azure/login ran with every input empty while the same file dispatched by hand signed in fine. The workflow now reads the client, tenant, subscription and App identifiers as secrets.COPILOT_REVIEW_… || vars.…, so the Agents store Copilot exposes is preferred and the repository variables still serve a manual run; the oidc-subjects guard reads the variable out of that expression; the standards record the identifiers-in-Agents exception; and the owner mirrored the four values the same day (step 2b, issue #381, closed 2026-09-06). And a review session runs MCP in read-only mode, keeping only tools whose servers annotate them readOnlyHint: true — the session log kept terraform and dropped every other server, the built-in Playwright included, as having no allowed tools. The Azure and GitHub servers annotate; the runbook records what that means for the two documentation servers and how to read the next session log. Every review session of #378 itself ran the step list main held at the time, so the first review after the merge is the one that shows the new steps.
  • Four landing pages no longer request hero images that do not exist (#380, issue #371). /gcp, /github, /terraform and /finops each listed five /images/<provider>-hero/N.png files that were never added, so every visit loaded with seven 404s and six broken image boxes (the audit's broken images class). The lists are empty until the sets exist — the carousel renders nothing for an empty list — and frontend/scripts/hero-assets-exist.test.js now fails the frontend suite on any /images/… path a landing page references that is not under public/, so a set cannot be referenced before it is added. #371 stays open for the assets themselves (1155×924 RGBA PNG, like azure-hero/).
  • The public podcast list hides episodes whose media is gone (#379, closes #372). PodBean's CDN has answered 404 for every stored episode since 2026-09-05, and /azure/audio rendered a player for each that played nothing (the audit's media not served class). listPodcasts now drops a row whose mediaUrl host is on RETIRED_PODCAST_MEDIA_HOSTS (mcdn.podbean.com, feed.podbean.com) or that carries a mediaUnavailableAt mark, and total counts only what it serves; the rows stay in Cosmos until the new host's feed (#349) replaces them, so no data-plane write is needed. The page falls back to its own "No episodes available yet." copy, which is the honest state. A per-run liveness check that sets mediaUnavailableAt is deferred until a feed exists again: with the list empty the timer has nothing to check.
  • TODO.md points at the board (#377, closes #362). The handling rule and the "where the open items live" section name the organization project (https://github.com/orgs/HybridCloudWorks/projects/1) and its Priority field; the owner enabled auto-add on 2026-09-06, so a new issue lands there without a manual step. The index table stays as the offline copy.
  • Every pre-rendered page was a spinner with the real page hidden below it; now the page is the page (#376, closes #370). The audit (#361) found React error 419 on every hydration and two inline scripts the CSP blocks. The cause was one React default: Fizz outlines any completed Suspense boundary larger than progressiveChunkSize (12.8 kB) — the shell gets the route's fallback spinner and a <template id="B:0">, the page follows as <div hidden id="S:0">, and an inline $RC script swaps them in the browser. Every real page is bigger than 12.8 kB, so since the pre-render step shipped (#296) each document has carried a visible spinner in <main>, a hidden copy of the page, and a swap script this site's hash-less CSP refuses to run — hydration then threw 419 and re-rendered from scratch, and anything that does not run JavaScript read the fallback. prerender-entry.jsx now passes progressiveChunkSize: Number.MAX_SAFE_INTEGER, so Fizz inlines every completed boundary and the output has no $RB/$RC scripts at all; the CSP stays exactly as strict as it was. prerender.mjs refuses to write the streamed form (findStreamedBoundary, tested for each of its three shapes), because the existing shell floor was blind to it: header, footer and spinner cleared 420 characters easily. Verified on a full local build: 120 documents, zero pending markers, zero runtime scripts, the route's <main> inline in each. The audit's console csp-inline-script and console react-hydration-419 classes — 107 of the 118 crawled pages — should clear on the next deploy.
  • Every published page audited, in a browser, with a verdict each (#375, issue #361). frontend/scripts/audit-published-pages.mjs crawls every URL in the live sitemap with Playwright Chromium and records HTTP status, failed and 4xx/5xx requests (the public API called out separately), console and page errors classified by name, provider identity (title, og:title and theme class against the URL's provider — the #183 class), empty-state copy and main-region size, broken images and unreachable media; verdict works / empty / defect per page, a matrix and a defect-class summary as JSON and Markdown. Failures the repository has already decided to live with (the #175 data snapshots) are notes, not defects, so a decided item is not re-raised every week. audit-published-pages.yml runs it Mondays and on demand and uploads the matrix; npm run audit:pages runs it locally. The first run — 120 sitemap entries, 118 unique URLs (one repeated three times; the crawler de-duplicates and reports the count), 0 works, 11 empty, 107 defect — is recorded at docs/architecture/published-pages-audit-2026-09-06.md, and its defect classes became five issues: #370 (every prerendered page ships a pending Suspense boundary and the CSP blocks React's completion scripts, so hydration error 419 fires site-wide — the reason 107 pages are "defect"), #371 (twenty landing hero images missing for four providers), #372 (podcast rows with dead PodBean media still served), #373 (six empty section pages in the sitemap) and #374 (curated article bodies hotlinking upstream images). The audit closes when every row has a verdict and every defect has an issue, which is now; fixing them is the issues' work.
  • Copilot code review gets an MCP configuration (#368). Every Copilot review had been ending with the hint to configure MCP servers for context-aware reviews. .github/copilot-mcp.json is now the reviewed source of record: five official, read-only servers chosen for what the repository contains. Three need no credential — Microsoft Learn (Azure Functions, Cosmos DB, Key Vault, Static Web Apps, Entra ID), Cloudflare's public documentation server (the edge availability probe and DNS) and the HashiCorp Terraform MCP server, pinned by image digest (@sha256:…, the immutable reference; the 1.3.0 tag beside it is for readability only, since a tag can be retargeted) and held to the eight Registry read tools on both its command line and Copilot's allowlist, with no HCP Terraform token so the live workspace stays unreachable. Two reach live systems and are kept read-only by construction. The Azure MCP Server signs in with no stored secret: copilot-setup-steps.yml runs azure/login under OIDC in the copilot environment as a new user-assigned identity, github_copilot_review in infra/oidc.tf — Reader on the four workload resource groups and nothing else, trusting only the …:environment:copilot subject in both forms — and the server runs --read-only with fourteen hand-picked control-plane tools (no Key Vault, Cosmos or blob content, no log queries), pinned on its command line and in Copilot's allowlist. It is a third identity rather than a reuse of github_reader, which holds a config write. The GitHub MCP server is widened to the Actions, code-scanning, Dependabot and discussions toolsets with no personal access token of any kind: a dedicated, read-only, single-repository GitHub App (HCW Copilot Review Reader) whose one-hour installation token the same setup job mints per session through scripts/github-app-token.mjs — now taking a GITHUB_APP_PERMISSIONS override so the manifest publisher's write scope and the reviewer's read scope come from one minter — and hands to the digest-pinned ghcr.io/github/github-mcp-server container over --env-file, running --read-only with thirty-one named read tools on its command line and in Copilot's allowlist. The App's read-only permissions are the ceiling for anything its key could ever mint. The oidc-subjects guard learned the new client-id variable, set-github-variables.ps1 seeds it, and the standards pages record the variables and the private key. The runbook Copilot code review MCP servers carries the reasoning, the tool table, the owner steps in order (apply, seed, prove the sign-in, token, paste) with what success looks like at each, and the rollback; the code-review skill names which server to consult per component. The owner steps that turn it on are issue #369.
  • ADR 0027 records the documentation-site decision (#367, issue #360). The move from the Wiki to docs/ shipped in #363 with its reasoning spread across the PR, the issue and the CHANGELOG; the ADR puts it in the register where the next reader will look — source layout, MkDocs Material and why not Docusaurus or Jekyll, the root-document hook, the Pages deploy and DNS-only CNAME, the redaction gate, the Wiki retirement, and the accepted consequences (strict builds, the uncurated archive, the one reused ADR number). Closes the last acceptance item on #360 that a session can close; switching the Wiki feature off after a week is the owner's.
  • Retire Wiki stubs the Home page too (#364). The dry run on 2026-09-06 reported keep (not in map): Home: Home maps to the site root, whose path is the empty string, and the guard tested truthiness instead of presence. It now tests is None, so 141 pages are stubbed rather than 140 and the Wiki's landing page points at https://docs.hybridcloudworks.com/ like the rest. Found by running the workflow with dry_run on, which is what the input is for.
  • Documentation moved from the GitHub Wiki to a MkDocs site under docs/, published to docs.hybridcloudworks.com (#363, issue #360). Owner decisions 2026-09-05: MkDocs Material, host name docs.hybridcloudworks.com, source is the former wiki/ folder plus the live Wiki. The live Wiki held 106 pages that were never staged in the repository (ADRs 0001–0017, the ADR template, the target-architecture and Well-Architected pages, the migration inventory and smoke-test records, and about eighty Firebase-era guides), so it was cloned and everything came across: 141 pages, arranged as decisions/, runbooks/, standards/, architecture/, content/, history/ (the Azure migration record, banner-marked) and archive/ (the Firebase era, banner-marked, indexed from its own page). Wiki-style links were rewritten to relative paths and mkdocs build --strict fails the new Docs site check on any broken link or anchor, which is what found the seven stale anchors that the Wiki had been serving as dead links. The duplicate ADR number 0021 (Container Apps CI runner, deferred, versus Key Vault purge protection) is kept as two records with a note on the first. README.md, CHANGELOG.md and TODO.md stay at the root and are pulled into the site at build time by scripts/docs/hooks.py, so nothing is copied by hand. Deployment is docs-pages.yml with OIDC to the github-pages environment; the Pages site was created with the Actions build type on 2026-09-05. sync-wiki.yml is replaced by retire-wiki.yml, a workflow_dispatch that overwrites every Wiki page with a pointer to its new URL (dry-run by default) — to be run once the domain serves, since a pointer to a host that does not resolve is worse than the old page. The repository-structure policy now sanctions docs/ instead of wiki/, allows mkdocs.yml at the root, and requires the README to name the docs site. Terraform adds cloudflare_dns_record.docs_pages, a DNS-only CNAME to hybridcloudworks.github.io (proxying it would leave GitHub unable to issue the certificate); the apply and the Pages custom-domain setting are the owner's, in that order. Every current-state reference to the Wiki in the repository was repointed (README, CONTRIBUTING, infra/README.md, the issue-template contact link, the code-review skill, the infrastructure plan's evidence URLs); the historical wiki/ links in this file were rewritten to their docs/ paths so they resolve again on GitHub.
  • Open work moved from TODO.md to GitHub issues (#359). Owner instruction 2026-09-05: the items are worked from the issues list from now on, so the tracker's remaining sections — the five optional seeds, the Labs test-coverage case and the four live confirmations — became issues #350–#358 and TODO.md keeps only the accepted risks, an index to every open issue, and a handling rule saying an open item is an issue. Three labels were added so the list reads without opening anything: owner-gated (a credential, console action or spend decision), live-check (needs an authorized operator against the deployed estate) and podcast. The three pre-existing issues were re-verified against main the same day and each carries a dated comment: #231 (no Cosmos exporter, no timed restore), #180 (createContentFromRecording still under rpc.notImplemented, though #342 removed its credential blocker) and #127 (no managed WAF ruleset in Terraform). Found while doing it, and filed rather than fixed here: the Podbean feed fetchPodcastFeeds has polled every two hours since #332 returns HTTP 410 Gone, so the timer has failed on every firing and the five podcast pages still advertise a PodBean subscribe button that leads to it (#348); and the replacement stack the owner asked for — RSS.com hosting, ElevenLabs speech as a third speech/index.js provider, StreamYard for live and recorded shows — is evaluated with vendor facts, prices, an integration design and the owner decisions it needs in #349. The check-todo-changelog-movement gate passes: no T- identifier left the tracker in this change.
  • The tracker is open work only, and the repository is leaner (#346). Owner instruction 2026-09-06: TODO.md keeps four things — the optional seeds, the one live test-coverage case, the live confirmations still to come due, and the accepted risks — and nothing else. Everything below moved out of it verbatim on that day (headings demoted so the changelog's own structure holds): the status narrative, the attack sequence, the "what is open" note, the watch-out-for note, the long-form T-518 record, the owner-gated preamble carried from REVIEW.md, the owner-decisions table, and the struck live confirmations and accepted risks. Removed from the tree in the same change, each with no remaining consumer: frontend/.firebaserc (the Firebase project pointer; the site has been on Azure since 2026-08-21), scripts/cutover/02-swa-token.ps1 (minted a deploy token that T-727 deleted on 2026-08-30), frontend/scripts/axe-quick-check.mjs (an ad-hoc accessibility probe wired to nothing), frontend/.lighthouserc.json and frontend/.pre-commit-config.yaml (neither referenced by any script, hook or workflow), and .azure/insights.json (an empty array). #346 also removed frontend/.copilot/ and .github/templates/; that was wrong, and #347 restored both the same day at the owner's direction — Copilot is GitHub's tooling and reads frontend/.copilot/ from the frontend project, and the 21 document templates are kept to be used as needed. The repository-structure validator's allowlist lines for both directories are back with a comment recording why.

Status — 2026-09-05

Nothing is open as of 2026-09-05. The last two items — T-518, arm the remaining timers, and T-726, prove the nightly refresh — closed on the owner's decision that evening, with all 18 timers armed and the observation reads no longer a gate. Both records are in CHANGELOG.md. What follows below the status is kept as the working method and the long-form evidence, not as a list of work.

The one live degradation closed overnight. T-763 — the manifest route merged on 2026-08-30 against a Function App last deployed 84 minutes earlier — was fixed by Deploy Functions run 81 at 2026-08-31 23:45 UTC. Verified rather than assumed: manifest run 11 at 2026-09-01 00:06 went green end to end and reported "No change to the published set", so the committed manifest already matched what the API serves. Recorded in CHANGELOG.md and removed from this list.

The detection gap is closed — T-519 was armed on 2026-09-01. The probe's secret held the Instrumentation Key rather than the connection string; the piped command in edge/availability-probe/wrangler.toml replaced it, twelve success == 1 / HTTP 200 rows landed on the */5 cadence before arming (a full PT30M window needs six), and the apply created alert-api-reachability-prod-cus in rg-web-site-prod-cus. The registered function count was 122 before and after the apply's restart. Recorded in CHANGELOG.md and removed from this list.

T-719 and T-721 closed on 2026-09-02, by owner decision. The host verbosity is cut at the source (#321, host.json default and Function to Warning), the deploy was dispatched the same day, and the owner closed both without gating on the cap-day reading — the below-cap volume is expected confirmation, not a closure criterion. The accepted risk (log alerts sleep if the cap ever binds again) is recorded below; the Basic-table-plan move stays in reserve in CHANGELOG.md if the reading ever says the cut was not enough. What remains is one observation and one procedure.

The architecture review is closed. All 62 findings (T-701…T-762) are resolved or carried below as owner gates. The per-finding record — method, evidence standard, every failure mode and outcome — is wiki/Architecture-Review-2026-08.md, which is now a dated historical document rather than a live list. Its last three owner-gated findings are all closed — T-749 on 2026-08-31, T-719 and T-721 on 2026-09-02.

The hcw-azure workspace is VCS-connected, and auto-apply is off. Merging infra code queues a plan that waits for approval at https://app.terraform.io/app/hcw/workspaces/hcw-azure/runs — it does not apply on its own, and terraform apply from a desktop is refused outright. Every run carries the permanent diff, which replaces three azapi resources and restarts the function app.

This file was wrong about that twice in opposite directions, and the reason is worth keeping: HCP Terraform's workspace Description is free text beside the real settings, validated by nothing, and it read "CLI-driven; no VCS connection". That sentence was read and reported as configuration. Corrected in the workspace on 2026-08-31. A field that contradicts its own workspace reads exactly like a setting.

This table is the list below, and nothing else. An earlier version counted five, because the two owner actions left by closed findings were tracked in a different section from the five findings that carry T- numbers — so the summary said five above a list of seven. That is the T-722 defect, in the document restructured to prevent it, and it is why there is now one table rather than a count and a list that can drift apart. Found by review, 2026-08-31.

# Open item Priority What closes it
— (empty since 2026-09-05) — —

The table emptied on 2026-09-05. T-726 (the nightly refresh) and T-518 (arm the remaining timers) were the last two rows; both closed by owner decision that evening and their records, with the wave table that T-518 carried, are in CHANGELOG.md. The 2026-09-02 audit that traced every container the public read layer reads back to whatever writes it found T-764 (closed 2026-09-04 by Wave 2) and T-765 (closed 2026-09-05: the owner retired the insights panel). T-766, the witness gap that Wave 2 exposed, closed 2026-09-05. All records are in the changelog.

When the table fills again, the table and the sections below it are kept in the same order, and that order is the one to work them in — not a sort of the Priority column. Checked against each other by number AND by T- identity, never by counting rows: on 2026-09-01 an edit reordered the table while renumbering the sections in document order, and because both still read 1..5 a digits-only check passed with every row pointing at the wrong section.

Closed on 2026-08-31 and removed from this list: seeding TFC_TOKEN (done), and T-749, the SCM lock — Terraform owns scm_ip_restriction_default_action, a live read shows Deny all, so the variable is already true and the last Deploy Functions run succeeded on 2026-08-30 at 01:21 UTC. The tracker had it open against a reality where it was applied.

The attack sequence — one working order across everything open

This section sequences what the rest of this file already tracks; it adds no items and restates no procedures. Each phase points at the one section that carries the commands and the success criteria, so this list cannot drift from those sections the way a restatement would — the T-722 lesson, applied in advance. The ordering rule is dependency, not priority: the timers arm after the verbosity cut has deployed, so their volume lands in real headroom.

Phase What Where the procedure lives Why this position
1 ~~Fix the probe's secret, wait for six availabilityResults rows, arm the reachability alert (T-519)~~ Done 2026-09-01 — record in CHANGELOG.md Twelve healthy rows, alert-api-reachability-prod-cus live in rg-web-site-prod-cus, function count 122 before and after the restart
2 ~~Settings sweep: delete the three stale workspace variables, set the production deployment-branch rule, decide the two ruleset booleans~~ Done 2026-09-02 — record in CHANGELOG.md Three variable rows deleted; production restricted to main; ruleset decided: branches must be up to date before merge, thread resolution not required
3 ~~Cut the host verbosity at the source (T-719), pull T-721's lever~~ Done 2026-09-02 — record in CHANGELOG.md Closed by owner decision with #321 merged and the deploy dispatched; the below-cap cap-day reading is expected confirmation, not a gate. The SWA tier question moved to the Owner decisions and external access table (Owner decisions, below in this file)
4 ~~Arm the remaining timers in three applies — 3a, 3b, then waves 4, 5 and 6 together (T-518)~~ Done 2026-09-05 — record in CHANGELOG.md All 18 timers armed; the two reapers armed one per apply with the delete switch flipped after two zero dry-runs; the observation reads dropped as a gate by owner decision
5 ~~Prove the nightly refresh's App-token path (T-726)~~ Done 2026-09-05 — record in CHANGELOG.md Closed by owner decision on the built-and-configured state; the first content change remains the live proof, and the failure signatures (401 = wrong App id, 404 = App not installed) are in the changelog entry
6 Optional features: seed the keys and documents you actually want (the insights-panel decision, T-765, was made 2026-09-05: retired) the owner-gated issues Decisions, not repairs — nothing above depends on any of them
7 Live confirmations as they come due: Entra token claims, the timed restore against RTO 8 h / RPO 24 h (issue #231), third-party webhooks, the authenticated Labs check the live-check issues and issue #356 Each needs a live environment or a third party on its own schedule; none blocks Phases 1–5

The deliberately unscheduled feature backlog — analytics-informed topic weighting, the voice-drift monitor, stale-post refresh, A/B titles, the duplicate-angle advisor — stays in Blog-Machine § Backlog on purpose: none of it is open work, and pulling it here would turn this file into a wish list. The one backlog entry that is also a real route gap, createContentFromRecording, already lives in the AI-providers and third-party rows of the Owner decisions and external access table (Owner decisions, below in this file) — it stays in .azure/api-surface.json notImplemented until its provider credentials exist.

What is open, and exactly what closes it

Nothing, as of 2026-09-05. The sections that stood here — T-726 and T-518 — moved to CHANGELOG.md with their evidence, including the wave table T-518 carried. The long-form records below stay as the account of how each was established.

Watch out for

  • Deployment drift is now measured, not noticed. .github/workflows/monitor-deploy-drift.yml runs every four hours and fails when a service has been behind main for 24 hours or more, emailing the owner the way the other monitors do. A failure names the service, the number of undeployed commits touching its paths, and the subject of the oldest one.

    The fix is to dispatch that service's deploy. Both are dispatch-only by a recorded decision, so merging never ships them; this check is the thing that says so. Deploying clears it on the next run.

    It measures AGE, not commit count, and that is the whole design: the manifest 404 was one commit behind and the frontend was thirty-five. No count threshold separates those. Both had sat undeployed for days.

The long-form records

Everything below is evidence for the items above, not a second list of them. Each carries how the finding was established, what was observed, and the limits of that observation — the part a summary loses and the next reader needs before changing anything.

T-518 — 15 of 18 timers are still no-ops; the mechanism is proven

Closed 2026-09-05: all 18 armed. Kept as the evidence record; the wave table and the closing decision are in CHANGELOG.md.

Gate: owner — TODO.md, Timers and the availability test.

functions/src/functions/schedulers.js checks FEATURE_FLAG_SCHEDULERS first and skips the handler before reading the per-timer flag. Until 2026-08-24 that setting was a hardcoded literal in main.tf, which meant enabled_timers could not arm anything at all and no document said so; it is now var.schedulers_master_enabled. Arming needs both it and a name in enabled_timers, through the four gates in Cutover-Runbook step 5 — where the acceptance criterion is the observed invocation, not the applied setting.

The master switch went true on 2026-08-30, with CHECK_AGENT_HEALTH and CLEANUP_TEMP_STORAGE in enabled_timers. That left sixteen no-ops at that moment — see the paragraph below for where the count stands now — so the platform still ran almost no scheduled work, but "nothing is scheduled" was no longer true and the arming mechanism was no longer an assumption.

PUBLISH_SCHEDULED_CONTENT was armed and then observed on the evening of 2026-08-30 Chicago time — 2026-08-31 in UTC, which is why the surrounding entries and the timestamps below appear to disagree. Every timestamp in this section is CDT (-05:00), matching what the host itself writes; the two UTC instants are named as UTC where they appear. The observation is unusually clean, because it straddles the apply:

publishScheduledContent  8 invocations  4 ran  4 skipped
                         first 2026-08-30 18:30:00 CDT  last 2026-08-30 20:15:00 CDT

Eight invocations at exactly fifteen-minute spacing with no gap. The four at 18:30, 18:45, 19:00 and 19:15 skipped; the four at 19:30, 19:45, 20:00 and 20:15 ran. The apply that set the flag landed at 00:30 UTC on 2026-08-31, which is 19:30 CDT on 2026-08-30 — so the split falls on the boundary rather than near it. That is the same timer, observed skipping and then running, with the flag change as the only variable: a stronger reading than a bare "it fired", because it also proves the gate the flag controls.

The clock came from the host in the same run: eight ScheduleStatus rows, all -05:00, firing on :00 :15 :30 :45 Chicago local. So both halves hold for this timer independently.

One honest limit, which the script prints itself: a RAN invocation is one whose traces carry no skip line, and a dropped .User trace would look identical. The durable side effect — content actually transitioning to published — is the second witness Cutover-Runbook Gate 4 asks for, and it only appears when something was genuinely due. Nothing was, so the no-op is correct and unwitnessed by that second path.

So of the eighteen: fifteen remain no-ops, and CHECK_AGENT_HEALTH, CLEANUP_TEMP_STORAGE and PUBLISH_SCHEDULED_CONTENT are armed and observed.

It gates two other things, which is why it outranks its own blast radius: the Blog Machine's scheduled throughput (forgeScheduled, publishScheduledContent), and the Cosmos backup-tier change from T-707, which only pays for itself once scheduled work is generating documents. It also gates any meaningful cost measurement — a bill taken while nothing is scheduled prices an idle platform (Migration-Plan §7).

Both halves of the gate passed on 2026-08-30. The clock half needed nothing armed: app.timer() registers on the real schedule unconditionally and the flag is checked inside the handler, so every timer had been firing since deploy and logging disabled — skipping, with the host writing ScheduleStatus carrying WEBSITE_TIME_ZONE offsets each time. cleanupTempStorage reported "Last":"2026-08-29T00:00:00.005764-05:00", "Next":"2026-08-30T00:00:00-05:00" — local midnight, offset applied, which is the §7 comparison delivered by the platform rather than computed by a script.

The handler half came from checkAgentHealth after arming: twelve invocations between 04:55:00Z and 05:50:00Z, exactly five minutes apart with no gaps, no disabled — skipping anywhere in the window, and the handler's own [checkAgentHealth] 0 agent(s) marked offline on each one. Host row and .User row are separate emitters, which is what "two independent witnesses" means; the zero is a correct no-op, not a missing witness.

Two departures from the runbook are worth recording. Both timers were armed in a single apply rather than one at a time — safe here only because TEMP_STORAGE_CLEANUP_DELETE pins cleanupTempStorage to dry-run (functionapp.tf), so the second timer could not touch data. And the evidence above was read with direct KQL, not through scripts/cutover/05-verify-timer.ps1: that script reported a tally of tens of thousands of invocations for a query returning two rows, and was rewritten the same day to aggregate in the workspace instead.

That rewrite treated a symptom, and the miscount was root-caused on 2026-08-31 instead. az on Windows is a batch file, which cannot receive an argument containing a newline, so a multi-line query handed to --analytics-query arrived truncated at the first line break: AppTraces survived and the whole pipeline after it was discarded. The call still exited 0. Azure ran the truncated query and returned every unfiltered row in the table, so the script rendered tens of thousands of rows with every projected column blank — and its preflight, truncated the same way, read a missing column as zero and reported "no worker traces" in the same run. The tell was in the record for a week: the count barely moved between -Hours 1 and -Hours 24, and a window that changes 24-fold cannot return the same total unless the window is not being applied. Fixed by flattening every query before it leaves PowerShell, and by asserting that returned rows carry the columns the caller asked for — a truncated query answers a different question rather than failing, which is why nothing in the error handling ever fired.

Confirmed end-to-end in the same run (2026-08-30 CDT / 2026-08-31 UTC), on the Windows host where it failed: the same command that had returned 58,265 blank rows returned one correct summary row, the ScheduleStatus section came back filtered to the requested timer and ordered, and the preflight reported 431 worker traces where it had been reporting none. That last number is the one worth remembering — the "telemetry gap" this script reported three times never existed.

The fifteen that remain go one at a time, each observed firing before the next is added.

Owner-gated work, carried from REVIEW.md

Everything from here to the end arrived from REVIEW.md on 2026-08-29. These need tenant administration, production approval, a credential, external access, or a live confirmation — none of them can be closed from a checkout.

One section was deleted rather than carried: "Immediate: restore admin access". It described a live 403 from POST /api/bootstrapCurrentUserAdmin and asked for the Entra Admin app role to be assigned. The owner confirmed on 2026-08-29 that the admin portal loads and signs in immediately, so the role is assigned and the 403 is gone. It had been sitting at the top of the document marked Immediate — the loudest item in the tracker, describing something that was already fixed. The Admin app role assignment survives as one clause of the Entra row below, which is where it belongs.

Owner decisions and external access

Item Human action required Safe repository-side state
Entra application Confirm SPA client ID, tenant ID, API audience/scope, redirect URIs, consent, and the Admin app role assignment frontend/.env.example documents names; no client secret is committed
Frontend release Approve whether releases remain manual or become push-triggered. The credential half of this row is closed (T-727, 2026-08-31): the deploy mints its token from ARM per run under federated identity, and the stored secret is deleted — there is nothing left to provide or rotate deploy-azure-frontend.yml stays dispatch-only
Production infrastructure Approve HCP Terraform plan/apply and any DNS, custom-domain, or Cloudflare changes Terraform remains the infrastructure source of truth
~~Timers and the availability test~~ Done 2026-09-05 — record in CHANGELOG.md All 18 timers armed by 2026-09-05 (T-518); the reachability alert armed 2026-09-01 (T-519)
Recovery objectives (decided 2026-08-30) RTO 8 hours, RPO 24 hours. Chosen to match what the estate can actually meet today — periodic Cosmos backup, one operator, no on-call — rather than an aspiration nobody has rehearsed. Deliberately not RPO 1 h: that needs the T-707 continuous-backup tier, which costs money while the platform is still nearly idle and would commit to a drill that has never been run. Revisit once scheduled work is generating documents, which is also when T-707 starts to pay for itself. The remaining work in issue #231 is now measurement against these numbers — a timed restore — not the numbers themselves Cosmos carries Continuous30Days; content/media storage is RA-GRS with versioning and soft delete; Functions host storage remains LRS with soft delete. No scheduled out-of-account Cosmos export exists, no restore has been timed, and no result is justified against a stated objective
Key Vault Provide only the secrets needed by enabled features; never put values in GitHub variables or Vite config. The approved procedure changed on 2026-08-29: seeding is now Admin → Platform → API Keys, and the desktop script is break-glass rather than the default path Code reads secrets server-side and degrades optional integrations when absent
Function App vault write (decided 2026-08-29) Approved. The app may create new secret versions, through a CUSTOM role holding only Microsoft.KeyVault/vaults/secrets/setSecret/action — not Key Vault Secrets Officer, which would also grant get, list, delete and purge. It may also refresh its own Key Vault references (Microsoft.Web/sites/config/Write, scoped to the one site, with config/list/action excluded so it cannot read its settings back). Weighed against what it replaces: the previous procedure opened the production vault's firewall to a human IP on every rotation, and left it open once The app cannot read a secret back out of the vault, cannot delete one, and cannot enumerate its own app settings through ARM. /api/cms/secrets is super_admin on both verbs and returns no value in any response — asserted by scanning the whole serialised body, not by trusting a field list
GCP pricing integration Seed GCP-BILLING-API-KEY if the GCP column in the public pricing tool is wanted, or leave it unseeded and that column stays absent. Get it from the GCP console: enable the Cloud Billing API, create an API key, restrict it to that API. This is not a billing credential — the Cloud Billing Catalog API serves the public price list, and it is read for the site's comparison tools, not for anything this estate is charged for No GCP credential is stored in the repository. The service-account JSON this row used to ask for is retired (2026-08-29): the API key is what Google documents for this API, and it removed a vault SDK client, an OAuth library and a bespoke seeding script
AI providers Decide which external providers should be enabled and provide their keys through Key Vault The AI router only enables a provider when its server-side key is present
Third-party integrations Provide owner-controlled Publer, Klaviyo, YouTube, Telegram, Hostinger, or other credentials and approve webhook changes before activation Integration secrets are server-side and optional paths remain gated
Listen & Learn speech Nothing to provide: it synthesises with Gemini TTS on the existing GEMINI-API-KEY. Audio is billed against that key at roughly $0.17 an episode / $0.87 a certification on the default model; every run is logged to the AI Engine usage tab under "Breakdown by Feature", so the spend is checkable there rather than estimated. Azure AI Speech is a written, tested fallback for the day the preview Gemini TTS models are retired; using it means creating a Cognitive Services resource, which is a spend decision and is not assumed Provider is chosen by key presence, Gemini first. With no key at all the feature still publishes each episode's transcript, takeaways and videos and records audioError instead of failing
Listen & Learn video links Seed YOUTUBE-API-KEY if the curated "watch next" links are wanted. One certification costs ~505 of the default 10,000 daily quota units Optional. Without it, episodes generate and publish with an empty video list
VPS Labs agent Provide the host operator, Entra client/certificate, API scope, and deployment approval for the Hostinger agent vps-agent/ uses the API and holds no database credential
~~Static Web App tier~~ Done 2026-09-05 — record in CHANGELOG.md Owner decided Free (#341). Both Free limits verified from the repository before the change: exactly two custom domains (apex and www, Free's maximum) and a build of about 117 MB against 250 MB. Two-way door: a third domain or a bandwidth overage is the signal to move back
~~ai_insights container~~ Done 2026-09-05 — record in CHANGELOG.md Dropped by #340's apply, which the owner read and approved at exactly one destroy plus the azapi baseline; #341 restored the container prevent_destroy guard in the apply that followed
~~Provider-section go-live~~ Done 2026-09-02 — record in CHANGELOG.md Owner decided all 24 pages ship at once. Every provider page is live; GUARDED_FILES in frontend/scripts/validate-provider-pages.js is now empty and the guard mechanism is retained for the next page that needs it

Live confirmations completed

response in the deployed environment.
change.
  • ~~Observe an alert actually being delivered.~~ Done 2026-08-30. A sample budget alert from ag-plat-prod-cus-01 arrived in the ops-email receiver's inbox (fired 21:36 UTC; receiver status Enabled). The email path is proven end to end, across the subscription boundary, for the first time. Use the CLI, not the portal button. The portal's Test action group reported "There was a problem completing this test" with status Unknown on two attempts and delivered nothing. The same operation through the CLI returned "Status": "Succeeded" / "state": "Complete" and the email arrived a minute later: az monitor action-group test-notifications create \ --action-group ag-plat-prod-cus-01 \ --resource-group rg-mgmt-plat-prod-cus \ --subscription 00000000-0000-0000-0000-000000000003 \ --alert-type budget -a email ops-email
    usecommonalertschema The API response and the inbox are two independent witnesses, which is the standard the Cutover-Runbook asks for and what the portal alone could never supply — its Unknown was a verdict on nothing. Note also the argument shape: it is -a email <name> <address> <schema>, not a --email-receiver flag, and the whole thing is one line — a backtick continuation pasted into Git Bash gets read as a command name. Second, this file named the group ag-ops-prod-cus until the same day, which is no resource: observability.tf:36 builds ag-plat-${environment}-${region_abbreviation}-${instance}, hcw-ops is only the short name, and it lives in rg-mgmt-plat-prod-cus in the Management subscription because the action group follows its resource group there. The SMS channel was proven the same evening and T-709 is closed. Same command shape, -a sms <name> <countryCode> <phoneNumber> in place of -a email …; read the values off the action group rather than retyping them. Both channels have now been observed delivering, which is the first time this estate has had a second path to fall back on. approved a real external mutation test.
  • ~~Apply the Terraform change that creates the listenandlearn blob container.~~ Applied 2026-08-30. az storage container-rm show reports listenandlearn with publicAccess: None — private, as intended; PUBLIC_MEDIA_CONTAINERS in blob-paths.js is what makes an episode reachable, not the container ACL. The same apply declares the fallback AZURE_SPEECH_* settings, which stay unresolved and inert.

Accepted risks that closed

Risk Accepted Reasoning
~~The Static Web Apps deployment token is a Terraform output (swa_token)~~ Raised as T-722, 2026-08-28 CLOSED (T-727). The output was deleted on 2026-08-30 and the owner deleted the AZURE_STATIC_WEB_APPS_API_TOKEN secret on 2026-08-31; the deploy mints the token from ARM under federated identity, per run. No stored, non-expiring credential remains in this repository's secrets. Not an accepted risk any more, and kept in this table only so the row does not appear to have been quietly dropped The token is in state via azurerm_static_web_app.hcw.api_key whether or not the output exists, so deleting the output would hide it rather than retire it. sensitive keeps it out of logs and plan output; it is still visible on the HCP Terraform Outputs tab to anyone with state read. It is the estate's last long-lived credential — everything else a workflow uses is federated OIDC. Compensating control, 2026-08-28: deploy-azure-frontend.yml now isolates it in a job that installs nothing, so a compromised build dependency cannot reach it (T-727). Retiring it means moving the SWA deploy to OIDC, or at minimum making this an environment secret on a protected production; both need owner access. The outputs.tf header now names the exception instead of contradicting it
  • T-518 closed: all 18 timers are armed (#345). Owner decisions of 2026-09-05, in order: the content reaper's delete switch flipped after two dry-run firings read [cleanupSoftDeletedContent] dry-run: would delete content=0 (user=0, policy=0) refused=0 examined=0 (#343); Wave 3b armed the same evening, one apply on its own; waves 4, 5 and 6 armed together in the last apply, with Publer and both Plaud tokens seeded beforehand so the wave 5 timers do real work. The owner then decided that the per-wave observation read is no longer a gate: the arming apply closes a wave, and the reads stay available from 05-verify-timer.ps1 and the public-witness script for anyone who wants them. Wave 3a's planted test page, soft-deleted on 2026-09-05, will be reaped a week later and is the standing live check of the origin classification. The wave table, as it stood at close:
Wave Timers Record
1 PLATFORM_JOB_SWEEPER, MONITOR_PUBLISHING_PIPELINE Done 2026-09-02 (#323): all four gates observed
2 SYNC_RSS_FEEDS, FETCH_PODCAST_FEEDS Done 2026-09-04 (#332): both witnessed through the public side effect
3a CLEANUP_SOFT_DELETED_CONTENT Destructive, solo. Armed 2026-09-05; delete switch flipped in #343 after two dry-run firings reported zero across the board; a mark with no recorded origin is refused in both modes (#334)
3b CLEANUP_REJECTED_CONTENT Destructive, solo. Armed 2026-09-05 evening, one apply on its own; marks aged rejections soft-deleted, reversibly, and only 3a turns the mark into a deletion
4 FORGE_SCHEDULED, GENERATE_REVIEWER_DIGEST, FETCH_BLOG_LISTINGS Each spends money or sends mail; all three approved 2026-09-05 and armed with waves 5 and 6 in the last apply
5 REFRESH_PLAUD_TOKEN, SYNC_SOCIAL_CALENDAR Both need owner-held credentials; Publer's two Key Vault secrets and both Plaud tokens were seeded on 2026-09-05 (#342) before the apply
6 CHECK_LIVE_LINKS, REVERIFY_CERTIFICATIONS, SCRAPE_SKILLS_HUB_RSS, CLEANUP_UNUSED_CERT_IMAGES Three weekly timers on Monday, Sunday and Friday plus one daily timer still dry-run pinned by CERT_IMAGE_CLEANUP_DELETE; armed in the last apply

Waves 1 and 2 were observed through all four gates (#323, #332). The per-category host.json overrides that served as witnesses for 3a through 6 came out in this PR, returning host.json to the #321 state: Function at Warning, Host.Results at Information.

  • T-726 closed: the nightly refresh's App-token path is accepted as built (#345). Owner decision 2026-09-05. The App is installed, the workflow mints its token from it, and the ruleset admits the App; the first content change remains the live proof, and its two failure signatures — 401 for a wrong App id, 404 for an App not installed on the repository — are recorded here so the day it fails is diagnosable without the tracker.

  • host.json raises the nine remaining timer categories to Information ahead of the last arming apply (#344, T-518 waves 4, 5 and 6). Each of forgeScheduled, generateReviewerDigest, fetchBlogListings, refreshPlaudToken, syncSocialCalendarScheduled, checkLiveLinks, reVerifyCertifications, scrapeSkillsHubRss and cleanupUnusedCertImages gets the per-category override that was decided as its witness on 2026-09-05, so 05-verify-timer.ps1 reads real invocations the moment the owner adds the nine names to enabled_timers. Until that apply the only cost is a disabled — skipping row per tick, which for the five-minute calendar sync is a few hundred small rows a day against a workspace running well under its cap. Every override comes out again in the PR that closes its wave.

  • Wave 3a's delete switch is on: CONTENT_HARD_DELETE is true (#343, T-518). Owner decision 2026-09-05 after reading two dry-run firings (01:00Z and 05:00Z, both on their Chicago even-hour ticks) at [cleanupSoftDeletedContent] dry-run: would delete content=0 (user=0, policy=0) refused=0 examined=0: nothing has been soft-deleted for longer than the reaper's 7-day grace window, so the first armed run deletes nothing. The origin refusal from #334 holds in both modes. The live witness for the classification is a test page the owner soft-deleted the same day, which the reaper should delete a week later as user=1 refused=0 with an admin_audit_logs record; the wave closes on that read, not on the flip. host.json also gains the Function.cleanupRejectedContent override so Wave 3b has its witness the moment it is armed.

  • The Static Web App runs on the Free tier (#341). Owner decision 2026-09-05, closing the question carried in TODO.md since T-721 closed on 2026-09-02. Everything the estate uses is in Free — custom domains with managed SSL, global distribution, SPA routing, preview environments, 100 GB of bandwidth — and the two Free limits were verified from the repository before the change: exactly two custom domains (the apex and www, Free's maximum) and a build of about 117 MB against the 250 MB app limit. What Standard bought and nothing here used: the 99.95% SLA and bandwidth overage; past 100 GB a Free site stops serving instead of billing, which Cloudflare caching in front makes unlikely. About USD 9 a month, the workload's one fixed line. Microsoft documents the move in either direction, so a third domain or an overage is the signal to move back.

  • The container prevent_destroy guard is back on (#341). #340 lifted it for one apply to drop ai_insights; the owner read that plan at exactly one destroy plus the workspace's permanent azapi baseline and approved it on 2026-09-05. The guard comment now records the procedure for the next deliberate drop: lift in its own PR, read the plan for one destroy, restore in the next.

  • T-766 closed: every remaining timer wave has a stated witness (#338). Found 2026-09-03 when Wave 2's gates returned nothing: #321 had cut the Function log category to Warning, which removed the Executed and ScheduleStatus rows the timer-observation gate read, while its own record said it had protected the table the gate used. The owner kept the cut and made the side effect the witness (scripts/verify-timer-witness.mjs, #328), which observed Wave 2; the per-category host.json override was decided for Wave 3a on 2026-09-04 (#334). On 2026-09-05 the owner decided the remaining waves in one sitting — Wave 4's three timers all approved, Wave 5's two credentials both held, and waves 4, 5 and 6 to arm together in one apply after 3b, since none of the nine deletes anything — and each row in the T-518 table now names its witness: the override, added for a wave when it arms and removed when it closes. The item's closing condition was a stated witness per wave, so it is met; the per-wave work stays in the wave rows. Two records were corrected on the way: neither Wave 5 timer loops on a missing credential (both skip), and the delta-import ordering rule on SYNC_SOCIAL_CALENDAR is moot since the import was retired on 2026-08-24.

  • host.json raises one log category, Function.cleanupSoftDeletedContent, to Information for Wave 3a (#334). The T-766 decision for the first wave with no public witness. The reaper's side effect is an absence, so no container read can tell an idle run from a timer that never fired; the override restores that one timer's Executed rows at a few lines per invocation, which does not reopen T-719. It is a wave-scoped change and comes out when the wave closes. 05-verify-timer.ps1 now reads a per-category override before the Function default, so it works for this timer again.

  • T-719 and T-721 closed by owner decision, 2026-09-02. The verbosity cut merged (#321) and Deploy Functions was dispatched the same day; the owner closed both without gating on the first clean cap-day reading — the below-cap volume is expected confirmation, not a closure criterion, and the evidence standard is the owner's to set on a single-operator estate. Held in reserve if a later reading says the cut was not enough: move AppTraces (~38% of the old cap volume) to the Basic table plan at roughly USD 0.65/GB versus USD 2.76/GB. The accepted risk (log alerts sleep from cap-hit to the 08:00 UTC reset if the cap ever binds again) stays recorded in TODO.md's Accepted risks; the Static Web App tier re-justification that lived inside T-721 moved to TODO.md's Owner decisions table rather than closing silently with it. The tracker drops to two open items: T-726 (waits on the first content change) and T-518 (arm the timers).

  • The Functions host stops narrating itself: host.json drops default and Function to Warning (T-719 decided, T-721's lever pulled). Twelve days of data showed daily ingestion pinned at the 0.25 GB cap — the ceiling, not demand — with the log-based 5xx/latency alerts going dark from cap-hit to the 08:00 UTC reset on most days. The owner's decision, 2026-09-02: this is a personal content site whose volume is host verbosity, not traffic, so the source is cut instead of the cap being raised for a measurement day. Host.Results stays at Information deliberately — it feeds AppRequests, which those alerts and the timer-observation gate read, and the pinning test refuses the change that would empty it; Azure.Core/Azure.Identity were already pinned quiet after T-514. The cap stays at 0.25 GB as headroom. Accepted and recorded with the decision: if the cap ever binds again, log alerts sleep until the reset — the T-519 probe covers unreachability on a pipeline the cap cannot touch, and logs_daily_cap warns at 80% first. The closure criterion stated here when this shipped — one cap-day measurably below 0.25 — was superseded the same day by the owner's decision recorded above; the below-cap reading is confirmation, not a gate. The Basic-table-plan move stays in reserve if that reading says the cut was not enough.

  • The settings sweep is done — the tracker's Phase 2, three settings that were choices rather than defaults, all clicked by the owner 2026-09-02. The three stale Terraform Cloud variables (migration_writer_enabled, cosmos_scratch_enabled, storage_scratch_enabled) are deleted from the hcw-azure workspace — deletion, not declaration, per the 2026-08-24 removal record; the success criterion (a plan printing no "Value for undeclared variable" warnings) is observable on the next queued run. The production environment now carries the main-only deployment-branch rule, closing the path where the environment-scoped federated credential matched from any branch (T-705's other half; required reviewers stay deliberately unconfigured per the 2026-08-29 decision). And the two Default-ruleset booleans are decided: branches must be up to date before merging (on — a stale head re-proves itself against the moved base before it can land) and required thread resolution stays off. "Settings still worth a look" is removed from TODO.md and the attack sequence's Phase 2 is struck.

  • Every pull-request check now runs only when its component actually changed. The 14 checks stay — every context the ruleset requires still reports on every pull request — but ci.yml's six jobs and CodeQL's three analyses adopt the filtering-inside-the-job pattern iac-validate.yml proved under T-523 (a trigger-level paths: filter on a required context leaves the PR waiting on "Expected" forever, so the job itself diffs against the base and skips its expensive steps, still posting success). Each filter names the component's real dependency set, not just its own tree: functions tests read infra/ Terraform source, .azure/api-surface.json and the wiki/Blog-Machine.md grammar contract; scripts tests pin .github/workflows/ and infra/roles; CodeQL filters by language file extensions. A docs-only pull request drops from six installs, a frontend build and three CodeQL analyses to nine ~10-second no-ops; pushes to main and the weekly CodeQL schedule still run everything, so the merged combination and newly published queries are never skipped. The decision, its rejected alternatives and its accepted risks are ADR 0026; a public write-up of the pattern was staged as Blog-FinOps-01-CI-Refund (that draft has since moved to the owner's private vault).

  • The code-review skill moved to .github/skills/code-review/, and its consumer is now GitHub Copilot code review. Owner decision, 2026-09-01: Copilot reads agent skills only from .github/skills/, Claude Code only from .claude/skills/, and rather than keep two drifting copies the one canonical copy lives where the automatic PR reviewer finds it. The trade-off, accepted explicitly: Claude Code sessions no longer auto-load the skill (an agent can still be pointed at the files when asked to review). The structure validator's Markdown allowlist gains .github/skills/ with the reasoning inline; the #316 entry above describes the skill's content, whose location this entry supersedes.

  • The Static Web Apps deployment token is minted per run instead of stored (T-727, #296). The decision was "move the SWA deploy to OIDC", which cannot mean what it sounds like: Azure/static-web-apps-deploy cannot authenticate with a federated credential at all (azure/static-web-apps#1304 is an open request for exactly that). So the token stays and the storage goes. The deploy job asks ARM for it under the federated identity the repository already uses, masks it, and it lives for one run.

Retired: the swa_token Terraform output, visible on the HCP Terraform Outputs tab to anyone with state read, and the long-lived AZURE_STATIC_WEB_APPS_API_TOKEN GitHub secret, deleted by the owner on 2026-08-31. No stored, non-expiring credential remains in this repository's secrets. Not retired, and the header says so where the output used to be: the token still exists in Terraform state as an attribute of azurerm_static_web_app.hcw, which no output block could ever have changed.

It cost no new federated credential. The deploy job already declares environment: production and github_deploy already held the matching subject. Microsoft.Web/staticSites/listSecrets/action is an action rather than a read, so Reader cannot express it and the built-ins that carry it also grant write over the site; the custom HCW Static Web App Deployer role grants the single action, assigned to the one site.

  • Three documents said the hcw-azure workspace was CLI-driven with no VCS connection (#297). terraform -chdir=infra apply, which this repository instructed the owner to run, answers "Apply not allowed for workspaces with a VCS connection". TODO.md had been wrong about this twice in opposite directions.

The root cause is worth more than the correction. HCP Terraform's workspace Description is free text sitting beside the real settings, validated by nothing, and it read "CLI-driven; no VCS connection". That sentence is what was read — while the entry claimed it had been read off the configuration. A description that contradicts its own workspace reads exactly like a setting. Corrected in the workspace and in TODO.md, wiki/Cutover-Runbook.md and .github/workflows/tfc-plan-check.yml.

  • Repository trackers reconciled against merged main (2026-08-30). The TODO.md status date and self-count now agree with its August 30 evidence; its owner-action table records the two proven timers rather than saying nothing is armed; and its recovery baseline now names the implemented Continuous30Days Cosmos backup, RA-GRS content storage, and remaining LRS Functions host storage. GitHub issues #127, #180 and #231 were re-read against current Terraform and .azure/api-surface.json: their unfinished decisions remain open, but superseded file anchors, endpoint counts and recovery baselines no longer describe old code as current state.

The live Default branch ruleset was also read through GitHub on 2026-08-30, rather than inferred from workflow files: it targets ~DEFAULT_BRANCH, blocks deletion and non-fast-forward updates, requires pull requests and all 12 documented status contexts, and has no bypass actors. Its two remaining hardening choices — strict/up-to-date status checks and required review-thread resolution — are both currently off and are recorded in TODO.md for an owner decision. The production environment's branch restriction could not be read through the integration, so that existing owner verification remains open instead of being reported as proven.

  • The Telegram webhook is registered against Azure, and the tracker was wrong about it (T-526 closed, 2026-08-28). getWebhookInfo returns https://api-azure.hybridcloudworks.com/api/telegram/webhook with zero pending updates, and /help answers in the chat — the acceptance criterion TODO.md set for itself. The Phase 5 approve-by-reply loop is live, which makes the whole Blog Machine Telegram path usable: a forge_ready notification with a signed staging link, /approve {id} as a reply, and publication through the full gated pipeline. The inline approve/reject buttons deferred to the backlog are unblocked by the same fact.

Nothing was re-run to close this. It had already been done, and three documents went on describing it as pending: TODO.md called it "the one deadline on this list" and a countdown against the GCP deletion, Migration-Plan.md §6 step 6 marked it OPEN, and its risk register read "High — now a deadline". It surfaced only because 04-telegram-webhook.ps1 -Mode Show was run to start the work.

That step's own text warned it was "the one that will be forgotten". It was right about the failure mode and wrong about the state — what got forgotten was that it had happened. A tracker overstating urgency is not harmless: it spends attention on finished work and, by being wrong in the direction of alarm, teaches a reader to discount the next alarm.

Three defects in the cutover scripts were found by running them, none of which reading had caught. Every script that opens a firewall window addressed the vault as az keyvault network-rule add --name <vault> with no --resource-group, which makes az resolve the group by searching the subscription; that search fails where az keyvault show --name succeeds, and reports The Vault 'kv-site-prod-cus-01' not found within subscription — which reads as "the vault does not exist". Name plus resource group is an unambiguous address with no search step. The vault read then failed ForbiddenByRbac, because management-plane and data-plane rights are separate and this estate deliberately grants the operator only the first (REVIEW.md §4.6 records exactly that response and calls it the correct posture) — so 06-seed-secret.ps1 now checks for a data-plane role at the management plane before prompting for a credential and before opening anything, rather than discovering it at the write.

  • The four planning documents are reconciled to their own conventions (2026-08-28). Two rules, applied consistently for the first time. TODO.md and REVIEW.md carry only open work — completed items are removed once the entry is here, which is TODO.md's own stated footer rule and which the file had stopped obeying: it had accumulated **Closed:** enumerations for the architecture review's 35 resolved findings and a "Critical — CLOSED" section, so a reader could no longer tell what was outstanding without reading past a page of finished work. REVIEW.md shed the executed migration-era teardown, the two closed live-confirmation bullets and the completed apex-DNS row for the same reason. Architecture-Plan.md and Migration-Plan.md keep every entry and strike it through — they are archived records whose value is the reasoning, so deleting a decision would delete why it was made.

Four things surfaced from doing it rather than being the point of it, and the largest is the one this rule is for: six merged pull requests had no changelog entry at all — #234's availability probe and #243–#248's post-program backlog work. The removal pass caught them because removing an item from TODO.md requires checking that CHANGELOG.md has it, and that check had been skipped while the completed items were merely being marked closed in place. Both entries are above. The open-item counts in TODO.md were also wrong — Low read "5 of 15 closed" against a list of seven, and the total was 32 against a real 30. Migration-Plan §6's rollback paragraph still promised "rollback is DNS for as long as Firebase remains deployed", which the owner's decision to delete GCP rather than soak had already reversed; it now records the inversion, because that decision is exactly what puts a deadline on T-526. And two of the plan's eight verification gates — the scheduled-job proof and the cost gate — turn out to be the only unmet exit criteria of the entire migration, which was not legible while they sat in a list whose met items were unmarked.

  • The apex serves the Azure site (T-517 closed, 2026-08-28). The cutover this repository was built toward is done: hybridcloudworks.com — the canonical hostname, the one host that was still Firebase — now resolves to calm-ground-0d0e6a010.7.azurestaticapps.net and serves the Static Web App. Evidence, in the order it arrived: the owner-supplied Cloudflare zone export of 2026-08-27 23:47 showed the apex CNAME at the SWA with no Firebase record remaining anywhere in the zone (the Runbook §3c repoint); the owner then verified serving on 2026-08-28 — the acceptance criterion this tracker holds cutovers to, because DNS state is desired state and T-513 is the recorded case where the two disagreed.

Two owner decisions recorded with the close-out. First, the DNS rollback is forgone: rather than holding the Firebase deployment through the Runbook's one-week soak, GCP is scheduled for deletion. Second, that decision converts the Telegram webhook re-registration (Runbook §3d) from a dormant follow-through into a deadline — the bot's webhook still points at the old Cloud Functions URL, and once GCP is deleted it goes quiet with no error anywhere in Azure. Tracked as T-526, to be run before the deletion.

The Runbook §3c soak criterion ("a full week including every scheduled job") could never have completed as written — no timer is armed (T-518) — so the owner's decision also resolves a dead-lock this tracker had flagged between the two items.

  • The two data-migration federated credentials are retired (T-524 closed, 2026-08-26). infra/oidc.tf trusted six OIDC subjects and now trusts four. The pair granted no permission of its own — a federated credential decides which subject may act as the deploy identity — and with the production-write grants already revoked, a data-migration token inherited the same reduced role set a branch token gets. What it removed was a standing trust relationship for a job that cannot run.

Held back from the earlier cleanup deliberately: retiring a trust relationship is an identity change rather than a Terraform tidy-up, which is why the remediation branch escalated it instead of deleting it. The owner authorised it on 2026-08-26.

Validated before the change, not after. Nothing in .github/workflows names that environment; its only consumer, migrate-data.yml, was deleted in 59e471b. Of the four workflows that call azure/login, one declares environment: production and three declare none, so they present the ref form — deploy-azure-frontend.yml names an environment but deploys through Azure/static-web-apps-deploy and never logs into Azure at all.

scripts/oidc-subjects.test.mjs was then run against three variants of the deletion, because the check that matters is the one that fails:

Variant Guard
Both credentials removed passes
Only the name form removed fails — no immutable-ID-form credential
Branch pair swept up with them fails — names all three ref-form workflows

Both forms went together for the reason the second row states: one without the other is half a credential and fails on whichever form the token happens to carry. If a migration workflow is ever rebuilt it needs both back.

  • The migration-era rehearsal estate is destroyed and the three production-write grants are revoked (B6/B7, applied 2026-08-25). The apply reported 3 added, 2 changed, 92 destroyed. The destroy count matched the authorisation in REVIEW.md exactly — 90 real destroys plus the 2 azapi resources replaced on every apply — which is the number that mattered, since the record insisted on approving against addresses rather than a count. The adds and changes came in below the recorded 17/5, and that is not a short apply: that figure was written before any of it ran, most of those adds were the alert rules, and #218 and #219 had already created them (ten of thirteen targeted resources, then the remaining three after ARM rejected them at create time). By the time this run planned, they were no longer adds. Everything was verified after the apply rather than inferred from the plan: rg-db-site-sbx-cus no longer exists; all four alert rules are still present and enabled; and the deploy identity is down to four operational roles — HCW Cosmos Container Definition Writer on the production account, Storage Account Contributor and Storage Blob Data Contributor on stsitefuncprodcus01 (the Functions host account, needed for the deploy firewall window), and Website Contributor on the Function App. The three revoked grants were scoped to dbs/hcw and to stsiteprodcus01, the content account; none of them appears. What this cost is already recorded: with the grants gone the deploy identity has no write path into the production Cosmos database or the content account, so the delta import is retired for good. The two data-migration federated credentials survive — federated_subjects still emits environment:data-migration twice — and remain an owner decision (T-524).

  • REVIEW.md §4.10 listed the wrong number of Terraform outputs. It said twenty-three and omitted deploy_principal_id, because it was assembled by reading infra/outputs.tf alone while three outputs live in infra/oidc.tf. There are twenty-four, and the section now lists them by file. Corrected against the apply's own output block, which is the only listing guaranteed to be complete. Recorded rather than quietly fixed because it is precisely the drift Part 4 exists to prevent, and it was introduced by the change that restored Part 4 four commits earlier.

  • The IaC checks are required to merge, and the three orphaned repository variables are gone (T-523 owner half, T-525, 2026-08-25). Ruleset 20680114 now requires 12 contexts rather than 10, the two additions being fmt, validate, tflint and Trivy IaC misconfiguration scan. Every "CI enforces this" line in CONTRIBUTING is now true when nobody is watching, where before a branch with a red Trivy run merged exactly as easily as one with a green run. The ordering the item insisted on held: #220 removed the paths: filter from the pull_request trigger first, and PR #221 — which touches only tooling/agent-registry.yml and no infrastructure at all — then confirmed the skip path end to end, its fmt, validate, tflint job reporting green in 22s with Detect infra changes succeeded and all six Terraform steps skipped. Had the ruleset been changed first, that same pull request would have been unmergeable. COSMOS_SCRATCH_ENDPOINT, STORAGE_SCRATCH_ACCOUNT and SCRATCH_RESOURCE_GROUP were deleted in the same pass; gh variable list now returns 20 names and REVIEW.md §4.2 lists exactly those 20, so the inventory and the live repository agree with no residue to reconcile. The variables named resources that will stop existing when the rehearsal teardown applies, and a value nothing reads is a value the next person assumes is load-bearing.

  • The frontend is on ESLint 10 (D-001 closed). The item said two plugins blocked it, on the strength of their declared peer ranges. That was half right and the wrong half was load-bearing, so it is worth recording what the block actually was.

Dependabot's bump failed at npm ci, not at lint — an ERESOLVE refusal from eslint-plugin-jsx-a11y's eslint@"…|| ^9" peer range. That is metadata, and npm overrides pinning both plugins' eslint peer to $eslint clears it. What remained was one real incompatibility: every rule that consults the React version died with contextOrFilename.getFilename is not a function, because ESLint 10 removed context.getFilename() and eslint-plugin-react calls it while DETECTING the React version. Detection only runs when settings.react.version is the literal 'detect', so supplying the version skips the removed API entirely. The config now reads it from the installed react/package.json rather than pinning a literal, so an upgrade cannot leave the linter reasoning about the wrong React.

Verified beyond a green run, because a plugin that silently loaded no rules would also look green: a probe file confirmed react/jsx-key, react/no-unescaped-entities, jsx-a11y/alt-text and react-hooks/rules-of-hooks all still report on ESLint 10, and npm ci — the command that actually failed — now succeeds.

One rule stays off, for a new reason. jsx-a11y/label-has-associated-control was disabled because it crashed on ESLint 9; on 10 it runs and reports 20 real unlabelled form controls. That is an accessibility fix rather than an upgrade, so it is tracked as A-001 and the config comment now says so — the stale one would have told the next reader the rule was unusable. - Listen & Learn spend appears in the portal, and ai_usage has one writer. The Usage tab has read that container since the port; until now only the AI playground wrote to it, so a Listen & Learn run — the second thing here that spends money on a model — would have been invisible. Each run now records a row per model call: one for the script, one for the synthesis, tagged listen-and-learn:script and listen-and-learn:audio.

The writer moved into ai/usage.js and ai/proxy.js now uses it, because the Usage tab does its arithmetic client-side over whatever rows it finds — a second writer with a slightly different shape would not error, it would silently total zero. Recording is best-effort by design, and a test pins the regression that made it otherwise: pricing a row used to happen outside the try, so a caller passing an ai without getCostEstimate threw a TypeError that propagated out and failed the episode whose cost it was recording.

TTS rates are in COST_TABLE from the published paid-tier pricing read on 2026-08-24 — gemini-2.5-flash-preview-tts at $0.50 in / $10.00 out per 1M tokens, the other two at double that, which is why the flash model is the default. Token counts come from the API's own usage object; when a response omits it the audio count is derived from duration at the documented 32 tokens/second and the row is flagged estimatedTokens, which the portal shows as "est." so a derived figure is never read as a billed one. On those rates a nine-minute episode is about $0.17 and a five-area certification about $0.87.

The tab gains a Breakdown by Feature table beside the provider one: provider answers "which vendor", which is useless when one vendor serves several features at rates an order of magnitude apart. The Listen & Learn page also reports the run's own cost when the job finishes. A test holds the tab's source labels against the backend's USAGE_SOURCES so a new source cannot ship as a raw slug — the same drift guard DEFAULT_PROVIDERS already has. - QueuePage.jsx is decomposed (T-412). 1,310 lines became a 320-line page over queue/itemHelpers.jsx, queue/QueueList.jsx, queue/constants.js and queue/useQueueActions.js. The hook is the reason for the split: the bulk paths transition many documents one at a time and each partial failure has to be attributed back to its own card, and that code could previously only be reached by rendering four hundred lines of card markup. It now has 22 tests covering the partial-failure paths — a run that half-works removes exactly the documents that moved, leaves the ones that did not, and writes a reason under each — plus the paging loop's zero-count guard and the rejected-filter refusal. Behaviour is unchanged with one fix found by the move: handleConfirm was useCallback(..., [confirmTarget]) while closing over handlers rebuilt every render, so it could act on items and selectedIds as they were when the modal opened. It is no longer memoized; the dependencies changed every render regardless, so nothing was gained by it. - Publicly readable blob containers now declare their writer. PUBLIC_MEDIA_CONTAINERS ⊂ UPLOAD_CONTAINERS held only because every public container happened to be one people upload to. Listen & Learn audio is written by a job, so GENERATED_MEDIA_CONTAINERS names that category and the test asserts each public container has exactly one declared writer and that the two sets are disjoint. Satisfying the old relation would have meant opening the episode container to the admin upload route, where any editor could put an arbitrary file behind an anonymous URL.

  • Every backend dependency has a live consumer (T-407); nothing was removed. The item asked whether cheerio, rss-parser, google-auth-library and the other non-route packages in functions/package.json still had one. All of them do, and each consumer is reachable from a registered function: cheerio from cms/content-quality.js, content/scrape.js, rss/feeds.js and sanitize-html.js; rss-parser from rss/ingest.js (the fetch-rss-feeds job and the syncRssFeeds timer) and timers/podcasts.js; google-auth-library from cloud-tools/pricing/gcp.js via the pricing index; and turndown, jsonwebtoken, jwks-rsa, @aws-sdk/client-pricing and the four Azure SDK packages from token verification, scraping, pricing, Key Vault, Blob and Cosmos. Recorded rather than closed silently, because "no packages were removed" is the finding.
  • The remaining upstream feature candidates are evaluated (T-410). Measured against the Site-Main checkout at 088f458, the same baseline the T-409 delta used: draw.io hotspot tooling is the one candidate worth porting. lib/drawio/parseDrawio.js and lib/drawio/hotspotGeometry.js are 258 lines of pure client-side XML parsing with no Firebase coupling and an upstream test and fixture; DiagramPanel's only backend seam is an image upload, which maps onto the existing POST /api/cms/uploads/{container}. It replaces manual hotspot authoring — today ArchitectureReviewBoard requires each hotspot's coordinates and label to be typed by hand — with generation from an uploaded .drawio file, and InteractiveDiagram already consumes the resulting shape. Admin queue improvements split in two. The bulk select, bulk reject, bulk delete and confirm-modal paths already exist in this repository's 1,310-line QueuePage.jsx; the upstream delta is a decomposition into five modules with tests, plus two additional actions (bulkApprove, bulkForge). The decomposition is worth doing against this repository's own file rather than porting upstream's, which is written against Firestore-era helpers. The Architecture listing pages are not worth porting as they stand. ArchitectureDesignsPage and ArchitectureCreatePage are 163 lines between them, but they are thin wrappers over ContentReviewBrowser (744 lines) and the components/admin/browser/ subsystem, useAdminBrowser and lib/adminBrowser — roughly 1,500 further lines — and their data seam, fetchContentList, is built from Firestore where() clauses. That is the whole-branch shape T-410 was written to refuse, and EditorListPage already filters admin content by type, architecture included. Which candidate is actually built is a product decision and now sits in TODO.md.

  • The ESLint 10 upgrade is still blocked, and by fewer plugins (D-001). Re-checked against the registry on 2026-08-24: eslint-plugin-react-hooks 7.1.1 and @typescript-eslint/eslint-plugin 8.67.0 now declare eslint@^10. eslint-plugin-react 7.37.5 still caps its peer range at ^9.7 and eslint-plugin-jsx-a11y 6.10.2 at ^9, so the frontend stays on the ESLint 9 line.

  • Retired the completed migration surface and reset the repository around the HybridCloudWorks website. Removed the old Firebase Functions package and Labs agent, the completed Firestore/GCS migration workflow and one-shot import tooling, the disabled infrastructure-delivery workflow, and unused Firebase hosting, scaffolding, screenshot, generator, and debug scripts. Retained the Azure container specification because Terraform still consumes it, and retained active Azure operations, CI, deployment, and smoke tooling.

  • Reconciled repository documentation. README.md now documents the website's features, architecture, local development, and delivery model; TODO.md contains only engineer-resolvable pending work; REVIEW.md contains only human-owned access, approval, credential, and live-verification items; and both plans are archived records rather than active instructions.
  • Moved Wiki-as-code to wiki/. The sync workflow and repository policy now use the root wiki/ staging directory, and the Wiki home/sidebar point at current website and Azure documentation. All repository workflows use GitHub-hosted runners.
  • Removed remaining active cleanup residue. Deleted the unused Functions parity contract and duplicate Markdown bug template, updated current infrastructure metadata and operator helpers to reference the website state, and allowlisted the repository's reusable .github/templates/ directory.

  • Retired the unused frontend Firebase platform surface. Removed the old frontend/firebase.json, Firebase rules/indexes/storage files, GCP Terraform configuration, and dangling test/lint exclusions; the current frontend package now validates only the live src/ tree.

  • migrate-data.yml rewritten. Dispatch-only; id-token: write; environment: data-migration; modes preflight | inventory-gate | export-dry-run | rehearse | verify | storage-inventory | storage-rehearse with target ∈ scratch (default) | production and a hard refusal of write modes against production. Step order is a correctness constraint: npm ci, the Site-Main checkout and the Cosmos probe all run before google-github-actions/auth, because the GitHub OIDC token it exchanges lives five minutes. Per-run storage firewall window with always() cleanup, mirroring deploy-functions.yml. COSMOS_DATABASE: hcw. Inputs reach the shell through env, never interpolated into run:.

  • COSMOS_ENDPOINT is a repository variable, not a secret. It is a public URL and a non-sensitive Terraform output; as a secret it was masked in logs and unverifiable in the UI. set-github-variables.ps1 now seeds it as a variable and deletes the old secret; both consuming workflows read vars.COSMOS_ENDPOINT. The script also takes -GcpWorkloadIdentityProvider / -GcpServiceAccount for the two WIF identifiers, and seeds STORAGE_ACCOUNT / STORAGE_RESOURCE_GROUP.
  • Migration scripts share one credential path. scripts/lib/cli.mjs gains connectFirestore() (ADC, explicit projectId), connectCosmos() (DefaultAzureCredential only), connectBlob(), classifyCosmosError(), writeReport() (full report + publishable .summary.json sibling) and showSamples(); the migrator, preflight and verifier use them. The manifest is re-baselined at Site-Main 088f458 with azure_architectures and azure_frameworks added as probe — not provisioned, so the generated container spec is unchanged.
  • Migration-Plan.md rebaselined against Site-Main @ 088f458. §0 is now donor/recipient with a pinned baseline instead of "reconcile weekly" (the two repositories finished Phase 1 in incompatible directions); §2 carries real status; §4 carries the measured inventories — the six HTTP handlers over the 230 s Flex cap, the 16 timers with NCRONTAB and zone, the 11 triggers with change-feed disposition and the three delete paths the feed cannot deliver, and the Vertex-default finding; §5 is rewritten around the tooling defects, the rehearsal estate, the five dispositions, the storage manifest and the public-repository rule; §6–§9 updated to match. The two links to the wrong GitHub org are gone.
  • Every image render site routes through resolveMediaUrl() (T-318, sixteen files, commit 09154ad). Stored site-relative /api/public/media/... paths now resolve against the Cloudflare API host, which the origin lock made the only working shape; absolute legacy URLs pass through untouched.
  • oidc.tf's "deliberately NOT granted" note now says what is true: the migration does use the deploy identity, on the scratch account at database scope, and holds nothing extra on production while migration_writer_enabled is off.

  • Every Terraform output renamed to the 2-word standard (workload owner directive, 2026-08-18: github_deploy_client_id was four words). The standard now explicitly covers outputs — they are operator-facing, read off the state backend's Outputs tab — and states that casing follows the language while the word count does not: UPPER_SNAKE for GitHub variables, lower_snake for HCL. Outputs that feed a GitHub variable now mirror it: client_id ↔ CLIENT_ID. Headline renames: github_deploy_client_id→client_id, github_deploy_federated_subjects→federated_subjects, function_app_default_hostname→function_hostname, static_web_app_default_hostname→swa_hostname, app_insights_connection_string→insights_connection, ci_runner_job_name→runner_job. Two genuine duplicates removed: azure_functions_hostname and azure_swa_hostname returned values identical to their non-prefixed twins and were folded into one output each. One genuine collision resolved with a deliberate third word — the Function App and the deploy identity both expose a principal id, so app_principal_id / deploy_principal_id. No resource address, azurerm_* argument, or state-bearing name changed; terraform fmt and validate pass. Terraform input variables were deliberately NOT renamed: they must match HCP Terraform workspace keys exactly and several are set live, so they are a coordinated setting-plus-code change — filed as TODO T-507 with the full proposed table. App settings read via process.env, VITE_* and GITHUB_TOKEN are contractual and untouched. The iac-repo-standardizer agent now sweeps every .tf file rather than a curated list — the gap that let ci_runner_job_name survive the first pass. (PR #117)

  • deploy-infra.yml rewritten while remaining hard-disabled — the prototype workflow applied with -auto-approve on every push to main, masked failed plans with continue-on-error, and used unpinned actions. The replacement is workflow_dispatch-only, runs in a production-infra GitHub Environment for required-reviewer approval, starts an HCP Terraform run whose apply is confirmed in TFC where the state lives, and keeps the if: ${{ false }} gate until production applies are authorized.

  • Self-healing computed properties — .github/workflows/ heal-computed-properties.yml re-applies cp_sortDate on any push touching the Cosmos Terraform or container manifest, and every six hours — because Terraform applies run in TF Cloud on their own clock, a push-time heal can itself be overwritten, so the schedule is what guarantees the wound closes. With PUBLIC_LIST_SQL_ORDER=1 live, a wiped property breaks the public content list, which is why this is automation rather than a runbook note. The OIDC deploy identity gains Cosmos Data Contributor scoped to exactly the content and blogs containers (infra/oidc.tf) — the one documented exception to its deliberate no-Cosmos posture, and the healer fails loudly on a schedule until that assignment is applied. (TODO.md T-206 follow-up)

  • cp_sortDate computed property + flag-gated ORDER BY — T-206's last step, authored as operator tooling. scripts/apply-computed-sortdate.mjs --inspect reports non-ISO date values (the evidence gate), --apply adds a computed property that resolves the five published-date aliases server-side with a total fallback, and PUBLIC_LIST_SQL_ORDER=1 then makes the public list's TOP window return the newest N documents instead of an arbitrary N. A computed property rather than a materialized field: no backfill, no write-site maintenance, and it cannot be missing — which is what makes ORDER BY on it safe under the module's own rule 2. The azurerm provider cannot express computed properties, so the script is the applier and the manifest records the drift hazard: a terraform apply that updates the container wipes the property. Applied to the live containers and flipped on 2026-08-14; the deployed smoke test passed against the ordered window, closing T-206 entirely. (TODO.md T-206)

  • Deployed smoke test — scripts/smoke-deployed.mjs, the runnable half of the work order's top item. Tier 1 exercises the anonymous surface with no side effects: the public filter and T-206 projection (asserting the eight excluded body fields stay excluded and explanation does not false-alarm), guard liveness on admin RPCs, CORS refusal and preflight, negative-cache headers, the health endpoint's non-disclosure, and that the seventeen notImplemented RPCs still 404. Tier 2 (--cosmos) executes the one assumption nothing has executed: that a failed Cosmos patch predicate surfaces through the JS SDK as code 412 and a missing document as 404 — the submission quota's correctness rests on it; it writes a single smoke-prefixed document into the TTL-bounded submission_quota container and deletes it. Tier 3 (SMOKE_BEARER_TOKEN) verifies a real token is admitted. Six unit tests pin the script's own assertion helpers, because a smoke test with a wrong filter passes against a broken deployment.

  • Anonymous public read API — GET public/content, public/content/{slugOrId}, public/snapshots/{id}, public/podcasts, public/feed. The published/draft boundary is enforced server-side, replacing the Firestore security rules that previously performed that role. (#45)

  • Rate-limited public submission endpoint — POST public/submissions with per-type validation, server-side document composition, and a rolling-hour anonymous quota, closing the unauthenticated addDoc-into-content path. (#45, #66)
  • Admin CMS REST surface — certifications, social posts, recordings, speaker events, settings, images, AI providers / MCP servers, and usage records under cms/*, all behind the two-gate role guard. (#46, #47)
  • Authenticated file upload endpoint — POST cms/uploads/{container} with a container allowlist, blob-path validation, and a server-enforced 15 MB decoded cap, replacing direct browser writes to Firebase Storage. (#62, #65)
  • Content pipeline RPCs — createContentItem, updateContentItem, transitionContentStatus, and the publish pipeline, ported with their original dedup, quality-gate, state-machine, and audit semantics. (#43, #44, #59)
  • Admin identity, snapshots, ops health, content workflow, gallery, labs, and image-prompt RPCs — 34 named RPCs total. (#50, #54, #55, #56, #57, #58)
  • getLabJob RPC — single lab job with output, replacing the Labs console's per-document realtime subscription. (#65)
  • GET public/platform-health — the landing page's four cloud-status indicators, ported from the Firebase original. Anonymous, with a five-minute cache that is the only thing bounding how hard the route can be made to hit four third-party status APIs; each provider degrades to UNKNOWN independently and the handler never returns 500, because a dead upstream must not blank the panel. Ported without adding a dependency — axios and rss-parser stay unreachable. (TODO.md T-316)
  • POST cms/telemetry/legacy-blogs-read — the counter that will justify retiring the blogs fallback container. Guarded at viewer, unlike the anonymous Firebase original: its only caller is an admin page, so anonymity bought nothing and left an unauthenticated write endpoint anyone could use to poison the evidence. (TODO.md T-316)
  • Anonymous media delivery — GET public/media/{container}/{*blobPath}, serving uploaded images through the Function App's managed identity with immutable cache headers and conditional-request support. The storage account stays closed to the internet; the container allowlist is a strict subset of the containers uploads may write to. (TODO.md T-105)
  • Self-hosted CI runner — Azure Container Apps Job with KEDA scale-to-zero, an ephemeral JIT-config runner image published to Docker Hub with a GHCR mirror, and a CI_RUNNER repository-variable failover switch. (#48)
  • Labs agent API — POST agent/claimLabJob, agent/heartbeat, agent/completeLabJob, behind a machine-identity guard (LabAgent App Role plus a lab_agents/{agentId} registry document bound to the credential's object id) that is disjoint from the admin role hierarchy. Claim atomicity is an ETag-guarded write with a lease, so a dead agent's jobs are picked up rather than stranded. (TODO.md T-401)
  • code-reviewer agent — carries the Code Review SOP (CODE_REVIEW_PROMPT.md v1.0) as agent 39 of the harness. (#68)

  • SOP working documents — TODO.md, CHECKLIST.md, CHANGELOG.md.

  • Firebase-era smoke scripts and nested workflows removed. Three live smoke scripts read VITE_GCP_FUNCTIONS_URL and built a firebaseConfig from VITE_FIREBASE_* — none of which the application sets any more, so they could not run — and frontend/.github/ held the source repository's Firebase deploy, E2E and secret-rotation workflows, inert but reading as live configuration. Deleted rather than ported: a half-migrated script that looks runnable and is not is worse than no script, which is exactly what these were. A deployed smoke test is still wanted, written against Entra and the Azure routes. (TODO.md T-317)

  • Six unused dependencies dropped from the functions package — sharp, replicate, turndown, @mendable/firecrawl-js, axios and rss-parser, none of them referenced anywhere under src/. (TODO.md T-407)
  • Frontend decoupled from Firebase. All 34 files importing firebase/firestore, 5 importing firebase/auth, and 4 importing firebase/storage now call the Azure Functions API. Public pages (#61), admin CRUD (#62), shared config libraries (#63), workflow pages and the editor (#64), remaining admin pages (#65), and submission forms (#66). The production bundle no longer contains a Firebase chunk.
  • Admin authentication swapped to Entra ID via MSAL — firebase/auth eliminated from the admin surface; MFA is now an Entra Conditional Access policy rather than app-managed phone MFA; the Entra object id is the admins/{oid} registry key. (#60)
  • Realtime listeners replaced with polling — the content editor polls its document every 20 s, the Labs dashboard polls a snapshot RPC every 15 s, and the Labs console polls an active job every 5 s. Conflict detection and online/offline semantics are preserved. (#64, #65)
  • Review.md renamed to REVIEW.md and its scope narrowed to human-resolvable blockers, per the SOP.
  • Repository structure policy (scripts/validate-repository-structure.ps1) now requires the five SOP documents, permits them at the root, and rejects case variants of their filenames.

Removed

  • The Perplexity key, which nothing read (#1029). Its Function App setting (PERPLEXITY_API_KEY, a Key Vault reference), its secret catalogue entry, its Integrations card and directory row, and its service docs page are gone; the AI router never implemented Perplexity. Kept on purpose: the deprecated-provider lists that clear old AI Engine rows, the pricing table that prices historical ai_usage rows, and the badge and emoji that label them. The tests that used Perplexity as their example of an untestable card now use Azure AI Speech. The hcw-azure run drops the app setting; the secret, and three more that nothing reads (GITHUB-APP-INSTALLATION-ID, HOSTINGER-API-TOKEN, GITHUB-APP-PRIVATE-KEY), are the owner's to delete afterwards, one line each in docs/standards/variables-and-secrets.md. The same pass corrects three records: Caddy's CLOUDFLARE_API_TOKEN is set on the lab host (read by name, and proven by the *.lab wildcard certificate only DNS-01 can issue); the hcw-lab workspace token stays missing because that workspace does not exist yet; and COPILOT_REVIEW_APP_PRIVATE_KEY is still absent, which costs Copilot code review nothing.

  • Unused files, scripts and tooling removed in a repository cleanup. Every removal was checked from origin/main: nothing imports, fetches, runs or links to it. The frontend build, all 1,703 frontend tests, lint, the format check and the three validators pass without them.

  • 636 unused font files (about 66 MB of frontend/public/fonts). index.css loads 14 of 650. The rest were never requested, including all of Aptos and Genos: --font-aptos names the family but no @font-face loads it. The OFL licences beside Goldman and Turret Road stay.
  • 20 public data files nothing fetched: frontend/public/frameworks/*.json (frameworks come from the API) and frontend/public/finops/architectures.json with its one diagram.
  • 11 orphaned components and helpers:
    • Firebase-era lib/errorHandler.js;
    • ui/alert, ui/alert-dialog and ui/scroll-area, with their two Radix packages;
    • news/ArticlesBentoGrid;
    • the ContentHubTemplate, PatternLayouts and LandingPageTemplate templates;
    • performance/LazyImage and performance/SuspenseBoundary, with their index. Skeleton stays.
  • Five Firebase-era admin smoke scripts and their smoke:* npm scripts. They waited for Firebase Auth state and a heading that no longer exist.
  • validate-keyboard-nav.js, which checked a page and route that no longer exist. a11y:audit now runs the contrast scan.
  • Two a11y:fix-* npm scripts that pointed at files that were never committed.
  • Husky, commitlint and lint-staged. The hooks were never installed: there was no prepare script, and .husky sat in frontend/ while .git is at the root.

  • Wrong paths fixed in the docs:

    • the code-review skill's frontend reference (script paths and the keyboard check);
    • ADR 0030 (content/digest.js, removed in #541);
    • the migration plan (infra/scratch.tf still exists, as the removal record);
    • the CommonJS marker in frontend/scripts/package.json.

  • useAuthRedirectLanding is gone, and App.jsx is guarded against its return (#531). The hook was mounted on every route in the application — including every anonymous visit to a provider news page — so that a regex could watch window.location.hash for an auth fragment that appeared on one.

It existed because redirectUri was window.location.origin, so Entra returned the browser to the public home page, which had never touched MSAL. #520 gave sign-in its own route, and the API registration's bare-origin redirect URIs were cleared on 2026-09-12 — so a fragment can no longer land on / at all. It was deliberately kept for one release after #520 rather than removed alongside it, because a browser mid-redirect at cutover still landed there.

App.jsx joins the entry points in msal-not-on-public-routes.test.js. It is on every route there is, so an auth import there is an auth import everywhere, which is precisely what the hook was. It passes because every page is lazyPage(() => import(...)) and that walk follows static imports only — the distinction that matters, since a lazy admin page costs a public visitor nothing and a static one costs them 236 kB. Verified to have teeth by planting a static entraAuth import and watching it fail.

The 2026-08-23 incident the hook documented lives on in AuthCallbackPage.jsx, so nothing is lost with it.

  • Three history pages that showed a visitor nothing (#506). smoke-test-signoff.md was a blank form: 37 rows, every value column empty, no run ever recorded. It is not evidence, it is the shape evidence would have taken, and the real cutover evidence is in the migration runbook and the phase-4 record. implementation-plan.md shared 10 of its 10 headings with implementation-todo.md. legacy-azure-migration-task-tracker.md was the earliest of the three checklists, superseded by both, still asking for an Azure OpenAI resource retired on 2026-08-19.

Runbooks were reviewed and all six are published unchanged, with their real resource names — owner decision, 2026-09-11. The names are public by construction: the Function App answers on its azurewebsites.net hostname, the repository is public, and infra/variables.tf carries the defaults. The reasoning in those pages is also inseparable from the names.

  • The 80-page Firebase-era archive is deleted from the documentation site (#505). It was 54% of the published corpus, it described a platform this workload stopped running on in August 2026, and ADR 0027 had recorded that pruning it was a separate decision. This is that decision.

The measurement is what settled it. Of the 79 content pages, 78 were reachable from nothing — no ADR, runbook, standard, architecture page or history page linked to any of them. The archive index's own justification, that "several ADRs are only legible next to them", was false when checked page by page: the single apparent match was an ADR quoting a former file path that happens to share a filename, not a link. mkdocs build --strict fails on a broken internal link and passes after the deletion, which is the same fact proved a second way.

It was also where every piece of noise lived. Every mention of a password and every contact address on the whole site was in archive/ and nowhere else, all of it in templates and shell examples for the retired platform. The redaction gate now reads 71 files instead of 151.

Deleted rather than hidden, because a page that is published, unmaintained, describes a retired platform and is linked from nothing is not history a reader benefits from — it is noise competing with the 68 pages that describe the system as it is. Git history keeps every one of them, which is where evidence of a retired platform belongs. What a reader loses is nothing they could previously reach by following a link: the Firebase era is still described where it is load-bearing, in ADRs 0001 to 0007, in history/, and in ADR 0023 on the estate's retirement.

The guard added hours earlier caught its own allowlist going stale, which is the clearest evidence it was worth writing. docs/archive/ sat in no-wiki-pointers.test.mjs's exemption list; the assertion that fails when an entry matches no tracked file fired the moment the folder went, so the dead exemption was removed in the same change rather than left as a hole.

  • Two workflows and the spent half of scripts/cutover/, after an audit of all nineteen. Owner instruction 2026-09-08: "if the ps1 is no longer needed, we need to clean that workflow — also any other workflow that is not needed has to go". Each removal below is a workflow that had already finished, or one that could not succeed by its own design. Everything else was kept, including two that looked dead and are not.

validate-deployed.yml — twenty-one days red, structurally. Dispatched twice on 2026-08-18 and failed both times; never run again. Both jobs fail for one reason: they run on a GitHub-hosted runner, and through Cloudflare a datacenter IP is answered by Bot Fight Mode with a 403 while direct to origin the origin lock answers 403. The surface job asserts test "$code" = "200" against the apex, so it cannot pass either — the deployment runbook's claim that it "still runs the DNS, TLS and frontend-surface job usefully" is contradicted by its own last run, where that job also failed. deploy-functions.yml depends on exactly this behaviour: it curls the same URL from a runner and FAILS the deploy on a 200, which is how the origin lock is proven. The checks themselves are not lost — they are docs/runbooks/edge-dns-verification.md and node scripts/smoke-deployed.mjs, both run from an operator machine Cloudflare admits. The script is untouched and still has its own tests.

retire-wiki.yml — a one-shot that ran, whose target no longer exists. It overwrote all 141 Wiki pages with pointers to the docs site on 2026-09-06 (dry run, then the real one). The Wiki feature is now off in repository settings — measured has_wiki: false on 2026-09-08 — so its first step, git clone ...wiki.git, could not succeed if dispatched. scripts/docs/wiki-redirects.json is deliberately KEPT: ADR 0027 cites it as the record of what moved where, and that sentence should stay true.

scripts/cutover/05-verify-timer.ps1 and its workspace-query helpers. The script proved a timer fired at its intended time by reading Trigger Details: ScheduleStatus from Log Analytics. Two owner decisions took that away from opposite directions: #321 dropped host.json's Function category to Warning to stay under the workspace's 0.25 GB/day cap, so the host stopped writing the line (measured 2026-09-08: 4,179 AppTraces rows in 24 hours, zero containing ScheduleStatus); and #345 closed the gate it served outright — "all 18 timers are armed and the per-wave observation read is no longer a gate". A tool whose instrument is switched off and whose question has been answered is not waiting for better days.

The clock method survives, in the successor. verify-timer-witness.mjs now carries it: compare a fixed-hour timer's observed firing times against its schedule and read the offset, which needs no host verbosity at all. The worked example is fetchPodcastFeeds either side of #416 — fires at 23:30Z and 01:30Z (odd UTC hours, so an even-numbered Central hour) and then 04:30Z (even, so UTC). The parity flip is the clock change, visible without a single trace row.

Two CI steps and one test file went with them, said plainly rather than quietly. repository-policy.yml no longer runs scripts/cutover/workspace-query.tests.ps1 — a STEP inside validate-structure, never the job, so the required context "Validate root and documentation policy" is unchanged. scripts/powershell-hygiene.test.mjs (5 tests) was deleted: every one of its assertions read the deleted .ps1 to pin its skip regex against the JavaScript log lines, so it had no subject left. And workflow-write-permissions.test.mjs lost its only ALLOWED entry, which TIGHTENS it: the assertion now reads "no workflow in this repository holds contents: write" rather than "exactly retire-wiki.yml does".

What was examined and KEPT, so the audit does not get repeated. heal-computed-properties.yml stays despite the standing "no healers" principle, because the wipe is still real at source: the content and blogs containers are still azurerm_cosmosdb_sql_container (infra/cosmos.tf:243), that resource still cannot express computedProperties, and PUBLIC_LIST_SQL_ORDER = "1" is live (infra/functionapp.tf:518), so an unhealed wipe breaks the public content list. It can go the day the two containers move to azapi_resource. tfc-plan-check.yml stays: its 2026-08-31 red was the Report step emitting a verdict, which is what that step is for, not a broken workflow. verify-alert-state.yml stays: read-only, dispatch-only, green on 2026-08-30, and the only way to see autoMitigate, which is invisible from the repository and from the TFC run list. In scripts/cutover/, 06-seed-secret.ps1 is named as the break-glass path by docs/runbooks/deployment-runbook.md and docs/standards/variables-and-secrets.md; 04-telegram-webhook.ps1 is the remediation functions/src/lib/secret-catalog.js tells an operator to run after a bot-token rotation; and 01-entra-spa.ps1 is parameterised for granting the Admin app role to a named user, which is a recurring operation with no other tool. None is spent.

  • The ai_insights container is dropped from the estate (#340). Owner decision 2026-09-05, after its reader was retired in #339 (T-765): nothing ever wrote it, so it held only the documents the 2026-08-21 migration carried across, with no reader. The manifest entry is gone, infra/cosmos-containers.json is regenerated (72 containers, from 73), and the container prevent_destroy guard is lifted for this one apply, because it covers every instance of the for_each and cannot be lifted for one. The apply is the owner's to approve, on a plan that must read exactly one destroy; the PR that follows restores the guard. This is the first container dropped from production since cutover, and it is done as its own apply rather than a rider on any other change, the rule T-302 set for cleanup timers and scratch.tf records for the sandbox.

  • The AI insights panel and its ai_insights read path (#339, T-765). Owner decision 2026-09-05: retire. NewsPage rendered the panel only when VITE_NEWS_ENABLE_INSIGHTS was true at build time, and the frontend deploy never set it, so no visitor has seen the panel since cutover; and nothing in this repository ever wrote ai_insights, so it could only have shown the documents the 2026-08-21 migration carried across, frozen. Removed: AiInsightsPanel.jsx, the insights branch of useNewsData and fetchPublicFeed, the build flag, and the second query in GET /api/public/feed, whose response no longer carries insights (the tests now assert one query, against rss_cache only). The container and its documents are deliberately not dropped here: that is an apply that destroys data, so it is recorded as its own owner decision in TODO.md rather than riding on the next timer apply.

  • The editorconfig findings that are real: one BOM, six missing final newlines, and one of them was being regenerated (#588). .editorconfig scoping is unchanged, because the diagnosis did not support changing it.

What was actually wrong. frontend/src/pages/admin/SocialHubPage.jsx carried a UTF-8 BOM against charset = utf-8, and six tracked files ended without the newline insert_final_newline requires — the four .vscode/*.json files, frontend/public/icons/certs/certification.svg, and frontend/data/content-manifest.json. #588 predicted "one UTF-8 BOM file and 5 missing final newlines"; the BOM is exact and the newlines are six.

The sixth is the interesting one. content-manifest.json is generated by scripts/build-content-manifest.mjs, which wrote JSON.stringify(manifest, null, 2) with no trailing newline — so fixing the file alone would have been undone by the next manifest run, and the finding would have come back looking like a regression. The newline is now part of next rather than appended at the write, so the unchanged-corpus comparison just above it sees the same bytes it writes and an unchanged corpus still produces no diff. The committed file is fixed too, so the repository is clean now and stays clean.

The 221 count is not explained by the hypothesis in #588, and is measured from a stale scan. That issue expects the bulk to be "JSDoc * continuation lines and markdown list indentation measured against indent_size = 2". Counted on main: 23,811 such JSDoc continuation lines across 800 files, and 572 odd-indent markdown lines. If indent_size were flagging those, the number would be two orders of magnitude larger than 221, so whatever produces 221 is not that, and rescoping .editorconfig against that theory would have changed nothing while looking like a fix. Per the 2026-09-15 note on #588, Qlty has analysed main exactly once (ccce064a, before #565, #584 and #589), so 221 is a figure from before those merges. The remaining count is worth re-reading after a fresh Qlty build rather than chased now.

Verified with the repository's own tooling rather than the scanner, which could not be installed here (its binary download is blocked): a tracked-file sweep reports no BOM, no missing final newline and no CRLF. frontend 1902 tests, scripts 409 tests, eslint and prettier --check all pass.

  • Two pointers to the timer catalogue and flags still named main.tf, which has not held them since the split. functionapp.tf was split out of main.tf on 2026-08-29 (T-754) and local.timer_catalogue and local.timer_flags went with it; two references did not follow.

The costly one is the enabled_timers validation's error_message. That is the sentence an operator reads at the moment a cutover apply has just failed, and it sent them to a file that does not contain the list they have to edit — next to a validation whose own comment says "a typo here is indistinguishable from a timer that does not fire, which is the single most expensive way to be wrong during a cutover window". The other is schedulers.js's header, which claimed every flag is "false" in infra/main.tf; it now also says the flags are generated from local.timer_flags, so the reader learns that arming a timer is an enabled_timers edit rather than a code change.

Deliberately unchanged: the many other infra/main.tf references across the repository are correct — the split moved the Function App, observability, OIDC, providers and variables out, and main.tf still holds Cosmos, storage, Key Vault and the rest. So is the historical note in variables.tf about what main.tf did before 2026-08-24. Only the two that point at moved locals were wrong.

  • Prerendered pages were thrown away and re-rendered on load (#604). App held the idle flag (requestIdleCallback, or a 1500 ms fallback) that defers the theme toggle and toaster. It usually flipped before the lazy page chunk arrived, re-rendering App while the route <Suspense> was still dehydrated. React 19 then client-rendered that boundary and discarded the server HTML, without calling onRecoverableError, so nothing was logged. Measured in Chromium, the first node inside <main> was replaced on AI-102, /azure, /aws, / and /about, and on all six pages with a slow chunk. The idle state now lives in a sibling DeferredChrome component that the boundary cannot see, and the node survives on every page, with or without the delay. e2e/hydration.spec.js now tags inside <main> rather than the app wrapper, which is outside the boundary. It holds back lazy chunks and covers a certification page; it fails on the old build and passes on this one.

  • Five published articles rendered broken hero images, and their link previews were broken too (#518). The Firebase Storage bucket hybridcloudworks-61e8d.appspot.com is gone. Checked 2026-09-12: the bucket root and all 28 distinct asset URLs still referenced by frontend/data/content-manifest.json return 404, verified against fonts.googleapis.com and the live site as controls, so this is not a network artefact.

normalizePublicImageUrl was believed to be a compatibility shim preserving images for migrated content — it was protected on that basis. It was not: it rewrote storage.googleapis.com/<bucket>/<path> into the firebasestorage.../o/<enc>?alt=media form, which is one dead URL into another dead URL. Its falsy branch only caught empty strings and videos, not a 404 at fetch time, so the browser got an <img> with a dead src and og:image / twitter:image pointed at 404s.

Both hosts are now treated as absent, which is exactly what this function already did for a video and right for the same reason: every caller's falsy branch omits the hero, drops the social tags, and falls a card back to its placeholder. A missing image renders better than a broken one. The same correction went into AboutPage.jsx, which carried its own copy of the rewrite.

Three of the five files named as shims were left alone, and that is the finding. gallery-images.js, cert-image-cleanup.js and fetch-image.js do not fetch those URLs — they map a legacy URL onto the Azure blob that replaced it so a delete or cleanup can find it, or skip it as not worth re-hosting. Deleting those branches would make the mapping fail for any row whose object was migrated, turning a successful delete into a silent orphan. Each now says so in place. The rendering side was where the dead URLs mattered.

This is not a data migration, deliberately. The rows still hold the dead URLs and are now inert; regenerating the manifest from Cosmos would bring them back and change nothing. A test reads the shipped manifest, extracts every Google Storage URL in it, and asserts each one normalizes to absent — so the guarantee holds whatever the data says, including for a row restored from an old backup.

  • A 401 from the API carried no reason, so the admin portal had to guess (#517). deny(401, …) set only Content-Type — no WWW-Authenticate, contrary to RFC 6750 — so an expired token, a rejected token and a token missing the delegated scope all reached the browser as the same bare status code. classifyFailure inferred "expired session" from all three and offered the same interactive re-authentication, which recovers only the first. For the other two it is a wasted redirect that teaches the operator nothing.

This is the case #503's own guard comment anticipated in so many words: "a misconfigured audience, say, where the new token is rejected exactly like the old one." The once-per-tab flag caught it after one wasted redirect; this catches it before.

The RFC's error codes happen to draw exactly the line the client needs. invalid_token means the credential is bad and signing in again fixes it. insufficient_scope means the credential verified and lacks a permission, so re-acquiring it returns an identical token and an identical refusal. A 401 with no credential at all carries no error code, which RFC 6750 reserves for a credential that was presented and refused. The client now keys on that: a new UNKNOWN_REASON.CONFIGURATION renders "Admin access is misconfigured" with the API's own description and no retry button, and it does not spend the one automatic recovery either.

One header made the difference between working and invisible. The API is cross-origin, and WWW-Authenticate is not CORS-safelisted, so res.headers.get('www-authenticate') returned null in the browser however carefully the API set it. cors.js now names it in Access-Control-Expose-Headers, on the actual response rather than only the preflight.

A 401 with no challenge — an older API, or a proxy that strips the header — behaves exactly as it did before, and there is a test pinning that.

A 401 carrying no credential at all sends a bare Bearer realm="". RFC 6750 §3 is explicit that such a response "SHOULD NOT include an error code or other error information" — there is no failed attempt to describe, and describing one would only tell an unauthenticated caller how the endpoint behaves. The client reads the absent code as "sign in", which is right.

The header builder escapes rather than strips, so a description survives the round trip to the client's parser intact; CR/LF are normalised to a space, because a quoted-string cannot carry them and a newline in a header value is response splitting.

The invalid_token description is a fixed sentence rather than the verifier's own message. verify-token.js says its claim assertions "land in admin_audit_logs and never reach the client", and the header becoming readable is exactly when that promise needed enforcing: those messages separate expired from bad-signature from wrong-tenant, and jwt audience invalid. expected: … names configuration outright. The client cannot act on the difference — all of them mean sign in again — so only the audit row keeps it.

  • The same GitHub exam was rendering at two different levels on one screen, and the Azure catalogue could not have been made right (#496). GH-100, GH-200, GH-300, GH-500, GH-600 and GH-900 were carried in both frontend/src/data/azure/certifications.js and frontend/src/data/github/certifications.js. /education renders every catalogue side by side, so GH-200 appeared in the Foundational row under Azure and the Associate row under GitHub simultaneously.

Four of the six disagreed, not one. The issue caught GH-200 because that is what the index surfaced; GH-100 and GH-500 are Professional in the GitHub catalogue against Fundamentals in Azure's, and GH-300 Associate against Fundamentals. Five of the six sat at Fundamentals in Azure — a batch default rather than a verified rung.

The Azure copy was unfixable, which decided it. That file's LEVEL_META defines Fundamentals, Associate, Expert and Specialty, so Professional has no rung there at all: GH-100 and GH-500 could not be stated truthfully in that file whatever was typed into them. The GitHub catalogue, by contrast, was verified per exam against each credential's own Microsoft Learn page three times (#461, twice under #469) and its header records which vendor pages are unreliable and why. Owner decision 2026-09-11: the six rows leave Azure, and GitHub exams belong to /github/education. Booking on Microsoft Learn is a real fact about these exams; it did not make a second copy of the rows worth keeping.

The six now-unreachable gh-* study-guide outlines went with them — outlineFor is consumed only by the Azure detail page, and study-guides.test.js asserts no outline exists for an exam the catalogue does not carry. eligibleCerts in scripts/update-study-guides.mjs reads the catalogue, so the generator drops them on its own from here.

Guarded, because nothing could have caught this. Every assertion in education-catalogues.test.js read one catalogue at a time, so a contradiction between two of them was invisible by construction. A new test fails when any exam code is carried by two providers — codes rather than levels, since the vendors' own vocabularies do not reconcile (AWS Foundational, Azure Fundamentals, GitHub Foundations are one rung) and a mapping table would be another thing to keep correct. Verified against the pre-fix file: it names all six exams and both levels for each.

  • The largest catalogue had no DATA_SOURCE, and the test that asserts one covered the other seven (#496). azure/certifications.js exported none, and it sat outside the CATALOGUES list in education-catalogues.test.js, so it was the one catalogue that could omit it without failing a build — and did, from #461 until now. EducationIndexPage carried a documented ?? fallback so the freshness line would still name a vendor.

It matters most there of any catalogue: Azure is the largest, it is the one refreshed automatically by update-learn-catalogue.yml, and so it is the one whose freshness claim is most worth a reader being able to check. The export now names the Microsoft Learn credentials browse page — the same URL the fallback used, so the rendered page does not change — the fallback and its constant are gone, and Azure joined CATALOGUES, which puts all eight under the assertion. Verified by removing the export again: the suite goes red naming azure DATA_SOURCE.label.

  • A botched find-and-replace had been live on the docs site for two weeks (#506). docs/standards/required-inputs.md opened by saying it had moved "from TODO.md on 2026-08-29, when that file was retired and its open work folded into TODO.md" — a sentence that says a file was folded into itself. Later it offered the reassurance that "a citation reading Required-Inputs §4.5 now reads Required-Inputs §4.5".

The source was REVIEW.md. git show e7271df9 records the rename as REVIEW.md => wiki/Required-Inputs.md, and that commit's own message warns "THE BLANKET RENAME WAS THE WRONG TOOL AND I USED IT FIRST." It caught the validator and the casing guard; it did not catch the prose, and nothing else could. Repaired from git history rather than guessed.

  • The naming standard told readers the wrong live resource names. It ended with "These are not the live names. Today's estate is hcw-functions-prod, hcw-cosmos-prod, hcw-keyvault-prod, hcwstorageprod in rg-hybridcloudworks-prod" — contradicting its own centralus section, every runbook, and infra/variables.tf, which declares func-site-prod-cus-01, cosmos-site-prod-cus, kv-site-prod-cus-01 and stsiteprodcus01. Checked against the live estate, not just the code: func-site-prod-cus-01.azurewebsites.net answers 403 (it exists, behind the SCM lock) and hcw-functions-prod.azurewebsites.net does not resolve at all. The grandfathering rule the paragraph existed to state is kept.

  • About thirty citations pointed at two root documents that no longer exist. CHECKLIST.md merged into REVIEW.md on 2026-08-20 and REVIEW.md was deleted on 2026-08-29; variables-and-secrets.md still uses CHECKLIST §n as a table column roughly twenty times.

The citations are kept and explained rather than rewritten, because a mechanical renumber would have been wrong: CHECKLIST was organised by consumer (§1 Functions identity, §7 CI inputs) and Required-Inputs by store (§4.1 workspace, §4.6 Key Vault), so §7 does not become §4.7. A note now says what CHECKLIST was and that its numbers do not carry across. The citations remain the record of where each value was inventoried when its placement was decided.

  • Two decision records claimed a reason that no longer held. The Container Apps CI-runner ADR was kept on the ground that infra/ci-runner.tf "still holds the gated-off resources"; that file, infra/runner-image/ and build-runner-image.yml were deleted on 2026-08-24. The cost analysis and the resource-validation report cited fix/go-live-remediation as a pending branch five times; it does not exist on origin. The ADR bodies are untouched — the register's rule is that an accepted ADR is immutable — and the notes above them now carry the correction.

  • Opaque references now say what they mean. Register rows reading "Give T-519's signal a path" and "Weigh closing T-718 against its cost" state the thing instead of the ticket, and one note explains that the T-NNN scheme was retired on 2026-09-05 when work moved to issues — so every T- in an accepted ADR is a citation, not something to look up.

  • Five history pages were bare checklists. Four presented unticked July-2026 boxes that read as open work on a site whose front page says the migration finished, including one instructing a move to the GitHub Wiki that ADR 0027 reversed. Each now opens with what a reader learns from it.

  • The front page linked ten operator pages and none of the five generic walkthroughs, which are the only pages written for a stranger and the only ones carrying no estate names at all. It now opens with where to start if you are not the owner.

  • Thirteen references still sent readers to wiki/, five days after it was deleted — including one inside a production alert (#502). The Wiki was retired on 2026-09-06 by ADR 0027 and wiki/ was removed from the repository, but the pointers into it were not, and none of them was a dead link in prose. They were instructions:

monitor-functions-registered.yml told the owner, in the body of the alert email that fires when the Function App loses registered functions, to read The failure with no alert in wiki/Alerting-And-Support.md. The one moment that path is read is the one moment it has to resolve, and for five days it resolved to nothing. Four files across two packages cited wiki/Blog-Machine.md as "the cross-package contract of record" — the document a reader is sent to in order to keep the frontend and backend module parsers in step. Two more cited wiki/0025-cosmos-firewall-datacenter-sentinel.md for why the manifest job opens a firewall window.

Four more sat in Terraform comments in infra/, naming the record for a firewall rule and an output. All thirteen now point at docs/, and the alert carries the published URL beside the path so it is reachable from a phone without a checkout.

Nothing noticed because nothing could. A comment is not compiled and a shell string inside a workflow is not linted, so a documentation move that passed every check left live instructions aimed at a deleted folder. scripts/no-wiki-pointers.test.mjs is the thing that notices: it walks every tracked text file and fails on a wiki/ pointer, with an allowlist for the places history must still be able to say the word — the changelog, ADR 0027 itself, the archived and historical snapshots, and the comments that explain the migration. A second assertion fails when an allowlist entry matches no tracked file, so the exemption list cannot quietly rot into a hole. Verified by reintroducing the alert's pointer: it failed with the file and line.

Review found the guard's own blind spot, which is the more useful half. The first version read Markdown, JavaScript, YAML and PowerShell — and not Terraform, so four live pointers in infra/ sat behind the gap: cosmos.tf, outputs.tf and two in variables.tf, each telling an operator which document records why a firewall rule or an output exists. The same blind spot was in the grep that found the original nine, so a guard written to catch this class shipped missing a third of it. The extension list is now broad, and a third assertion fails when a tracked file type carrying five or more files is never read at all — so the next language added to the repository is a decision rather than an oversight.

The migration itself was complete, and that was checked rather than assumed. All 142 Wiki pages as they stood before retirement were matched to a file under docs/: 137 by name, and the five that are Wiki furniture rather than content — Home, _Sidebar, the ADR index, the ADR template and the legacy documentation index — to docs/index.md, the MkDocs nav, docs/decisions/index.md, docs/decisions/template.md and docs/archive/index.md. Every page in the live Wiki is already a stub pointing at the docs site, and the Wiki is disabled: a reader who follows the old URL is redirected to the repository. The docs site serves all of it at https://docs.hybridcloudworks.com/, over five spot-checked sections.

  • The Social Hub's "AI Caption" button reported "Failed to generate caption" and nothing else, while the API had named the cause one field away (#498). generateSocialCaption answers { error: 'Failed to generate caption', message: <why> }, and lib/api.js threw only error. So the toast showed the label and discarded the sentence that said whether a provider had no key, a credential was rejected, or the model returned nothing. The same shape that cost #358 two days on Publer — the status shown, the reason dropped — reproduced inside this client. The thrown message now carries message after the label, guarded so a message that merely restates the error is not repeated; five cases hold it. The underlying cause of the owner's failure is not known from here, and that is the point: the next press will say.

  • Listen & Learn's heading was white on white in light mode. The one bare text-white the #497 sweep missed, because the component lives under components/education/ and the sweep's file list did not reach it. Fixed the same way and the file added to the guard, so it cannot come back.

  • The two secondary links in the Azure detail sidebar now line up with the primary button. They centred their content, so labels of different lengths started at different x; all three are left-aligned with a shared icon column, which is what the owner drew a line beside.

  • The weekly Learn catalogue workflow ends quietly when nothing changed instead of failing at "Artifact not found" (#461 item 3). The refresh job uploads the catalogue only when it changed, but the commit job downloaded it unconditionally, so the first hand-triggered run after #467 (34408439025, 2026-09-09) failed on a catalogue that was already current. The refresh job now exports changed and the commit job runs only when it is true, so an unchanged week shows the job as skipped; the tripwire test asserts the gate.

  • js-yaml 4.3.1 → 4.3.2 in the frontend lockfile (Dependabot alert 148, GHSA-2883-xcg3-v3hh, high: maxTotalMergeKeys did not bound CPU use for empty merge sources). Lock-only: package.json already allowed the patch.

  • Listen & Learn is mounted on the AWS certification pages, its "coming soon" copy no longer names GitHub, every provider in the registry is checked for a dated Learn catalogue, and a flag-disabled timer is visible in Log Analytics once per restart (#461, items 10–12). The audit found ListenAndLearn.jsx telling readers that study podcasts were "live for Azure, GitHub and AWS" while only the Azure detail page mounted the component: AWS had a detail page without it, and GitHub has no detail page at all. aws/education/CertDetailPage.jsx now mounts ListenAndLearn with platform="aws" under the same gate the Azure page uses — the block, and the audio control inside it, render only when published episodes exist — and the backend's SUPPORTED_PLATFORMS already accepted aws, so nothing changed there; a page test covers both the populated and the empty case. The copy now says Azure and AWS (owner decision: honesty over building a GitHub page). Audio stays Gemini TTS; the speech provider code is untouched. provider-coverage.test.js gains a registry-driven check: every provider in PROVIDER_ALIASES must have a src/data/<provider>/certifications.js or education.js exporting a real DATA_AS_OF and no row findStaleStatuses flags, and it names the provider that has none. That was finops, whose page carried its own four rows with no date: they are now src/data/finops/education.js, verified 2026-09-09 against learn.finops.org (where finops.org/certification/ redirects) — Practitioner, Professional and Engineer are listed and active, the Engineer row takes the Foundation's name "FinOps Certified Engineer" and its ten-hour sizing, every link points at the page that exists today, and "FinOps for Platform Engineers" was never a certification: the nearest thing, the "FinOps for Engineers" course, reads "Not currently available", so the row is retired with that evidence and points at the Engineer credential. The FinOps page renders the "Catalogue checked against …" line and a status badge from the dates, like the other hubs. The AWS detail page's Helmet <title> is one template string, the form the Azure page uses, because the multi-child form pre-renders as an empty title. And functions/src/functions/schedulers.js logs the first "disabled — skipping" of each timer per process at Warning — host.json holds the Function category at Warning, so the Information line it used to write never reached Log Analytics and a timer whose flag was off left no trace but a short DurationMs — then drops to Information for later skips, so a deliberately-off timer is seen once after every restart without a Warning per tick; schedulers.test.js runs the real registered handler twice and asserts one Warning then one Information.

  • The GitHub, AWS, Terraform, Google Cloud and VMware Learn catalogues match the vendors again, say when they were last checked, and derive "retiring"/"beta" from dates instead of a typed field (#461, items 5–9). The 2026-09-09 audit found every provider page hard-coded and behind: GitHub had GH-100 on Foundations, GH-300 on Advanced Security and GH-500 on Administration with no GH-900 or Copilot exam; AWS listed SOA-C02 under the CloudOps name a year after SOA-C03 replaced it and knew nothing of the MLA-C01 (2026-09-28), SAP-C02 (2026-11-16), DVA-C02 (2026-11-30) and ANS-C01 (2026-12-31) retirements or the AI Business Strategist beta; Terraform advertised the 003 exam retired in January and a Consul exam HashiCorp retired on 2026-07-15 while omitting both Advanced credentials; GCP was missing five of Google's fourteen and still called the Workspace credential Professional; VMware carried the superseded VCAP-DCV Deploy and no VCP-VVF. Each catalogue now lives in frontend/src/data/<provider>/ with a DATA_AS_OF and DATA_SOURCE the page renders as "Catalogue checked against … on Sep 9, 2026" — the one freshness claim that is true. AWS's two duplicate lists (landing page and CertDetailPage.jsx) are one file, and /aws/education/soa-c02 still resolves through previousSlugs. The pages share the Azure entry's frontend/src/lib/certStatus.js, which gains an upcoming status (a version announced but not yet deliverable, such as SAP-C03 from 2026-11-17), isIsoDate, describeCertStatus (the badge wording, "Retiring · last day to test Nov 16, 2026 · replaced by SAP-C03"), and three more stale checks; every page reads "today" through useToday(DATA_AS_OF) so the prerendered HTML and the hydrating render agree, and education-catalogues.test.js fails the moment any dated row is past (first due: MLA-C01 on 2026-09-29), which is the reminder to re-verify against the vendor. VCTA-DCV is kept because Broadcom's 2024 FAQ names it and no 2026 Broadcom page contradicts it.
  • Azure Learn statuses are derived from their dates, the ten retired exams say so, and every landing card opens a real detail page (#461 items 1 and 2). The Azure catalogue (frontend/src/data/azure/certifications.js) had not been checked since 2026-04-16, so the live page showed AI-900, AI-102, AZ-204, AZ-500, PL-200, PL-500, PL-600, MB-280, MB-335 and MB-700 as "Expiring Soon" ten weeks after Microsoft retired them, twelve GA exams as "BETA · ends Jun 30, 2026", MB-240 as active, and no retirement for AZ-800/AZ-801. Re-verified against the Microsoft Learn credentials browse API and the credential-retirement page on 2026-09-09: those ten plus MB-240 are retired with their dates and replacements (AI-900 → AI-901, AI-102 → AI-103); AB-210, AB-250, AB-410, AB-620, AI-103, AI-200, AI-300, AI-901, DP-750, DP-800, GH-600 and SC-500 are active; SC-730 is retired (Microsoft withdrew it after the beta, no date published); AZ-800 and AZ-801 are expiring 2026-09-30 → AZ-802, and MS-102 expiring 2026-11-30; AZ-802 (active), AB-650 (beta) and AI-500 (beta) are added; five applied skills are retired with Microsoft's dates. The pages now compute status from the dates at render time (frontend/src/lib/certStatus.js, deriveStatus), render a Retired badge, filter and replacement code, and print DATA_AS_OF where they used to claim "scraped weekly", "tracked weekly", "updated weekly" and "Refreshed by Microsoft each month". A vitest fails the day any stored expiring/beta outlives its date. The Azure detail page drops its own 15-entry copy (five of them retired, DP-203 gone) and reads the shared catalogue, so all 68 landing links resolve instead of 15; the 68 /azure/education/<slug> routes are pre-rendered — with a single-string Helmet title, since the multi-child form wrote <title></title> — and the Listen & Learn block appears only when published audio exists (verified for azure/AB-100).
  • ElevenLabs no longer reads Listen & Learn, and the Listen & Learn pin no longer governs the podcast: speech providers are chosen per product (#436, #432). Owner rule 2026-09-09: ElevenLabs is only the podcast voice — article and Plaud transcripts to RSS.com — and Listen & Learn audio is Gemini TTS, generated on demand and stored as MP3, with Azure AI Speech as the GA fallback. #447 had put ElevenLabs first in one global order read by every caller, so the paid podcast voice read every study episode the moment its key landed, and LISTEN_AND_LEARN_TTS_PROVIDER — also read by every caller — would have pinned the podcast too, which is why the owner's instruction to set it to gemini could not be applied as the code stood. speech/index.js now holds SPEECH_PRODUCTS = { listenAndLearn: ['gemini', 'azure'], podcast: ['elevenlabs'] }; resolveSpeechProvider, synthesizeDialogue and estimateSpeechCostUsd require product and refuse to guess without one; each product reads only its own pin (LISTEN_AND_LEARN_TTS_PROVIDER, PODCAST_TTS_PROVIDER), a pin may name only its product's providers, and LISTEN_AND_LEARN_TTS_PROVIDER = elevenlabs fails every run with a sentence quoting the rule. Nothing falls through between products: the podcast without ELEVENLABS_API_KEY saves a transcript-only draft whose audioError names that setting (pinned in lib/podcast/generate.test.js through the real switch), never a Gemini reading, and #447's out-of-credit fallthrough is removed because its only path was the cross-product one. infra/functionapp.tf sets LISTEN_AND_LEARN_TTS_PROVIDER = "gemini" on the Function App, deliberately Terraform-managed so the rule is stated in the estate and a portal edit is drift (one app setting added; app-settings-secrets.test.js and secret-catalog.test.js still pass). The 202, the queued line and the speechEstimateForRun estimate describe the Listen & Learn product only — Gemini or Azure, never ElevenLabs. ADR 0029 §2b records the rule and keeps §2a as history, saying it was written as if ElevenLabs replaced Gemini for Listen & Learn, which was wrong.
  • Listen & Learn audio generates again: the Gemini TTS provider read the SDK's output_audio accessor, which the REST reply does not have (#458). Every generation since the provider landed ended with audioError: "Gemini returned no audio for this dialogue" on a 200. The request matched the documented multi-speaker contract; the parser did not match the response. POST /v1beta/interactions returns an Interaction object whose audio sits in steps[].content[] as { type: 'audio', data, mime_type, sample_rate, channels }, with a top-level status and optional errors[]. output_audio.data is a convenience accessor the Python and JavaScript SDKs put on their Interaction object, and the guide's examples use it; the REST JSON never carried it, so audio in steps was treated as no audio. speech/gemini.js now exports extractAudio, which walks the model_output steps (keeping output_audio as a fallback), joins several audio blocks in order, reads a WAV header for its rate and channel count and strips it, and averages stereo to mono for the MP3 encoder. A reply with no audio now says why on the episode card — the status, each errors[].code and message, and the shape seen by type only (model_output[text], never the text); failed and incomplete statuses are refused explicitly, the latter because the audio would stop mid-sentence. The default model moved from gemini-2.5-flash-preview-tts to gemini-3.1-flash-tts-preview (owner request; COST_TABLE already priced it at USD 20 per million audio tokens, twice the 2.5 flash rate), with LISTEN_AND_LEARN_TTS_MODEL still the override. Verified against the Interactions API reference and the speech-generation guide on 2026-09-09; the module header records what was checked.
  • Listen & Learn scripts have a portal toggle, the docs build no longer dirties the tree, and the router says who writes a key verdict. Three small things left behind by #449, #435 and the docs hooks. script.js declared feature: 'listenAndLearn' behind the injected generate, where ai-call-sites.test.js cannot see it, and no listenAndLearn entry existed in AI_FEATURES — so the AI Engine page had no switch for it and the router treated the call as ungated. The feature is now declared at a literal call site in listen-and-learn/generate.js, the same shape as podcast/generate.js, and the catalogue lists it as Listen & Learn scripts; the source scan and the catalogue tests cover both directions. scripts/docs/__pycache__/*.pyc was tracked and rewritten by every mkdocs build --strict, so every docs session had to check it out before committing; the two files are untracked and __pycache__/ and *.pyc are ignored. The router's key-verdict docstring named "the Publer client" as the other source of a rejected-key verdict; it now names lib/key-verdict.js as the single writer and lists the three reporters that feed it (the router, the Publer timer client, the Publer REST proxy).
  • A rejected Publer key no longer fails syncSocialCalendarScheduled every five minutes, and the API-keys page now says the key is rejected (#358). Measured in Log Analytics on 2026-09-09: Publer answering 401 to a stale key had failed the timer 429 times in 36 hours, 0 successes — 288 exceptions a day, and alert-app-exceptions-prod-cus never fired, because the only thing that knew the key was wrong was Publer, saying so into a log nobody reads. That is the blind spot secrets-health.js names in its own header: it detects unresolved, not incorrect. A rejected credential is a configuration state, not a transient fault, and createPublerClient was treating a 401 like a 500.

Three things change. The timer's run now returns { skipped: true, reason: 'credential_rejected', status } on a 401/403 after one warning naming PUBLER_API_KEY / PUBLER_WORKSPACE_ID — the pair travels together, and a key valid for a different workspace answers 401 exactly like a stale one; a 500 or a timeout still throws, because those ARE transient. The Publer client and publerProxy report the verdict for PUBLER_API_KEY through the path the AI router already had — its onKeyVerdict writer, promoted from a private function on the router's default instance into lib/key-verdict.js so the two reporters cannot disagree about what a rejected credential is — so the red light on https://hybridcloudworks.com/admin/integrations comes on whichever path sees the rejection first, and the first success after that — the rotation — turns it green again in the same worker: a recorded failure re-arms the once-per-worker success report, which the router's private version had never done (Copilot review of the PR). And PUBLER_API_KEY carries probe: 'publer' in the catalogue, because the page prints "no liveness check for this one" beside any light nothing reports on, and secret-catalog.test.js holds that a probe exists only where a reporter is wired. The proxy's response shape and message are untouched; the Social Hub still reads "Publer answered 401". Klaviyo and Linkie do not opt in: their 401/403 semantics have not been read, and a scope-limited key answering 403 on one endpoint and 200 on the next would flap the light.

  • Five education hubs and three detail pages were white text on a white background in light mode, and had been since they were written. Headings, card titles and stat values used a bare text-white with no dark: qualifier, while src/index.css sets --background and --card to pure white in light. A learner on a default iPhone saw a hero and then blank space where every section heading should be.

The fix already existed in the repository and had never been propagated. azure/EducationPage.jsx pairs its headings as text-slate-950 dark:text-white and reads correctly; the other five hubs carried 15 bare instances each and zero dark variants. 102 sites changed, read one at a time rather than swept: 25 were deliberately left, every one of them white-on-an-opaque-coloured-button, where white is correct.

Status chips had the same problem in a subtler form. CertStatusBadge used text-amber-300 and friends over *-500/20 tints, roughly 1.5:1 against a near-white surface. Each status now carries an *-800 light foreground with the original as its dark: variant, around 6:1.

Why nothing caught it. The only colour-contrast gate is a Playwright spec covering one route, and it does not run in CI. It has now been widened to all eight hubs and the three detail templates, but deliberately NOT wired in as a required check: /aws/education was already in its route list while that page was white-on-white, so the spec cannot have been passing, and making a red gate required would block every PR. It wants a reading first.

What replaces it in the meantime is a test that does run. pages/education-a11y.test.js reads the nine files and fails when a text-white token has neither a dark: variant nor an opaque background, when a status loses its light foreground, when a carousel dot loses its label, or when a decorative icon loses aria-hidden. Verified to fail by reverting one heading: it named the file, the line and the fix.

  • The education carousels were silent to a screen reader, and their controls were 10 pixels wide. The pagination dots were <button> elements with no children, no aria-label and no aria-current, announced as "button, button, button" — and they are the only route to anything past the fourth card, which on Azure means past the first four of 108. Each dot now sits in a 24-pixel transparent hit area, meeting WCAG 2.5.8 without changing the 10-pixel dot itself, and carries its page number and current state.

168 decorative material-symbols-outlined spans gained aria-hidden. Because the font renders a ligature, a screen reader had been reading "workspace_premium", "arrow_forward" and "open_in_new" aloud as words — and on a slow connection those words are briefly visible too, since the font loads with display=swap. Every icon-only control was audited first: the one without an accessible name, Azure's "open on Microsoft Learn" link, got one rather than being hidden.

  • refreshPlaudToken no longer disconnects a document that has an access token but no refresh token (#342). It warns and leaves the document as it is; the access token expires on its own schedule.

  • 05-verify-timer.ps1 reported a zero-row answer as a failed query (#336). Invoke-WorkspaceQuery ended with return $rows, and PowerShell unrolls an array onto the pipeline on return — an empty one unrolls to nothing, so a query that ran and found no rows reached the caller as $null, which every caller reads as "the workspace query did not run", with the advice to check az login, the subscription and Log Analytics Reader. None of those was wrong on either occasion it fired: on 2026-09-03 the syncRssFeeds -Hours 8 read after the #321 cut, and on 2026-09-05 the first cleanupSoftDeletedContent read before that timer had fired. The helper now returns ,$rows, so an empty answer arrives as an empty array and reaches the branch that already existed for it — "No invocation rows for these timers" with its two causes — and $null is reserved for the call that failed. The #327 stderr capture had made a genuine failure legible; this makes the absence of failure legible too.

  • The podcast pages had no data, and the azure one now plays real audio (T-764, #PR). Found 2026-09-02 beside the empty news pages: podcasts had one production writer, the fetchPodcastFeeds timer, and its flag had never been true, so every provider podcast page rendered empty and, unlike the news pages, there was no admin job to fill the container by hand. The only way in was Wave 2 of T-518, recorded in the #332 entry under Added.

Closed on two reads, both from the owner's laptop on 2026-09-04. The witness: podcasts.updatedAt fresh at the 13:30Z firing. The page: https://hybridcloudworks.com/azure/audio-architecture renders the episodes, and the public list behind it carries, for every episode, a mediaUrl on podbean's media host with mimeType audio/mpeg and a link to the podbean episode page — so Play and Download reach the file, and Open reaches the page, which is what the player is built to do.

What this does not change: PODCAST_FEEDS holds exactly one feed, so the aws, gcp and vmware podcast pages stay empty by configuration. Adding feeds for them is a content decision, not a defect.

  • fetchPodcastFeeds reports its failures at Warning and Error, so the #321 cut cannot hide them (#330). The first Wave 2 witness read split: syncRssFeeds fresh at the 02:00 Chicago boundary, fetchPodcastFeeds with nothing newer than the May migration. That FAIL could not be read, because the ingest caught every failure — the feed fetch, each episode, an empty feed — and reported it through context.log, which is Information level and gone since host.json gated Function at Warning. A run that fired and failed looked exactly like a run that never fired. Now a feed that throws logs at Error with the provider and the message, an episode that throws logs at Warning with its position in the feed, and a feed that answers with no items logs at Warning — the state nothing reported before. The rows carry no URL and no title: the provider names the feed (PODCAST_FEEDS holds one per provider) and a title is third-party text. The Information summary line is unchanged. 05-verify-timer.ps1 does not surface these rows (its filter is Executed and the skip marker); they are one query away, and they are the one trace the Warning cut still keeps for this timer:
AppTraces
| where TimeGenerated > ago(6h)
| extend cat = tostring(Properties.Category)
| where cat startswith 'Function.fetchPodcastFeeds'
| where SeverityLevel >= 2
| project TimeGenerated, SeverityLevel, Message
  • Three scripts exited 0 without running when invoked on Windows (#329). apply-computed-sortdate.mjs, smoke-deployed.mjs and build-content-manifest.mjs compared import.meta.url against `file://${process.argv[1]}`, which never matches a C:\... path — so smoke-deployed.mjs run from PowerShell reported nothing and exited 0. Now pathToFileURL(process.argv[1]).href, as in check-deploy-drift.mjs; a spawn-based test asserts each entry point actually fires.
  • The timer-observation gate was blind for every invocation after 17:59Z on 2026-09-02, and the record explains how (#328). #321 dropped host.json's Function category to Warning to take host verbosity off the daily cap. The Executed and ScheduleStatus rows that 05-verify-timer.ps1 reads are Information-level in that category, so the host stopped writing them the moment the deploy landed. The #321 entry below says Host.Results was kept at Information because the timer-observation gate reads AppRequests — but the verify script's own header says AppRequests has been empty for the app's whole life and is not the oracle (T-514). The cut kept the table the gate never used and removed the one it did, and neither document mentioned the other.

Found when Wave 2's gates returned nothing on 2026-09-03: the workspace was RespectQuota, the app was running (the admin RSS job had processed 21 feeds three hours after the deploy), and platformJobSweeper -Hours 24 returned 37 invocations ending at 13:00:00 CDT — 18:00Z, the deploy minute — then nothing. The first hypothesis, the daily cap, was wrong and is recorded as wrong.

Owner decision the same day: keep the cut, make the durable side effect the primary witness. Tracked as T-766, High, because waves 3–6 cannot produce their own evidence until each timer's witness is settled. The verify script now warns at the top of its invocation section when host.json gates Function above Information, so a zero after the cut reads as "instrument off" rather than "timer dead" — the confusion this repository has paid for more than once.

  • Wave 1's invocation gate is confirmed, retroactively and by accident (#328). The #323 record says all four gates were observed; the operator had reported them passed, but the run that would have shown gates 3 and 4 had in fact hung on az's hidden install prompt (#327), so no figures were ever seen. On 2026-09-03, with the extension installed, platformJobSweeper -Hours 24 returned 37 invocations, 37 ran, 0 skipped, at exact 15-minute boundaries with -05:00 offsets on every ScheduleStatus line — 04:00:25 to 13:00:00 CDT, continuous until the #321 deploy silenced the host. Wave 1 was real. The #323 entry stands, and this is the observation it was missing.

  • ContentListingTemplate invented a publication date for any item that had none (#325). The blog-variant card read {item.date || 'Feb 10, 2026'}, so an item without a date was stamped with that specific day — not a visible placeholder but a plausible, wrong fact, rendered identically to a real one. Found while removing the hardcoded stamps above: dropping the date fields would have replaced eighteen varied stale dates with one uniform fabricated date, which is worse, so the fallback had to go first.

An absent date now renders nothing. The surrounding row is conditional too — an item with neither date nor author would otherwise leave an empty mb-2 div above the title — and the bullet separator appears only when it has both sides to separate. This reaches every caller of the variant, including the already-live terraform, github and ansible code pages, whose items come from useCoderCornerData and are not guaranteed to carry a date.

  • A stale count in frontend/src/main.jsx (#325). A comment on the hydration-mismatch handler described "the 104 pre-rendered documents". The build has written 120 since 2026-08-28, when 16 standalone routes were added and the sitemap moved 104 → 120 with it; the comment was not updated. The number is removed rather than corrected, because the comment's point is that a silent mismatch wastes the pre-rendering, and that holds at any count.

  • Two links in wiki/Required-Inputs.md pointed at a heading that is not in that file (#324). Both read [Accepted risks](#accepted-risks), and the accepted-risks table lives in TODO.md — so the in-document form resolved to nothing and silently did nothing when clicked. One of them is the only explanation a reader gets for why purge_protection_enabled is false, which is exactly the link worth following. Now ../TODO.md#accepted-risks, matching how the other Wiki pages reach the root documents.

Found by an audit of every relative Markdown link and heading anchor in the 74 tracked Markdown files, prompted by a near-miss in #323: a heading was renamed there and its in-document link had to be updated by hand, with nothing in CI that would have caught it. These two were the only dead anchors in the repository.

  • Three stale wiki records corrected before they cost a round trip (#314). Architecture-Review-2026-08.md still read T-714 "OPEN — needs an owner decision" (closed by #296, hydration landed) and T-709 "owner action open" (closed 2026-08-30/31, both channels observed delivering); each block now carries a dated closing status pointing at the record. And Resource-Validation-Report.md §3 still instructed enabling Key Vault purge protection — the exact opposite of the 2026-08-24 accepted risk — and now marks that item superseded with the reasoning, so the next reader does not re-raise a decided question.

  • The nightly manifest job opens a pull request instead of pushing to main (T-726, #304) — and the reason it needed to was not the one given. The entry that landed it said the ruleset listed the Actions token as a bypass actor, that a bypass belongs to the token rather than to a workflow, and that every workflow holding contents: write therefore inherited the ability to push past twelve required checks.

There is no bypass actor and there was none. Reading /repos/HybridCloudWorks/HCW-HybridCloudWorks/rulesets/20680114 returns enforcement: active, no bypass_actors, and the rules deletion, non_fast_forward, pull_request, required_status_checks. The push was not privileged, it was refused — and had been since the ruleset was last updated on 2026-08-25. The bot's only successful manifest push is 4b8c36d, dated 2026-08-23; every scheduled run from 08-24 to 08-29 went green because the published set had not moved and the push was never attempted.

A workflow can be broken for five days and report success every night, when the only thing that would exercise the broken part is a change that did not happen. That is the durable lesson, and it is why the correction is recorded rather than quietly edited: the wrong version is what justified adding a stored App private key, and the honest justification is narrower — the refresh has to reach main, every route to main is a pull request, and a pull request opened with GITHUB_TOKEN runs no checks.

Corrected in .github/workflows/publish-content-manifest.yml, scripts/github-app-token.mjs, scripts/open-manifest-pr.mjs, scripts/workflow-write-permissions.test.mjs and TODO.md. The third owner action — "remove the bypass actor" — is deleted; there is nothing to remove.

  • A 404 from the manifest route no longer reads as "the app itself" (T-763). scripts/build-content-manifest.mjs said "a 403 usually means the per-run origin window is closed or has not propagated; anything else is the app itself." The nightly job failed with a 404 on 2026-08-30 and 2026-08-31, and that sentence sent the investigation at a Function App reporting 121 registered functions and every health row green.

The app was fine. public/content-manifest arrived with #277 at 2026-08-30 02:45 UTC; the last Deploy Functions run was 01:21 UTC on 7fd9f2a, and git cat-file -e 7fd9f2a:functions/src/functions/public-content-manifest.js reports the path absent from that commit. The route has never been deployed. deploy-functions.yml is dispatch-only by a recorded decision, so merging a route does not deploy it, and #277 shipped a caller for a route it did not ship.

describeFetchFailure(status, url) is now exported and asserted: 403 names the origin window, 404 says the host answered and points at Deploy Functions, anything else is the app. Four tests, mutation-verified. The deploy itself is an owner dispatch, tracked as item 1 in TODO.md — along with the gap it exposed, that nothing detects a route present in main and absent from the deployed revision, because Monitor Functions Registered counts functions and a count cannot see a missing one.

  • The hourly registration monitor is delivered 22% of the time, and its header said otherwise (T-519). monitor-functions-registered.yml asks GitHub for a run at :41 every hour. Measured across the 132 hours from 2026-08-26 08:23 to 08-31 20:23 — the workflow's whole life, on a cron that never changed — 29 of 133 slots produced a run. When one lands it is a median 36 minutes late; only 3 of 29 arrived within 5 minutes of :41. The longest window with no check at all was 12.7 hours, and three separate gaps ran over 10. One check every 4.6 hours, from a workflow documented as hourly.

The header had claimed "an hour of detection latency on a personal site is not the binding constraint." The premise was never true on this repository, and the reason it survived is worth keeping: it reasoned from the cron expression rather than from the run history the cron produced. A schedule is a request, not a guarantee, and nothing had ever compared the two.

Not one unlucky workflow. monitor-unresolved-secrets.yml (29 */6 * * *, added 08-30) has produced 2 runs where about 8 were due; publish-content-manifest.yml (15 6 * * *) has landed anywhere from 07:05 to 18:37. GitHub documents that schedule is delayed under load and that delayed runs are dropped rather than queued. Nothing in this repository can fix that, so nothing here tries.

What changes instead is which half of the pair is treated as load-bearing. This workflow checks the CAUSE — function count, AzureWebJobsStorage, RUNTIME_CONFIG_WRITER, firewall posture. The SYMPTOM is watched by edge/availability-probe, a Cloudflare Worker on a 5-minute cron that GitHub has no say in; worker.js records success = res.status === 200, so an unregistered host — which 404s every route including /api/health — is caught twelve times an hour rather than once every 4.6. That probe's alert is gated on availability_probe_alert_enabled, which is still false, so today only the slow, unreliable half is live. TODO.md raises arming it from Medium to High on that basis, and the workflow header now carries the measurement, the correction, and the dependency.

The budget argument was wrong too, in the direction that mattered less: it read "~720 minutes a month" assuming every firing lands. At 29 runs per 5.5 days the real cost is about 160. Lowering the cron would buy nothing — delivery rate is the constraint, not the requested rate.

  • The manifest route is deployed and the nightly refresh is green again (T-763, closed 2026-09-01). Deploy Functions run 81 ran at 2026-08-31 23:45 UTC on b24371e, the first deploy carrying functions/src/functions/public-content-manifest.js. Manifest run 11 at 2026-09-01 00:06 then went green end to end — Build the manifest succeeded where it had answered 404 on 08-30 and 08-31.

Two things that run confirmed beyond the route itself. The commit job reported "No change to the published set", so the manifest committed on 2026-08-23 already matched what the API serves — the refresh was blocked, but the pre-rendered data was never wrong. And it was the first real exercise of the action majors merged in #303: the log shows digest-mismatch: error, Expected Digest: sha256:dac23a82…, SHA256 digest of downloaded artifact is dac23a82…, Artifact download completed successfully — upload-artifact v7 and download-artifact v8 working under v8's new strict default, which no CI job could have shown because neither workflow using them runs in CI.

What is still unproven: because the published set had not moved, every step after that check was skipped, the GitHub App branch included. T-726's path is exercised for the first time on the first run where the set actually changes.

  • The reachability alert's window never moved, so the "fixed" rule tolerated nothing (T-745, re-opened and closed 2026-09-01). edge_probe_availability fires when successful probes in its window are LessThan 3. T-745 was recorded FIXED on 2026-08-28 as "30-minute window expecting 6 results, firing below 3" — and the threshold did move to 3, and the resource description, the inline comment and edge/availability-probe/wrangler.toml all moved to describe a 30-minute window. window_duration stayed at PT15M.

At the Worker's */5 cadence a 15-minute window holds 3 rows, so a threshold of 3 fires unless every one of them arrives and is queryable. wrangler.toml records that App Insights availability rows "land 1-3 minutes late and sometimes later" — which is why the 30-minute window existed. The finding was not just left open; it was inverted. The shape it replaced (PT15M, threshold 2) tolerated one miss. The shape that replaced it tolerated none, on a Sev 1 rule re-evaluating every 5 minutes.

Caught while preparing to arm availability_probe_alert_enabled for the first time, so it never fired. window_duration is now PT30M. Both halves now carry the pairing rule in the place someone would edit them — observability.tf beside the window, wrangler.toml beside the cron — and the review entry records the correction rather than quietly flipping back to open.

The transferable part: every artefact that described the fix was updated, and the one line that implemented it was not. Nothing compares a Terraform attribute to the prose around it, so four consistent descriptions read as four confirmations.

  • The probe's secret held the wrong value, and the diagnostic built for exactly this found it in one look (T-519, T-746). availabilityResults was empty for the whole table — not just for edge-api-health — since the Worker was deployed on 2026-08-31. wrangler tail named the cause on the first */5 invocation it saw:
"*/5 * * * *" @ 8/31/2026, 8:15:28 PM - Exception Thrown
  Error: APPLICATIONINSIGHTS_CONNECTION_STRING must carry InstrumentationKey and IngestionEndpoint
    at parseConnectionString (worker.js:15:11)

The cron, the deploy and the schedule were all healthy; parseConnectionString threw before a byte was sent, every five minutes — silent from Azure's side, which is the only side anyone was watching.

It was not silent in Cloudflare, and that distinction is the whole value of T-746. runProbe deliberately does not retry or trap the ingestion POST, so a failure yields no telemetry rather than a failure row, and a dead cron, a dead Worker and a dead ingestion path all render as the same empty table. What separates them is the .catch in scheduled that logs and rethrows — the [availability-probe] run failed: line quoted above is that console.error. It paid for itself here.

The instruction is what caused it, so that is what changed. wrangler.toml said "Azure portal → the Application Insights resource → Connection String" — and the portal shows Instrumentation Key directly above Connection String, only one of which parses. It now carries a piped one-liner that reads the value from ARM and feeds it straight to wrangler secret put, so it never reaches a screen, a clipboard or shell history and cannot be the wrong field. The comment also said appi-site-prod-cus; the resource is appi-site-prod-cus-01.

Two eliminated hypotheses are recorded so nobody re-runs them: the workspace is not over quota (dataIngestionStatus: RespectQuota against the 0.25 GB cap), and the cross-subscription read works — rows: [[0]] is a valid result set, not a denial. log-plat-prod-cus-01 is in the Management subscription, so az against rg-mgmt-plat-prod-cus needs --subscription or it answers ResourceGroupNotFound.

TODO.md's verification command was replaced for the same reason the instruction was: (… | ConvertFrom-Json).tables[0].rows yields $null silently when az fails, and -o table renders nothing for this nested shape at any row count. Both make a broken tool and an empty table print the same blank line. It is now raw -o json over summarize count(), which always returns exactly one row.

  • T-726's owner actions are complete, and the merge question is answered (2026-09-01). The App is created, MANIFEST_APP_ID and MANIFEST_APP_PRIVATE_KEY are stored, and auto-merge is enabled. The open question — whether the ruleset would let the manifest pull request merge unattended — resolves to yes, on evidence rather than assumption: every pull request merged on 2026-08-31 and 09-01 (#301-#308) went in with no APPROVED review, #308 carrying two COMMENTED ones, and with no bypass actors each merge had to satisfy the pull_request rule on its own terms. So required_approving_review_count is 0.

Still unproven: the App path itself, which runs only when the published set moves. Manifest run 11 reported "No change to the published set" and skipped the mint entirely. The first article published is the test.

  • "Wait for a success row" was the wrong gate for arming the reachability alert (T-519). The runbook written earlier the same night said to arm availability_probe_alert_enabled once a success == 1 row was visible. That is necessary and not sufficient, and the gap is arithmetic rather than judgement.

The rule fires when successes in the trailing PT30M window are LessThan 3, and the Worker writes one row per 5 minutes, so a full window holds 6. When the probe's secret was fixed at 2026-09-01 01:35 UTC the table went from 0 rows to 2 within ten minutes — satisfying the old gate exactly. Arming there would have counted 2 successes, which is less than 3, and paged Sev 1 on a probe that was working perfectly, for the same reason arming with zero rows would.

The gate is now the count itself: 6 or more, which is 30 minutes of healthy probing and means the window is full and the designed tolerance — 3 missing out of 6 — applies from the first evaluation.

Third time in one night that this alert's window and threshold had to be reasoned about together (after T-745's half-applied PT15M, and the pairing warnings added beside both halves). The window is the parameter everything else in this rule is measured against, and every mistake so far has come from reasoning about one of the pair without the other.

  • Every Terraform run printed the same deprecation twice for an output nothing consumed (2026-09-01). Once for the refresh and once for the plan, both naming the same line — two complete, identical blocks in the run output, which is where the count comes from. Distinct from the "Value for undeclared variable" warnings on those runs, which come from three stale TFC workspace values and are an owner-side deletion; that distinction is the reason the count is stated at all. cloudflare_plan read data.cloudflare_zone.current.plan, deprecated in Cloudflare provider v5:

Please use the /zones/{zone_id}/subscription API to update a zone's plan. Changing this value will create/cancel associated subscriptions.

Removed rather than migrated, because the replacement the warning names is not usable here. In v5 the non-deprecated path is cloudflare_zone_subscription — a resource, not a data source — so reading the plan through it would put Terraform in charge of the subscription, with exactly the create/cancel behaviour the warning describes. Trading a cosmetic warning for a plan Terraform can cancel is not a trade worth making for a value nothing reads.

The output existed since 2026-08-20 to settle an architecture argument about the plan tier, and that argument is settled. The tier still matters — ADR 0024, wiki/Availability-Probe.md, infra/observability.tf and infra/variables.tf all reason about "this Cloudflare plan" — but it is a static fact that changes only when a human changes it, so wiki/Required-Inputs.md now records where to read it instead of Terraform re-reading it on every plan. The cloudflare_zone data source went with it; the output was its only consumer, and var.cloudflare_zone_id is still used by infra/frontend.tf.

  • The frontend deploy's pre-render guard could not pass, and blocked every deploy for two days (2026-09-01). deploy-azure-frontend.yml greps the built /about page to prove pre-rendering produced a page rather than a shell. Its pattern was the literal <div id="root"><[^/], written when the mount point carried no attributes. #296 (T-714) then stamped data-prerendered-route onto that same div, so the literal stopped existing anywhere in the output:
dist contains 121 HTML documents
##[error]dist/about/index.html has an empty mount point — it is a shell, not a pre-rendered page.

The page was fine — 21 KB of real markup. The check could no longer pass on any input, however correct.

It went unnoticed because it fails closed. A broken guard here blocks a deploy rather than admitting bad output, so nothing shipped wrong and nothing looked wrong; it simply waited until someone deployed. Two runs died on it before the cause was read. That is the good failure direction and it is still a two-day outage of the deploy path, which is worth separating: failing closed bought correctness, not visibility.

Fixed structurally, because the pattern was a truth in two places. A bash literal in a workflow described a shape prerender.mjs emits, with nothing holding them together. The pattern is now MOUNT_POINT_PATTERN, exported from prerender.mjs; the workflow greps that exact text; and prerender.test.js reads the pattern back out of the workflow file and asserts it against real injectIntoTemplate output — matching a rendered page, matching one whose seed contains a bracket, and rejecting both a shell and the untouched vite template. Editing either side alone now fails the frontend test job minutes after a push, instead of a deploy whenever the next one happens to be. Mutation tested: restoring the old literal fails the assertion and names both patterns.

escapeAttr also escapes > now. A raw > inside a quoted attribute is legal HTML so nothing rendered wrong, but it makes the value unscannable by the [^>]* the guard relies on — a seed containing > would have made the check read the tag as ending early. The test's hand-written unescapeAttr inverse learned it too, which is exactly the drift that function exists to catch.

  • The alert rules re-notified every five minutes, because a scheduled query rule is stateless by default. alert-app-exceptions-prod-cus fired at 23:06 on 2026-08-25 — the first firing of any rule on this platform — and then kept firing. That was not the rule detecting anything new. autoMitigate defaults to false on azurerm_monitor_scheduled_query_rules_alert_v2, and a stateless log rule notifies on every evaluation whose condition is met, so at PT5M Azure sends a fresh Sev1 mail every five to ten minutes for as long as the condition holds. The window makes it worse rather than better: PT15M evaluated every PT5M means the same burst of exceptions is counted by three consecutive evaluations, so the mail continues for a full fifteen minutes after the last exception was thrown. alert-func-latency was the worst of the three at a PT30M window — half an hour of mail about latency that had already recovered.

auto_mitigation_enabled = true on alert-func-http5xx, alert-func-latency and alert-app-exceptions. Each now fires once, stays fired while the condition holds, and sends one Resolved mail after the condition has been clear for three evaluation periods.

Detection is unchanged and nothing is suppressed — same evaluation frequency, same query, same threshold, same severity. That is the entire reason this change was made on one night's evidence rather than waiting for the week of firing data ADR 0022 asks for: it is the only available remedy that does not trade coverage for quiet. Every other lever — filtering the query, requiring two failing periods, raising the threshold, lowering the severity from Sev1 — makes the rule detect less, and none of them should be pulled before the diagnostic query now in Alerting and support says what is actually throwing. Those levers are documented and ordered there; the thresholds themselves remain the first estimates ADR 0022 declared them to be.

mute_actions_after_alert_duration is mutually exclusive with auto-mitigation and stays where it already was, on alert-logs-capacity alone: its condition cannot clear before the 08:00 UTC cap reset, so there is nothing for auto-resolution to resolve. The two metric alerts needed no change — metric alerts are stateful by default. Recorded as decision 6 in ADR 0022.

  • The SCM lock is armed, and the per-run window is proven under Deny (T-520 closed, 2026-08-25). functions_scm_lock_enabled was set to true on the hcw-azure workspace and applied; az functionapp config access-restriction show now reports scmIpSecurityRestrictionsDefaultAction: Deny with a single Deny all rule. The Kudu endpoint no longer answers the internet.

Deploy run 32902534458 is the evidence that matters, because arming a lock is only half a claim until something has to get through it:

scm default action before the window: Deny
Will use Kudu https://<scmsite>/api/publish to deploy since Flex consumption
  plan is detected.
Successfully deployed web package to Function App.
functions registered after sync: 109
scm window closed — default action 'Deny', no temporary rules left

Every part of the design did what it was written to do. The open step read the baseline as Deny rather than assuming Allow, which is why the same steps worked unchanged before and after the flip. The deploy published through Kudu with the standing default denying everyone else. The close step removed the rule, confirmed the posture it found was the posture it left, and confirmed no ci-deploy-scm-* rule survived — the assertion that keeps a window which silently failed to close from passing as a green deploy.

Sequencing mattered and is recorded because it is the part that would bite on a repeat: the window shipped in #220 and was observed working under Allow (run 32894382986) before the variable was flipped. The first deploy after a premature flip is the one that cannot get in to fix itself.

This closes the reachability half of the exposure. The credential half was already closed — basic authentication is off on both SCM and FTP — so the endpoint now requires an Entra token and an allow-listed source.

  • Production deploys could not authenticate at all, and the cause was a workflow edit rather than an identity problem. de99aa0 put deploy-functions.yml behind environment: production to gate production deploys. That is correct in itself and had a consequence nothing accounted for: declaring an environment changes the OIDC subject GitHub composes. It becomes repo:<org>/<repo>:environment:<name> rather than repo:<org>/<repo>:ref:<ref>, so the branch credential cannot match a job that names an environment — the ref form is simply not what is presented. infra/oidc.tf trusted ref:refs/heads/main and environment:data-migration and nothing else, so every production deploy failed at azure/login:
AADSTS700213: No matching federated identity record found for presented
assertion subject 'repo:HybridCloudWorks@312844660/
HCW-HybridCloudWorks@1268997852:environment:production'

Observed on run 32892582041, the first dispatch after that merge. It had been broken since 2026-08-24 and stayed invisible because no deploy ran in between — the failure is silent until someone deploys, and by then it reads as a permissions or tenant problem rather than as the consequence of a workflow edit.

Fixed by trusting environment:production in both the name and immutable-ID forms, matching the existing pair for every other subject: six federated credentials against a cap of 20. The branch pair is not redundant now and must not be swept up in a future data-migration cleanup — heal-computed-properties.yml and publish-content-manifest.yml declare no environment, so they still present the ref subject. The rule is per-workflow, not per-repository.

A guard now fails the build instead of a deploy. scripts/oidc-subjects.test.mjs cross-references every workflow that uses azure/login against the subjects infra/oidc.tf declares, and fails naming the missing credential and the error it would have produced. No linter or terraform validate could have caught this — both files were individually valid, and only the relationship between them was wrong. The guard was verified by reverting the fix and confirming it fails, rather than assumed to work because it passes. It also asserts both subject forms exist for each environment, since one without the other is half a credential that fails on whichever form the token carries.

  • SCM reachability is closed by a per-run deploy window (T-520, #220). scm_ip_restriction_default_action was the literal "Allow" while the front-end origin was locked to Deny. Verified against the live app 2026-08-25: SCM default Allow, main site Deny with 17 rules, scmIpSecurityRestrictionsUseMain false. Changing the literal was never the fix — the Flex Consumption deploy publishes through Kudu and GitHub-hosted runners have no stable egress IPs, so a standing Deny breaks every deploy. deploy-functions.yml now opens a window before the deploy and unwinds it before the storage window closes. Three things differ from the storage window and each is deliberate: no default-action flip is needed, because App Service honours SCM IP rules normally where the storage firewall ignores them for same-region callers, so the standing posture is never widened; the baseline is read rather than assumed and the close step asserts the posture it found is the posture it left, which is correct both before and after arming, where asserting "Deny is back" would fail every deploy until the flip; and the open step fails if scmIpSecurityRestrictionsUseMain is not false, since SCM would then inherit the Cloudflare-only origin lock and the runner would be refused with the window apparently open. The close step also asserts no ci-deploy-scm-* rule survives — a window that silently failed to close is worse than one that never opened, because the deploy stays green while the endpoint stays admitted. functions_scm_lock_enabled defaults to false, so the apply is a no-op on behaviour; arming it is a workspace edit and the window must be observed working on a real deploy first. The credential half was already closed: basic authentication is off on both SCM and FTP.

  • iac-validate reports on every pull request, so it can become required (T-523, #220). Both jobs are meant to be required contexts on the main ruleset, and adding them while the workflow stayed path-filtered to infra/** would have deadlocked the repository: GitHub does not auto-satisfy a required context whose workflow was filtered out, so every pull request not touching infra/ would have held at "Expected — waiting for status to be reported" indefinitely. The trigger therefore changes before the ruleset does. pull_request loses its paths: filter and the filtering moves inside each job — check out, diff against the base commit, skip the expensive steps when infra/ did not change — while the job still completes and posts its context. push keeps its path filter, because required contexts are a pull request concern. Two details fail silently if got wrong and are recorded in the workflow: fetch-depth: 0, since a shallow clone does not contain the base commit; and the detect step overriding the terraform job's working-directory: infra, since run from infra/ the ^infra/ prefix match never matches. Skipping is at step level rather than job level because whether a skipped context satisfies a required check is behaviour worth not depending on. The ruleset half remains owner-gated.

  • The unlabelled form controls are associated, and the rule that finds them now runs (A-001, #220). The rule could not run at all, so the twenty violations recorded against it had never been observed: it crashed with (0 , _minimatch.default) is not a function on the first file containing a label, and an ESLint rule crash aborts the entire run. The cause was this repository's own supply-chain override rather than the ESLint version. eslint-plugin-jsx-a11y declares minimatch: ^3.1.2 and imports it as a default export, while package.json overrode minimatch to ^10 tree-wide to carry the brace-expansion advisory fix — and minimatch v10 exports no default. The override that closed one supply-chain hole had silently disabled an accessibility rule. Repaired with a scoped override giving the plugin minimatch ^3.1.2 (resolves 3.1.5, past the 3.0.5 ReDoS fix) and brace-expansion ^1.1.12 (resolves 1.1.18, past the advisory), leaving the rest of the tree on 10.2.5; the lockfile change is 38 lines, all additions. With the rule running, the twenty findings proved to be two different things. Seventeen are genuine — a <label> that is a sibling of its control with no association, so a screen-reader user gets no field name — fixed by pairing htmlFor/id across ArchitectureReviewBoard (2, keyed per hotspot id since they render in a map), FrameworkReviewBoard (4), MetadataTab (4) and SpeakingEventsPage (7). The other three are not defects: in ListenAndLearnPage the label already wraps its control, which is an association, and the rule reported them only because it cannot see a custom <Input> as a control. Those are fixed by configuring controlComponents, because rewriting working markup to satisfy a misconfigured linter would have been the wrong repair. The rule is now error rather than off, so neither half can regress.

  • REVIEW.md Part 4 is restored as the required-inputs inventory (T-521, #220). 59e471b cut REVIEW.md from 1,011 lines to 58 and moved the narrative to the Wiki, leaving twelve references to PART 4 — REQUIRED INPUTS across eight files pointing at a section that no longer existed. Two of them were live procedure with nowhere to land: CONTRIBUTING tells a contributor to record new required inputs there, and the Deployment Runbook tells an operator to move an entry from SET to VERIFIED after an apply. The defect was the absent section rather than the references, so the section is restored and all twelve pointers are untouched. Only the inventory comes back — the original §4.0 naming and placement rules are now wiki/Variables-And-Secrets.md's job, and restoring them verbatim would have recreated exactly the duplication the Wiki move ended; each file's intro now names the other. The statuses do not share one confidence level and the section says so rather than presenting a uniform claim: GitHub variables (23), secrets (1) and environments (3) were enumerated live on 2026-08-25, so their presence is observed; Key Vault was not readable, az keyvault secret list returning ForbiddenByRbac because the caller holds no data-plane role, which is itself the correct posture, so §4.6 lists the nineteen secrets infra/main.tf references — establishing each name and consumer but not its presence; and the HCP Terraform workspace was likewise not read, so §4.1's statuses are labelled as carried forward from 2026-08-20. The Terraform tables are generated from the configuration rather than transcribed: 8 of 58 variables have no default and must be set in the workspace, and the seven posture switches are listed with what arming each one does, since those are the entries most likely to be misread as settings. The same live pass is what confirms T-525's three scratch variables are still set with no reader, and that the data-migration environment outlives the workflow deleted in 59e471b.

  • The anonymous feed endpoint is bounded in articles, not just in feeds (T-319). GET /api/public/feed capped how many rss_cache documents it returned but not how many items each one carried, and one document is one whole feed — so a hundred bounded documents could still be an unbounded response. Each surviving document is now trimmed to its newest twenty items by pubDate, with itemCount rewritten to match so the count cannot describe items that are not in the response. Undated items sort last and are dropped first (Date.parse('') is NaN, and a missing date is not "now", so one malformed item cannot evict a dated article); an all-undated feed keeps its stored order, and a document whose items is absent or not an array is passed through untouched rather than turned into a plausible-looking empty feed. The read ceiling is a second copy of the ingest writer's MAX_CACHE_ITEMS_PER_FEED because public-reads.js deliberately has no imports; public-reads.test.js asserts the two agree so they cannot drift.

  • The ingest cap keeps the newest items rather than the first (T-319). buildCacheItems sliced the parsed feed in arrival order. Feed order is conventionally newest-first but nothing enforces it, and both readers of the array sort by pubDate — buildHomepageFeedItems and useNewsData.js — so a feed publishing oldest-first cached its archive and never showed its recent articles, with the cache looking full throughout. The sort now happens before the slice, and processSingleFeed no longer pre-slices in feed order, which would have decided the selection before buildCacheItems could. Drafting is unchanged: it still walks the first ten items of the parsed feed.

  • PATCH /api/cms/{ai-providers|mcp-servers}/{id} no longer persists the hasOauthToken read artefact. stripOAuthToken synthesises the flag on every read in place of the write-only token, so a form PATCHing a field it read back sent the boolean with it — and putConfig already dropped it for exactly that reason while patchConfig did not. Reads recompute the flag, so it shadowed nothing; it was a stale copy of a secret's state written next to the secret, which a later revoke would not have cleared. A PATCH body left with no updatable field after id and hasOauthToken are dropped is now a 400 rather than a write that touches only updatedAt and reports success.

  • CORS_ALLOWED_ORIGINS could never have worked — the name collides with a platform-injected variable (T-513). App Service injects read-only CORS environment variables derived from siteConfig.cors.allowedOrigins, which is a string[]. Ours is unset, so the worker received the serialisation of an empty array — the literal two characters [] — in place of whatever was written to the app setting. parseExtraOrigins split that on comma and produced one "origin" called [], which matches nothing.

Renamed to EXTRA_ALLOWED_ORIGINS. Nothing here may be called CORS_* or WEBSITE_* again.

Three independent writers proved it, and the last one was conclusive. Terraform via azurerm, Terraform via the azapi strip, and a plain az functionapp config appsettings set each put the correct value in ARM; all three times the worker reported []. The final experiment carried three keys in one CLI write — RUNTIME_CONFIG_GENERATION, RUNTIME_CONFIG_WRITER and the origins. The worker reported the first two verbatim and the third as []. Same write, same instant, same process, two distinct HostInstanceIds. Only the name differed.

That sequence also exonerated Terraform and the azapi rewrite, which had been the prime suspect on the reasonable grounds that they were the newest thing rewriting the whole settings collection. They were innocent, and the generation/writer sentinel is what showed it: the workers reported writer=azapi-strip and later writer=cli with the current generation every time, so they were never stale and never missed a write. Without the writer dimension the conclusion would have been "stale worker" and the search would have continued in the wrong place.

  • Telemetry had been dead since 01:33Z and request telemetry had never worked at all (T-514). Two faults wearing one coat, both in host.json.

log-plat-prod-cus-01 caps ingestion at 0.25 GB/day and read OverQuota. What filled it was not the application: Azure.Core logged 39.3 MB across 76,125 messages in 24 hours — every SDK HTTP request and response at Information, driven by the host's continuous blob-lease polling — with Azure.Identity adding 4.4 MB. Application logs were collateral. Both categories are now Warning; fixing the noise beats paying for it.

Separately, Host.Results was set to Error. Request telemetry is emitted at Information, so that one line emptied the AppRequests table permanently — it had zero rows, ever. That is the table that answers "did the timer fire", so Migration-Plan §7's scheduled-job gate was unobservable by construction. Restored to Information. Host.Aggregator was left on Trace, the most verbose level available, for a diagnosis nobody recorded; now Warning.

Two conclusions this reverses. The [cors] diagnostic from the previous entry was written correctly and discarded at ingestion, and the [telegram] control that appeared to prove "no worker logs reach App Insights" was a false negative — worker logging works.

And a tooling trap. az monitor app-insights query --app <appId> returned zero rows for every query, including with no time filter, while the workspace held 138,220 traces. The component is workspace-based with the workspace in another subscription, and the proxy returns empty rather than erroring. Query the workspace and the AppTraces / AppRequests tables directly.

  • The inbound Telegram bot is ported (T-512), not retired. Migration-Plan §6 step 6 said to rewrite getTelegramWebhookUrl() and re-run setWebhook; there was nothing to point a webhook at, because no receiver had been ported — notify.js only sends, and no route accepted a Telegram update (checked against the deployed route table, not just the source). Unlike Cloud Tools (T-410) or Listen & Learn (T-411) it was never recorded as a deliberate demotion, so nobody had decided it. The owner chose to keep the bot.

POST /api/telegram/webhook now serves the eleven commands and the free-form Q&A from Site-Main's telegram-bot.js + telegramWebhook. Two things changed in the port, both forced by the platform:

  • Long commands enqueue instead of running inline. Upstream answered Telegram with 200 immediately and kept working afterwards, which Cloud Functions tolerates and Azure does not — an invocation ends at the response, so /forge and /inspect would have been dropped silently about as often as they ran. Those two and /rss now enqueue the platform job that already exists for each (forge-article, batch-inspect, fetch-rss-feeds, T-322) and reply with the job id.
  • The route is anonymous, and that is the only option. Telegram cannot send a bearer token, so requireRole has nothing to check. It is guarded by two independent checks instead: the X-Telegram-Bot-Api-Secret-Token header compared in constant time against sha256(TELEGRAM_BOT_TOKEN) — the secret is derived, not stored, so there is only one thing to rotate — and the sending chat id against TELEGRAM_CHAT_ID. The first proves Telegram sent it; the second proves the owner did, because anyone who finds a bot can message it. An unauthorized chat gets no reply at all, so the bot cannot be used to confirm it exists. telegram/webhook is in PUBLIC_ROUTES with that reasoning recorded next to it.

It always answers 200 once the secret validates: Telegram retries non-2xx, so a 500 on a bad command turns one broken message into a retry storm that re-runs the command every few seconds. 32 tests, weighted on the two authorization checks, since a mistake in either makes this an unauthenticated remote control for the platform. scripts/cutover/04-telegram-webhook.ps1 re-registers the webhook and preflights the receiver first — a webhook aimed at a 404 makes Telegram back off, so the bot stays broken after the real fix.

  • Eight CMS functions never started — seven route templates were each declared two or three times (T-510). The Azure Functions host keys its route table on the route template alone, not template + method, so two functions declaring the same route with different methods conflict: the host starts one and refuses the other with "is in error: The route specified conflicts with the route defined by function X". The losing verb answers
  • GET/PATCH/PUT were lost across cms/certifications, cms/certifications/{id}, cms/recordings, cms/social-posts, cms/settings, cms/config/{collection}/{id} and cms/keyword-config/{collection}/{id} — list, edit and save for most of the admin UI, all of which the frontend calls. Confirmed live before the fix: POST /api/cms/certifications 401, GET /api/cms/certifications 404.

Present since the 84-function deploy (App Insights, 21:39:15Z 2026-08-21) and invisible because the admin surface is not deployed yet, so nothing had ever called them. Each pair is now one registration via the new httpRouteByMethod, which declares every method on one template and fans out on request.method; each verb keeps the guard it already had. 79 HTTP registrations become 71 and the deploy total 104 → 96, all serving.

route-inventory.test.js could not have caught it: its mock is http: (name, options) => httpRegistrations.set(name, options), a Map keyed by function name, so both halves of a conflict register and pass properties 1–3 — every one of them was individually correct. New property 4 asserts no two registrations share a route template (parameter names collapsed, case folded, matching how a router compares them) and that every method a merged registration declares has a handler behind it. Verified by injecting a conflict and watching it fail.

  • The keyless AzureWebJobsStorage is written by Terraform, not by the deploy — the attribution in infra/main.tf and deploy-functions.yml was wrong, and the Azure activity log is the only place the two are distinguishable: the 20:02Z deploy deleted the setting, Terraform's 20:31Z apply was the only sites/config write after it, and the setting was back. azurerm_function_app_flex_consumption re-injects it on every apply whatever storage_authentication_type says, without surfacing it in plan (azurerm#29149, open on the pinned 5.1.0). Nothing in this repository can stop the write, so it is now stripped inside the same apply that creates it: an azapi_resource_action reads the settings azurerm has just written and an azapi_update_resource writes them back without that key. The setting never survives the run, so there is no post-apply step, no scheduled job and nothing to remember. deploy-functions.yml asserts it is absent and fails rather than deleting it — a repair there would hide a regression in the strip, which is how this stayed a recurring incident instead of becoming a bug: every occurrence was quietly cleaned up by the next deploy.

Not used: "AzureWebJobsStorage" = "", the workaround the issue is best known for — it stopped working in early May 2026, per three reporters — nor rewriting the function app as a raw azapi_resource, which trades a well-understood resource for a hand-written ARM body to dodge one bad key. Both misattributing comments corrected; T-511 tracks the upstream close.

  • The public content list failed the moment PUBLIC_LIST_SQL_ORDER went live — Cosmos: "The index path corresponding to the specified order-by item is excluded". Computed properties are not covered by the /* wildcard (the comment in public-reads.js said they were); /cp_sortDate/? is now an explicit included path on content and blogs, applied live through ARM with the property preserved and carried in the generated spec so Terraform agrees. 40 minutes of 500s on the list endpoint, 2026-08-21.
  • New functions were not registered after the deploy — SyncTriggers failed on a keyless AzureWebJobsStorage connection string the deploy leaves behind (the same cause as the 2026-08-20 every-route-404). 83 deployed, 80 registered; enqueueJob, getJob and the job worker did not exist until the setting was deleted and triggers re-synced by hand. deploy-functions.yml now does both after every deploy and fails if the registered count is zero.
  • cp_sortDate is live on content and blogs (healer run 32448029469, 2026-08-21, first successful run on this estate) and the healer workflow can now be dispatched with mode=inspect to check the precondition for PUBLIC_LIST_SQL_ORDER=1, which infra/main.tf now sets — the public content list asks Cosmos for the newest N rather than an arbitrary N. T-206's final step. The custom role the healer needs is created once by the owner from infra/roles/cosmos-container-writer.json and consumed by data source; the Terraform identity deliberately cannot define roles (#137).
  • The healer can now actually heal. heal-computed-properties.yml had never succeeded on this estate: cp_sortDate was absent from both content and blogs on 2026-08-21 with 1,142 documents in content. Setting computedProperties is a control-plane operation, and the SDK's container.replace() sends it to the data plane, which Cosmos refuses with an AAD token regardless of roles. --apply now does an ARM PUT on the container resource (polling the async operation and re-reading to confirm), authorized by a new custom role — SQL container read + write on the one account, nothing else; not "Cosmos DB Operator", which is databaseAccounts/* minus keys. buildArmBody() strips the read-only keys and is unit-tested. New output cosmos_resource_group → variable COSMOS_RESOURCE_GROUP (T-508).
  • deploy-functions.yml's storage window now survives a same-region runner (T-509): the same default-action Allow/Deny bracket migrate-data.yml gained in #134, with the Deny restored first and verified.
  • heal-computed-properties.yml still read secrets.COSMOS_ENDPOINT after the value moved to a repository variable and the secret was deleted (2026-08-20); its next run failed with "COSMOS_ENDPOINT is not set". Now vars.COSMOS_ENDPOINT. The #128 changelog entry said both consuming workflows had been switched; only migrate-data.yml had.
  • preflight-firestore-inventory.mjs referenced FIRESTORE_PROJECT_ID without importing it. Introduced when the Firestore connection moved into connectFirestore(); node --check and the 65 tests all passed because an undefined identifier is a runtime error on a line no test reaches. The first mode=preflight dispatch from main (run 32435060952, 2026-08-21) found it — after proving the GCP Workload Identity Federation chain end to end, which is the part that could not be tested locally. Fixed, and scripts/ now has an ESLint config with no-undef as an error, run by the scripts (migration) CI job; a sweep of every script found no other instance.
  • migrate-data.yml carried COSMOS_KEY and COSMOS_DATABASE: hybridcloudworks. Key auth is disabled on the account and the database is hcw, so every import would have failed — with an error naming neither. Both removed; the workflow also lacked id-token: write, so it had no OIDC path to either cloud.
  • Eleven moved blocks removed from infra/main.tf. Verified no-ops: the centralus rebuild recreated every container from the spec while all were empty, and terraform state list shows only the for_each form. A three-line note records that the partition-key change happened through the rebuild.
  • Stale counts and comments. main.tf's partition-key comment (67 on /id and five exceptions, not 62 and four); the cosmos_database_name comment (the scripts default to hcw, not hybridcloudworks); cosmos-client.js (67 of 72, not 66 of 71); and the storage lifecycle rule for articles/ is now documented as inert — Azure matches <container>/<blob> and no articles container exists.
  • set-github-variables.ps1 and REVIEW §4.2 omitted FUNCTION_APP_NAME, which is set and consumed by deploy-functions.yml.

  • Azure/functions-action@v2 does not exist. Found while resolving tags to SHAs: that action's newest tag is v1.5.7 and its release branch is releases/v1, so deploy-functions.yml carried a reference that resolves to nothing and would have failed with "Unable to resolve action" the first time the workflow was enabled. Pinned to v1.5.7. The workflow is still if: false, which is why no run had ever surfaced it.

  • frontend/.env.example rewritten against the real environment surface (T-403). VITE_ENTRA_API_SCOPE was required and undocumented — without it every token is acquired for no scope, so sign-in succeeds and every API call fails on audience. The file meanwhile documented VITE_OWNER_ADMIN_EMAIL / _UID, which nothing reads, and carried Firebase secret-set instructions for decommissioned tooling. Rewritten against the actual import.meta.env references.

  • queryDocs does not discard the continuation token (T-311) — recorded because the opposite was asserted in review, and a wrong finding costs more than none. fetchAll() consumes the token rather than dropping it: the SDK's toArrayImplementation loops while (hasMoreResults()), accumulating every page. No change was made because none was needed.

  • iac-validate.yml Trivy job unresolvable action pin — the gate shipped in PR #103 referencing aquasecurity/trivy-action@0.28.0, a tag that no longer resolves: Aqua's 2026-03-19 security incident (trivy discussions #10425) saw trivy-action git tags re-pointed to malicious commits, and the v0.69.4 binary release was itself malicious. The job now installs the Trivy binary pinned to v0.69.3 — the latest release the advisory names safe — from the project's own release artifacts, verified against the release checksum manifest, and no longer uses the marketplace action at all. (PR #104)

  • The frontend CI gate now runs the whole test suite. test:admin was a hand-curated file list — every new test file had to be added by hand, and eight known-stale failures elsewhere were simply never run. The eight were stale expectations, not application defects, and are fixed: the route contract now asserts the real pages behind /gcp, /terraform, /github, /finops, the three /tools routes and the two news routes (mocked, as the suite already did for other providers); and the PublishedPage tests drive the publish flow that actually exists — a pre-publish checklist modal whose "Publish Now" is what publishes — with the checklist itself now unit-tested. test:admin is plain vitest run; the one legitimately unrunnable file (firestore.rules.test.js, which needs the retired Firestore emulator setup) is excluded in vitest.config.js with the reason recorded. Default run: 15 files, 115 tests. (TODO.md T-320)

  • One anonymous list request could eat four seconds of the database's entire budget. The public content list ran SELECT TOP 1000 * with no WHERE — an arbitrary 1000 documents of a ~1k-document container (so published articles could vanish from listings non-deterministically, made intermittent by the 300 s cache), each transferred whole at ~20 KB including article bodies no list consumer renders. The public filter now runs in SQL, so the window counts published documents of the requested type/provider; and the projection is an audited explicit field list — the union of what the public list consumers actually read, pinned by a test naming the consumer behind each field. Of nine heavy body fields exactly one has a list reader (explanation, a Coder Corner excerpt fallback); the other eight stay out, which is where the RU win lives. The in-memory sort and the ORDER BY avoidance stay until a materialized sort field plus composite index can be deployed. (TODO.md T-206, steps 1–2)
  • The API contract can no longer lie about what exists. It documented seventeen RPCs the admin UI invokes that were never registered — every call a live 404, invisible because nothing compared the document to the code. The contract now carries an explicit rpc.notImplemented block (all seventeen, blocked on provider credentials), and a test holds the whole document to account: invoked = implemented + notImplemented exactly; every implemented entry resolves to a registered route with the methods it advertises; registered method+route pairs and contract claims form a full bijection. Making the bijection true surfaced more drift, now fixed: getLabJob was implemented but missing from the invoked list, the Labs agent API had no contract entry at all, six registered admin/public routes were undocumented, and the CRUD shorthand entries now enumerate their real routes — recording honestly that social-posts has no PATCH and recordings no DELETE. (TODO.md T-207)
  • Public news pages showed no curated imagery. #63 moved the cached-image lookup off an anonymous Firestore read onto an editor-gated cms/* endpoint, reached through a token acquisition that throws outright without a signed-in account. The hook runs on the public /{provider}/news route, so for every anonymous visitor the lookup failed and the grid rendered nothing where cached images used to appear. Reading a cached image is now anonymous (GET public/curated-image/{articleId}, returning only the URL — never the document, which carries an internal blob path and prompt metadata), while generating a missing one stays behind the admin gate and is no longer attempted without the editor role that the server requires — not merely when nobody is signed in, since a signed-in viewer would have collected a 403 per article. That also keeps MSAL off the critical path of a public page. Archived images are withheld, so retiring an image in the gallery now keeps it off the public site, and a cache miss is cached for a minute rather than an hour so a freshly generated image is not hidden behind its own absence. (TODO.md T-210)
  • The anonymous submission limit of five could be turned into two hundred. The quota read the counter, compared it, and wrote it back as three separate operations, so simultaneous requests all read the same value, all passed the check, and all wrote count: 1 — accepted submissions bounded only by how many the caller sent, each landing in the review queue, and a counter left at 1 so the trick repeated every burst rather than once an hour. The accepted path is now a single conditional atomic increment: Cosmos evaluates count < limit and applies the increment as one operation, and writes to one document serialize, so exactly five concurrent callers get through. Starting a window and rolling one over are the two things a predicate cannot express, so they go through operations that have a loser — a create that 409s and a replace that 412s — and the loser re-evaluates rather than assuming. (TODO.md T-204)
  • An IPv6 client had an unlimited submission budget. The quota key was the hash of the full address, and a standard residential IPv6 allocation is a whole /64 — 2^64 addresses, each hashing to its own counter, every one of them reading well under the limit, with submission_quota growing a document per address as a side effect. Addresses are now normalized before hashing: full address for IPv4, /64 prefix for IPv6, with :: expanded first so one address written three ways lands in one bucket, and ::ffff: v4-mapped addresses treated as the IPv4 clients they are rather than collapsing every such client into a single shared bucket. (TODO.md T-205)
  • The editor could silently overwrite a colleague's save. Replacing onSnapshot with a twenty-second poll left behind a one-shot "this response is my own write" flag that was consumed by whatever the next tick happened to return. At millisecond latency that was reliably our own write; at twenty seconds it can be a collaborator's — and the branch then adopted their edit marker as our baseline, so the next save passed the server's optimistic-concurrency check and their work vanished with no warning to either person. saveEditorDraft now returns the blogEditedAt it wrote and the client matches on that identity, so the flag is gone rather than merely narrowed. It also fixes an adjacent bug: a second save inside the poll window used to send the pre-save marker and conflict against the caller's own previous write. (TODO.md T-208)
  • Twenty seconds was long enough to lose an image reorder. The poll had no change detection, so every idle tick re-applied the remote document over orderedImageUrls — local state the user drags into order and that is only persisted on save — and re-rendered the whole editor while doing it. Ticks that carry a marker we have already seen now return early. A genuine remote change still replaces the order; the tests assert both directions, because an early return that goes too far is just a stale editor. (TODO.md T-209)
  • total reported the page size. Two public list endpoints measured it after slicing, so it always equalled items.length and any paginating consumer would conclude there was exactly one page. (TODO.md T-407)
  • Two routes the frontend called did not exist. recordLegacyBlogsRead and getPlatformHealth were registered nowhere — both 404s. The health one meant every anonymous visitor saw four CHECKING indicators resolve to "Health check unavailable" on the landing page; the telemetry one meant fallback-container reads went unmeasured, which is the evidence for retiring that container. Both were invisible until T-101, because until then they were pointed at the decommissioned Google host. (TODO.md T-316)
  • Scheduled publishing works. scheduledPublishDate had a complete write side and no read side: an operator scheduled a post, the server validated and stored the date, the UI confirmed it, and nothing ever published it — no error, no alert. publishScheduledContent now runs the same processPublishContent pipeline the Publish button uses, rather than a second implementation of it, clears the schedule only after a publish that actually happened, caps each tick at 25 with carry-over, and records failures under the scheduled_publish_failures alert type the ops dashboard has counted since the migration without ever having a producer. (TODO.md T-301)
  • Two concurrent publishes can no longer both succeed. patchDoc gained an optional ifMatch, and the publish write is now conditioned on the ETag read at the top of processPublishContent — the status gate, quality and image reports and slug were all decided from that document. A lost race is reported as skipped rather than counted as a publish that did not happen. Timer-driven publishing is what turns this from theoretical into reachable. (TODO.md T-301)
  • The four timers no longer share one flag. Enabling the scheduled publisher would also have armed cleanupTempStorage, an unimplemented TODO that deletes blobs. Each timer has its own flag; FEATURE_FLAG_SCHEDULERS is a master kill switch. The blob-GC job itself is still unwritten and still flagged off. (TODO.md T-302, flag half)
  • Every admin list sort worked again. PublishedPage and EditorListPage kept the Firestore-only ?.toMillis?.() || 0, which against the ISO strings Cosmos returns scores every document 0 — so every comparator returned 0, the lists rendered in raw database order while the sort controls appeared to work, and the timestamp columns showed an em dash. One lib/dateUtils.js now backs all of it. The review counted seven copies of that helper; there were ten, and a source guard in the new test file found the last three — one of which only surfaced when the bundler refused a redeclaration that ESLint had passed. (TODO.md T-304)
  • The review board no longer blanks on a scheduled item. BlogReviewBoard called .toDate() on what is now an ISO string, inside a setTimeout and so outside the error boundary. (TODO.md T-303)
  • A published article can no longer 404 because a draft shares its slug. The detail lookup ran SELECT TOP 1 with no ORDER BY and applied the public filter afterwards, so it picked arbitrarily among duplicates and then rejected the winner. It now orders by _ts — a system property present on every document, so the drop-on-undefined trap does not apply — and finds the first public candidate. (TODO.md T-305)
  • The Labs dashboard reported agents "connected" through an outage. The staleness clock advanced only inside the snapshot fetch's success path, so a failing poll froze it: now - lastSeenAt stopped growing and every agent stayed online for exactly as long as nothing was reachable. The clock is an independent interval again — it has to keep running when the fetch does not, which is the only condition under which it says anything. (TODO.md T-309)
  • A timed-out lab job was polled forever, and a network blip was displayed as a failure. The console's terminal-status set omitted timeout, which the agent does report — while the output pane in the same file had the correct four-element list, so the loop kept polling a job its own display had already called finished. Both now read TERMINAL_JOB_STATUSES from lib/labsPolling.js. A transport error no longer writes status: 'failed' onto the job, which was indistinguishable from a real failure and stopped the poll permanently; it is separate state, shown as "still running — retrying", and the poll backs off from 5 s to a 60 s ceiling without ever giving up. (TODO.md T-308)
  • Overlapping polls could render an older document over a newer one. Both the Labs snapshot (15 s interval) and the editor's remote-document watch (20 s) allow a 20 s request timeout, so ticks overlap under load and responses can land out of order. Both now skip a tick while one is in flight. In the editor the flag is released in a finally: its catch returns early on a missing document and on cancellation, and either path would otherwise have stopped the poll for the lifetime of the page. (TODO.md T-309)
  • The browser called Google Cloud, not Azure. api.js, publicApi.js and legacyBlogsTelemetry.js each resolved VITE_GCP_FUNCTIONS_URL — a decommissioned Google Cloud Functions host — so roughly sixty call sites, including every authenticated admin request, would have been sent off-platform with an Entra bearer token attached. lib/functionsBase.js is now the single resolver over VITE_AZURE_FUNCTIONS_URL; the dead azureConfig.js provider switch was deleted. The base carries the Functions api route prefix and accepts either /api (same-origin) or an absolute origin (cross-origin), so deployment topology is configuration rather than code. A deploy build with no base configured now fails instead of shipping. (TODO.md T-101)
  • Every upload and every gallery delete would have thrown. blob-storage.js required STORAGE_CONNECTION_STRING, which no file in infra/ has ever produced — the code was written for shared-key auth while the infrastructure was built for managed identity. It now uses DefaultAzureCredential against STORAGE_BLOB_ENDPOINT, matching cosmos-client.js, and generateSasUrl signs with a user-delegation key instead of an account key. No key or connection string was added. (TODO.md T-104)
  • Uploaded images were unreachable, and the URL to them was stored anyway. allow_nested_items_to_be_public = false is an account-level master override, so the three containers declared public in Terraform served 409 — while uploads returned the raw blob URL for pages to persist into Cosmos. Uploads now return the media-route URL, non-public containers return none, and the Terraform containers are declared private, which is what they always were. (TODO.md T-105)
  • Scheduled-publish dates were silently dropped — scheduledPublishDate and the editor's blogEditedAt were parsed with Firestore Timestamp-only code paths that returned 0 for the ISO strings the API now returns. This would have emptied the scheduling calendar and disabled external-edit-conflict detection. (#64)
  • Labs agents would have shown permanently offline — the staleness calculation understood only Timestamp.toMillis(). (#65)
  • Admin list projection was missing workflow fields — scheduledPublishDate, softDeletedAt, blogEditedAt and eight others were absent from the snapshot projection that replaced whole-document Firestore reads. (#64)
  • MCP server connection state always read as disconnected — the write-only oauthToken strip left consumers unable to detect a stored token; reads now carry a hasOauthToken boolean while the value itself never leaves the server. (#62)
  • Public list endpoint under-projected — it returned a card-field subset while consumers read frameworkConcepts, featured, altCoverImageVariants and more; it now returns full documents with internal fields stripped. (#61)

Security

  • The alert checker signs in as an identity that can read alert rules and nothing else, in both subscriptions (#1053, owner's choice 2026-10-11). verify-alert-state.yml ran as github_reader, which also holds a Function App configuration write and could not read the Management subscription. So the capacity rule, the lab rules, the Management role-change rules and the action group read UNREADABLE every week.

  • infra/oidc.tf adds github_alert_reader, with main-branch sign-in links in both the name and the immutable form, and grants it HCW Alert Rule Reader in rg-web-site-prod-cus and rg-mgmt-plat-prod-cus:

    • The role, in infra/roles/alert-rule-reader.json, is the four reads of scheduled query rules, metric alerts, Activity Log alerts and action groups, with no data action, so the identity cannot query a log.
    • The owner creates it once, as for every custom role here.
    • principal_type is set on both grants, because the identity is new in the same apply.

    The outputs gain alert_reader_client_id, and federated_subjects gains the new pair. - The workflow signs in with ALERT_READER_CLIENT_ID and stays switched off until it is set. It pins no subscription (allow-no-subscriptions), because the identity's one role is at resource-group scope. It passes both subscription ids as ALERT_SUBSCRIPTION_IDS, from SUBSCRIPTION_ID and the new SUBSCRIPTION_MGMT_ID. The script uses them for any subscription the identity's list leaves out, and an entry that is not a name and a GUID stops the check. - scripts/verify-alert-state.mjs empties its UNREADABLE table and keeps the mechanism. Its tests now use a record of their own, and a new test pins that a refused Management read fails the run. - scripts/oidc-subjects.test.mjs maps the new variable to the identity. - The plan checker declares the five creates. - The alerting runbook and the required-inputs table say all of this.

  • postcss-selector-parser is pinned to 7.1.6 in the frontend, closing Dependabot alert #159 (quadratic selector parsing, CPU exhaustion, every version below 7.1.6). The only consumer, @tailwindcss/typography, still pins 6.0.10 in its newest release (0.5.20), so the fix is an npm override. The production build's CSS is byte-identical with it, so typography's output is unchanged. It is a build-time dependency that only ever parses this repository's own stylesheets.

  • Qlty security findings on Terraform and workflows cleared, by owner decision (ADR 0031). Qlty listed 19 open checkov and trivy findings on main, and the owner decided each group on 2026-09-14.
  • Fixed in code:
    • Key Vault purge protection is on. It is one-way, and supersedes ADR 0021.
    • Function host storage moves from LRS to GRS.
    • The two remaining CKV_GHA_7 workflows lose their dispatch inputs.
    • Storage resource logs reach Log Analytics through new diagnostic settings: queues on both accounts, and host blob writes and deletes.
  • Host blob reads are not logged. Measured at about 573,000 a day, they would trip the 0.25 GB/day cap that the log alert rules depend on.
  • Recorded decisions, cited by inline skips: Microsoft-managed keys, no private endpoints, no infrastructure encryption, and no Flex zone redundancy.
  • Public network access stays enabled. Disabled would also refuse the service-endpoint firewall rules, and every data service already defaults to Deny.

  • Local checkov now reports 0 failures on infra/ and the workflows, and trivy 0 on infra/.

  • The 21 remaining zizmor template-injection findings are fixed, and Qlty's TODO noise is triaged (#588). ci.yml, codeql.yml, iac-validate.yml, repository-policy.yml, deploy-functions.yml and publish-content-manifest.yml no longer expand github.event_name, github.run_id, the pull request base sha or matrix.* inside run: scripts; each reaches the script through env: or the runner's own $GITHUB_* variable. None was attacker-controlled, and every command runs with the same values. zizmor 1.30.1 now reports no template-injection. .qlty/qlty.toml ignores ripgrep TODO/NOTE/OPTIMIZE and radarlint-iac terraform:S1135 in infra/, keeping FIXME, HACK and BUG; the triage record's rule snippet is corrected, because a rule key containing a colon only matches with its plugin prefix. Three files Qlty's prettier check flagged are formatted.

  • Every security scanner finding on main is triaged, and most are resolved (#567, closes #562). docs/security/scanner-triage.md has one row per finding from a local Qlty run: 113 across zizmor, checkov, trivy, bandit, gitleaks and radarlint-iac. Each row gives the verdict and its evidence.

  • Fixed, 41. Every actions/checkout sets persist-credentials: false; no job pushes with the checkout token. deploy-functions.yml reads repository variables and step outputs through env:, not ${{ }} in run:. hooks/claude_event.py and tooling/workflow.py resolve git with shutil.which.
  • Two free, in-place Terraform changes. allow_nested_items_to_be_public = false on the Function host storage account, and access_key_metadata_writes_enabled = false on Cosmos.
  • Silenced, 36 (false positives and accepted risks). Each one points back to its row: inline #checkov:skip and #trivy:ignore comments, # nosec with the reason, fingerprints in .gitleaksignore, and one entropy rule in .checkov.yaml. The repository-structure check now allows those two root files.
  • Left visible, 17. They need an owner decision on cost or irreversibility: private endpoints, customer-managed keys, infrastructure encryption, host storage geo-redundancy, storage logging and Flex zone redundancy.
  • radarlint-iac S1135, 19. Each hit is the file name TODO.md in a comment. They wait on the .qlty/qlty.toml triage rule from #568.

  • #562. No unpinned uses: exists anywhere.

  • Author-written HTML can no longer claim ids the application looks up (#298). DOMPurify's default configuration strips an injected <script id="…"> but keeps an injected <div id="…">, and getElementById returns the first element with that id of any tag. Article bodies render inside the app's own markup, so an author could answer a lookup the application makes — reproduced against this repository's own DOMPurify rather than argued.

src/lib/sanitizeHtml.js enables SANITIZE_NAMED_PROPS, which prefixes author id and name with user-content-, and an afterSanitizeAttributes hook rewrites same-document fragment links so in-page anchors keep working. That is what made it safe to ship without a content decision: nothing disappears. The three call sites — RichTextBody and BlogDetailTemplate's two branches — now share one function, because three bare DOMPurify.sanitize calls are three configurations that can drift invisibly.

  • The hydration seed moved off an id-addressed island (#296). The seed was a <script type="application/json" id="__PRERENDER_DATA__"> found with getElementById. By the clobbering path above, an injected <div> carrying that id sits inside #root, comes first in document order, and would have supplied the seed for its own page — an author-controlled object flowing into every href and src on it. It now rides on the mount point, which the template writes ahead of everything the pre-render puts inside it, so nothing an author writes can precede it. This is what cleared CodeQL's 27 js/xss-through-dom alerts; safeUrl did not, and its header says so.

  • safeUrl guards every href and src fed from content data (#296). Allows http, https, mailto and relative references; refuses javascript:, data: and everything else. The alerts were new and the weakness was not — the same values previously arrived from fetch() and reached the same attributes unchecked.

  • A pre-existing HTML injection in the canonical link (#296). socialTags interpolated the route unescaped into href while every tag beside it went through escapeAttr, so a content slug could put a live element in <head>. Found by an escaping test written for something else.

  • The plan-check tool was PowerShell for an hour, and that was the wrong language (2026-08-29). 07-check-plan.ps1 shipped in #271 and was replaced by scripts/check-tfc-plan.mjs before it was ever successfully run. Two reasons, both mine to have seen first. The other cutover scripts are PowerShell because they drive the Azure CLI from a Windows desktop; this one only feeds a Node script. And T-724's remaining half is running this inside iac-validate.yml — on ubuntu-latest, where a .ps1 is the wrong artefact. It also failed the simpler test of being runnable at the bash prompt the owner was actually sitting at: bash: ./scripts/cutover/07-check-plan.ps1: No such file or directory.

Calling checkPlan directly instead of shelling out removed three problems the PowerShell version needed code to handle: no temp file, so the plan's sensitive variable values and copies of state never touch disk; no deletion path to get wrong; and no native exit-code trap, since the checker exits 1 to mean "unexpected plan" and PowerShell 7.4+ turns that into a thrown error.

Running it found something the inspection pass could not. An invalid token is refused with 403, not the 401 the PowerShell version special-cased — so that branch was close to dead code and the common failure fell through to a bare status line. The header now separates what was observed (403, reproduced against a junk token) from what is documented but unreproduced here (404 for an organization token, from HashiCorp's /plans reference), rather than asserting both with equal confidence.

  • "Immediate: restore admin access" deleted; it had been fixed for some time (2026-08-29). The section described a live 403 from POST /api/bootstrapCurrentUserAdmin and asked for the Entra Admin app role to be assigned. The owner confirmed the portal loads and signs in immediately, so the role is assigned and the 403 is gone. It had been sitting at the very top of the tracker marked Immediate — the loudest item in the document, describing something already done. Same shape as T-526, where three files insisted on work that was finished. The Admin app role assignment survives as one clause of the Entra row, which is where a standing requirement belongs.

  • scripts/check-tfc-plan.mjs: the plan assertion in one command (2026-08-29, toward T-724). assert-expected-plan.mjs has existed since T-724 and compares a plan against the known permanent diff by resource address; what nobody had automated was getting a plan into a file to feed it, so it was run by hand or not at all. This resolves the workspace's latest run, downloads the JSON plan, runs the checker and reports the verdict.

Why the summary line cannot be the check, stated in the script: the azapi read-then-strip pair working around azurerm#29149 means every plan reports a diff, and the steady state is 3 add / 1 change / 3 destroy. Three different replacements produce the same three numbers.

Two failure modes are called out because both mislead. The json-output endpoint answers 404 for a token without workspace admin — including any organization token — which reads like a missing plan rather than a permissions problem. And the checker exits 1 by design to mean "unexpected plan", which PowerShell 7.4+ turns into a thrown native-command error under $ErrorActionPreference = 'Stop'; unsuppressed, the one outcome the script exists to report clearly would have surfaced as a raw stack trace. Caught by inspection, being the same trap already fixed in 05-verify-timer.ps1.

The downloaded plan carries sensitive variable values and copies of state, so it goes straight to a temp file, never into a shell variable, and is deleted in finally including on Ctrl-C.

  • Follow-up to the main.tf split: the counts it left wrong (2026-08-29). Removing T-754's row from TODO.md left the summary table reading Low | 2 / Total | 9 over a list of eight, and the prose still said "seven of the nine". That is the same accurate-when-written count nobody updated that this session had already corrected twice — in Architecture_Plan §7 and in Migration-Plan's five self-counts — committed by me about an hour after fixing the second one. Corrected to 8, with T-718's and T-749's anchors repointed from main.tf to cosmos.tf and functionapp.tf.

wiki/Architecture-Review-2026-08.md's ~30 main.tf:NNN anchors are deliberately not repointed. They are evidence of what a reader saw at commit 31f9613; rewriting them to today's lines would make the record claim something it never checked. A note at the top says they are pinned, unmaintained, and that resources kept their addresses through the split — which is how to find them now.

  • main.tf split by concern: 2,286 lines → 97 (2026-08-29, T-754). Seven new files — functionapp.tf, storage.tf, cosmos.tf, frontend.tf, keyvault.tf, budget.tf, network.tf — alongside the observability.tf, oidc.tf and hub.tf that were already carved out. main.tf keeps what everything else references: the locals, azurerm_client_config, the resource groups, and the two telemetry sinks the diagnostics point at.

Done block by block, never by line range. The concerns were interleaved — Cosmos containers at 404 and again at 2125, storage role assignments at 1745 sitting between the azapi resources and the Key Vault — so ranges would have meant hand-arithmetic over 2,286 lines where one off-by-one silently moves half a resource. A parser took top-level blocks with their attached comments, assigned each by address, and then proved the result: the multiset of non-blank lines is identical before and after, 2,107 of them, with zero lines lost and zero non-comment lines added.

That proof earned its keep on the first run. The parser's original "a block ends at the next line that is }" swallowed data "azurerm_client_config" "current" {} — a single-line block — together with the section header and the entire locals block after it, silently. The count assertion caught it and it was rewritten to track brace depth, ignoring braces inside strings and after a #, because a comment mentioning admins/{oid} is otherwise an unbalanced brace.

Nothing moved between states and no resource address changed. Terraform reads every .tf in a directory as one module, so this is invisible to state, and an empty plan is the proof. The guards that read the configuration as text did not need to care, because #268 had already pointed them at the whole module rather than at main.tf — which is the entire reason that PR came first.

  • Five guards asserted things about a file when they meant the module (2026-08-29). cors-platform-origins, secret-catalog, app-settings-secrets, timer-catalogue-sync and scripts/assert-expected-plan all read infra/main.tf by name. Terraform does not work that way — every .tf in a directory is one module, one namespace, one state — so each was checking one file and reporting on the configuration.

One of those gaps was already live. assert-expected-plan.test.mjs matches ^resource "azapi_..." to prove its permanent-diff allowlist names every azapi resource. An azapi resource in observability.tf, oidc.tf or hub.tf was invisible to it: the allowlist would look complete while missing an entry, and that resource's permanent replacement would be reported as drift — training an operator to ignore the one tool that says the plan is wrong. cors-platform-origins's expect(source).not.toMatch(/support_credentials \s*=\s*true/) had the same shape, and would have passed vacuously if the Function App ever moved.

All five now read every .tf in the root module, through functions/test/terraform-source.js (and a deliberate eight-line duplicate in the scripts package, which is a separate npm package with no workspace between them). The helper refuses to run on fewer than four files, because a wrong path yields '', and every "these two lists agree" assertion passes by comparing two empty lists.

app-settings-secrets also stopped bounding a block with someone else's resource. It read from app_settings = merge({ to resource "azapi_resource_action" "function_app_settings" — a different resource seventy lines further down that merely happened to follow. It now ends at }), the line that actually closes the block, so where the block ends is a property of the block. As a side effect it stops over-reading those seventy lines into the settings it checks.

Verified by simulating the split these guards have to survive: moving the azapi read-back into a new .tf file leaves all five green, and breaks two of the five old ones — expected -1 to be greater than 58686 for the settings parser, and a straight miss for the plan checker.

  • REVIEW.md retired; one open-work document from here (2026-08-29). It held the owner-gated half of the open work — and every item in it was already mirrored in TODO.md under Gate: owner, because TODO's own header said it should answer "what is still open" without a second document. Two files, one restating the other.

Split by what the content actually was, rather than moved wholesale. The four work sections and the handling rules (~75 lines) are appended to TODO.md unabridged. PART 4 — the 228-line required-inputs inventory — became wiki/Required-Inputs.md, keeping its §4.1…§4.10 numbering because about sixteen code comments cite those numbers. Putting a reference inventory inside a task tracker would have recreated the problem one level down.

REVIEW.md's header argued PART 4 sat there because the two procedures that write to it are access-gated. That stopped holding when the file went away, and it was always weaker than it looked: wiki/ is reviewed through pull requests exactly as the root is.

The blanket rename was the wrong tool and I used it first. A REVIEW.md → TODO.md sweep produced 'TODO.md', 'TODO.md' in the validator's allowlist, a casing guard reading "Review.md and TODO.md collide", and — the one that would have failed CI — a $requiredFile list that still required a file being deleted. Reverted and redone with ordered rules: §4.x citations to the new page first, then Part 4 prose, then the §0.x/§2.x/§8.x citations, then links, then the bare filename. \bREVIEW\.md\b throughout, so PREVIEW-SIGNING-SECRET and REVIEWER survive.

Found on the way: citations to REVIEW §0.1, §0.2, §0.4, §2.2, §2.4, §8.1 and §8.2 were already dangling — that numbering has not existed in REVIEW.md for a long time. Those now name TODO.md without a section number, because a number resolving nowhere is worse than no number.

validate-repository-structure.ps1 drops REVIEW.md from the allowlist, the casing guard and the required-file list, so it is now rejected at the root rather than permitted. Its comments record that this consolidation has happened twice — CHECKLIST.md and Variables.md merged into REVIEW.md in 2026-08, for the same "several documents describing one thing, disagreeing with each other" reason.

The root is three files: README, TODO, CHANGELOG.

  • The two planning documents are closed and moved to the Wiki (2026-08-29). Both had been marked ARCHIVED since 2026-08-24 while sitting at the repository root beside TODO.md and REVIEW.md, which is where current work lives — so a reader met four documents in the same place, two of which were history.

Both last open risks closed by owner decision, and neither by being fixed — each row says so, because a risk register that records only the risks that got solved is a worse record than one that admits when a risk was accepted or became irrelevant. Cron syntax / time zone: the configuration is as correct as it can be made without arming, and settling the clock from history is now a single script run; closed as a migration risk, while arming the timers continues as T-518 in TODO.md and is unaffected. Cost overrun: a migration risk exists to inform a go/no-go, and Azure is the permanent and only environment — an overrun changes what gets built, not where it runs. Budget continues as a standing requirement on every deployment.

With those closed, Migration-Plan has no open items of its own. Its last exit criterion (§6 step 7) was already tracked as T-518.

Moved to wiki/Architecture-Plan.md and wiki/Migration-Plan.md, which is what validate-repository-structure.ps1 already said should happen: "Narrative documentation still belongs in the Wiki … [wiki/] is the one sanctioned in-repo home for narrative documentation." Both names were hyphenated to match every other Wiki page, and the root allowlist now rejects them at the root rather than permitting them, so the move cannot quietly reverse. 29 files of citations were updated with them, and three Wiki pages that linked out with ../../Migration_Plan.md now use the sibling page form — those links would have 404'd on the published Wiki, which is where those pages are actually read.

The four documents left at the root are exactly the four that are maintained: README, TODO, REVIEW, CHANGELOG.

  • Migration-Plan disagreed with itself in four places about how much of it was left (2026-08-29). Asked what remained, the document gave five different answers: the header said "two things", §6's heading said steps 6 and 7, §7 said one of eight, §9 said "the last two", and the risk register said "three of fifteen" above a table with two live rows.

Two of those I caused an hour earlier by retiring the cost gate without following it through the document. The other two had been wrong since 2026-08-28, when T-526 closed §6 step 6 and the Telegram risk row with it. Architecture-Plan §7 already names this failure — "an accurate-when-written count that nobody updated when the list moved past it, which is the most common way a document like this goes quietly stale" — and it happened in the risk register one section below a note saying so.

All five now agree: one thing is left, §6 step 7 / §7's scheduled-job proof (T-518). The two live risk rows are updated rather than closed — the cron/time-zone row because its check no longer needs anything armed, and the cost row because its gate was retired while the risk itself was not.

  • The timer verification script could not succeed, and said so reassuringly (2026-08-29). scripts/cutover/05-verify-timer.ps1 is the tool for Migration-Plan §7's last open gate. It queried requests through az monitor app-insights query --app <id> — both of which the Cutover-Runbook warns against in its own words: "AppRequests is empty and is not the oracle. Zero rows for this app's entire history" (T-514), and "Query the workspace, never az monitor app-insights query --app <appId> … that proxy returns zero rows for every query rather than erroring". An empty table, read through an endpoint that cannot return rows.

Its answer to that was no invocations recorded, followed by "That is the expected result while the flag is false." A reassuring sentence for a query that was never capable of returning anything — the T-514 failure re-enacted inside the tool built to catch it.

Rewritten to read AppTraces from the workspace, and to separate "no evidence" from "evidence of absence": a failed query now says the query failed, an over-cap workspace throws before any result is read as evidence, and a cold app with no traces is reported as inconclusive rather than as a timer that did not fire. Three defects found by inspection before it shipped — a duplicated segment in the workspace name, a category filter that missed the Function.<name>.User rows where disabled — skipping actually lands, and three az calls outside try/catch that would throw past their own error handling on PowerShell 7.4+.

  • Migration-Plan's last two exit criteria are now one, and it was wrong about the other (2026-08-29). §7's scheduled-job gate read "nothing is armed, so nothing has been observed", which conflated two different things. Nothing has run — true, no handler has done work. But every timer has been firing on schedule since deploy: app.timer() registers unconditionally and the flag is checked inside the handler. Since the clock question is entirely about firing, and the host stamps ScheduleStatus with WEBSITE_TIME_ZONE offsets on every invocation, that half is answerable now, from history, with nothing armed and no risk. Only the handler half needs arming. Recorded with the note that a 5-minute timer cannot prove a clock at all — it fires at :00, :05, :10 in every zone — so the check needs a fixed-hour timer, and midnight's CLEANUP_TEMP_STORAGE is both the largest Chicago/UTC gap and dry-run by default.

The cost gate is retired as an exit criterion (owner decision). It required a measurement "before decommissioning"; GCP is dead and Azure is the permanent and only environment, so that moment will not come and the gate can be neither met nor failed. Staying inside budget became a standing requirement on every deployment instead — a higher bar than one reading, not a lower one. Architecture-Plan §8 carries the same correction, and its criterion is deliberately left unstruck: striking it would read as "done and behind us".

  • An API-keys page in the admin portal: paste a credential, never read one back (2026-08-29). Rotating a key meant opening the production Key Vault's firewall to a human IP, running 06-seed-secret.ps1 from a desktop with the Azure CLI signed in, and closing the window again — three steps, one of which leaves production open to the internet if the operator is interrupted. This repository has already made that mistake once. The app never had that problem: it is inside the vault's integration subnet, which network_acls already admits. It was missing permission, not a network path.

Write-only, enforced below the code. The Function App's identity gets a custom role holding exactly one data action — Microsoft.KeyVault/vaults/secrets/setSecret/action — and not getSecret, delete or purge. Key Vault Secrets Officer would have been one line of Terraform and would have granted all of them. A promise the platform does not enforce is one refactor away from being false, so a future change that tries to read a value back gets a 403 rather than a secret. The honest limit, stated in the code: a secret that has resolved into an app setting is in process.env by definition, so this hides other secrets, other versions, and anything with no reference — not the values the app is actively using.

Terraform stays the source of truth, for what it was already the source of truth for. It declares which credentials exist and how the app finds them — 21 @Microsoft.KeyVault(SecretUri=…) references. It must not hold values: the azapi_resource_action that reads the settings back exports the entire live map into state and plan JSON unredacted, which is what app-settings-secrets.test.js exists to prevent. secret-catalog.js is checked against main.tf in CI, pair by pair, so the page can never offer a secret the application cannot read, and the UPPER_SNAKE ↔ UPPER-KEBAB translation that REVIEW.md §4.5 warns about cannot drift.

Four lights, where three were asked for. Gray (never seeded), green (live), red (resolved but the upstream service rejects it) — and amber, because App Service caches Key Vault references and "refetches them every 24 hours". For a while after a paste the vault has the new value and the running worker does not. Green there would claim a rotation had taken effect when it had not; gray would say "never inserted" one second after inserting it. Amber is computed from a fact rather than a guess: environment variables are materialised at process start, so a write later than that cannot be reflected yet. A Microsoft.Web/sites/config/write grant, scoped to the one site and deliberately excluding config/list/action (the action that reads settings back), lets the app force the platform to re-resolve, so new workers pick a key up immediately instead of within a day. That call is best-effort: if it is refused, the secret is already written and the only cost is the slower cycle.

Red is real, not decorative. secrets-health.js says in its own header that it cannot see "a setting whose reference resolves to the WRONG secret … only the upstream service can say it is wrong". The AI router now says it: it already separated a rejected key (401/403) from a bad request, and reports that verdict. 404 and 429 are deliberately not reported — a wrong model id and a busy account are not a bad credential, and would send an operator rotating a key that is fine. Successes are recorded once per worker per provider, so the bookkeeping stays off the hot path of every generation. Only the three router providers may claim a liveness check, and a test binds the two lists: the page prints "no liveness check for this one" beside every green light that has none, rather than implying a check it does not run.

Ten mutants confirm the guards fail closed, including two that survived a first pass and exposed real gaps: reading the environment before the write timestamp (which made a first paste show gray — the exact "looks broken" failure amber prevents), and spreading the state record into the response instead of naming its fields.

  • GCP pricing moves from a service-account JSON to the API key Google documents, and takes three dependencies out with it (2026-08-29). The Cloud Billing Catalog API serves the public price list. Google's own guide says what it wants: "Before you can use the Cloud Billing Catalog API, you'll need to enable the Cloud Billing API and get an API key." This estate was doing a signed-JWT token exchange instead.

What that cost. The credential was a ~2.3 KB multi-line service-account JSON, which cannot be an app setting, so it needed a run-time vault client to fetch it — and that client needed @azure/keyvault-secrets, a KEY_VAULT_URI app setting, a data-plane RBAC grant on the production vault, and functions/src/lib/key-vault.js. The token exchange needed google-auth-library. The --file seeding requirement needed scripts/cutover/03-keyvault-secrets.ps1, a whole script for one secret. Six moving parts, one caller, to read prices anyone can read.

What replaced it. GCP_BILLING_API_KEY, an app setting holding @Microsoft.KeyVault(…secrets/GCP-BILLING-API-KEY) like every other credential here, read through readKey — which already treats an unresolved reference as absent. The key travels as a key query parameter on both the services list and every SKU page; there is no Authorization header left in the module. Absence still throws, unchanged and deliberately: returning null would be indistinguishable from "GCP has no price for this service" and would quietly become a baseline row, and a comparison table showing stale numbers is worse than one that errors.

What is gone. functions/src/lib/key-vault.js and its test (zero call sites once gcp.js stopped importing getSecret), @azure/keyvault-secrets, google-auth-library, the KEY_VAULT_URI app setting, and scripts/cutover/03-keyvault-secrets.ps1 — superseded by 06-seed-secret.ps1, which seeds any secret and now has no exception to carry. Every secret in the estate reaches the app the same way, which is why app-settings-secrets.test.js's allowlist of secret-shaped settings that are not vault references is now empty. main.tf declares 21 references and no run-time reads.

This module is an information tool, not a billing integration. It is read by the public pricing comparison so a visitor can see three clouds side by side; this estate runs on Azure and is not charged through anything here. The earlier design treated a public catalog lookup as if it were account access, and the grant, the SDK and the script all followed from that one mistake.

Four mutants confirm the new guards fail closed: bypassing readKey so an unresolved @Microsoft.KeyVault(…) string goes out as ?key=, removing the throw on absence, dropping the key from the SKU page URL, and putting it back in a bearer header. Each is caught.

  • The architecture review's remaining Medium and Low findings, 20 more closed (2026-08-28; #257, #258 and this change). The review's own tally goes 35 → 55 of 62. What is left is seven: T-714 (an owner decision on hydration), T-718 and T-728 (each needs an Azure apply to mean anything), T-719 and T-721 (measurements and a cost decision), T-749 (a workspace flip), and T-754 (splitting main.tf, which the review itself asks for in a PR whose plan shows nothing else).

The gate was not checking what it was believed to check. vps-agent — the component that shells to docker run with a payload from the platform API and holds a certificate on a third-party VPS — had its CI row run npm ci and stop, so an edit dropping --network none shipped green. Its sandbox flag list is the entire security boundary and now has 37 tests over the argv it produces, plus a refusal for any capability that tries to set a sandbox-controlled flag: Docker is last-wins, so extraDockerArgs could previously have overridden the boundary silently. Job images move from mutable tags to digests, because docker run pulls before any sandbox flag applies. Separately, the harness check drove every node through the unavailable-agent branch, so the handoff validator it exists to protect was never called — root cause being that no agent definition is committed, so no node could ever be available.

Two unbounded runtime paths got bounds. The change feed handed 50 documents to an invocation in which one content document can require four Replicate generations, an inspection, a caption and a Publer call; it now takes 8 and carries a wall-clock budget that throws rather than returns, because returning would advance the lease past documents it never looked at and their triggers would never fire again. And the daily forge budget — the system's only AI-spend ceiling — was enforced in one place, against a count written by a best-effort write that swallows its own failures, computed once before a loop so a concurrent manual forge was invisible, and incremented after the model calls so a run killed mid-flight spent tokens the ledger never recorded. It is now a server-side compare-and-increment taken before any model call.

Two live user-visible frontend bugs. An anonymous visitor to /{provider}/news downloaded and executed 236 kB of MSAL to look at a news grid: the hook's own header claimed its role gate prevented that, but a runtime gate cannot stop a bundler resolving a static import. Measured against real builds, the route's static closure drops 1,060,504 → 816,158 bytes. And four copies of provider canonicalisation had drifted — one knew vmware/ansible, another did not — so those documents appeared on the landing page and vanished from their own provider's blog list.

Claims became checks. outputs.tf said sensitive key outputs are "intentionally omitted" eleven lines above one that is not. A comment said a plan of a particular shape "and nothing else means NO DRIFT", where "and nothing else" is the whole claim and the part a human reading a summary line cannot verify — three destroys look like three destroys whichever three they are. The azapi read-back exports the entire live app-settings map into state, safe only while every secret-shaped setting is a Key Vault reference, which nothing asserted. Each is now a test or a program, and the two genuinely accepted exposures — the SWA token and cloudflare_origin_secret — are recorded in REVIEW.md with their rotation consequences rather than left to be rediscovered.

Four Key Vault failures stopped being one symptom. Unseeded, RBAC revoked, firewall denying, and rotated-then-broken all arrive as the literal @Microsoft.KeyVault(…) string, which the code correctly treats as "no key" — so a feature turns itself off in production, indefinitely, with no exception in Application Insights because the path taken is a clean fallback. /api/health now reports how many settings are in that state: a count rather than the names, because that endpoint is anonymous and T-402 already established that an unauthenticated inventory is what host enumeration looks for.

Three things the review got wrong, and one it under-stated. T-757 claimed the frontend gate runs an admin subset; test:admin is plain vitest run with no include filter and has been since T-320 closed exactly that — the finding was inferred from the script's name, so the name is gone. T-731's prescribed "return early" would have silently dropped documents. T-724's recommended CI check cannot be written without a TFC token the repository does not have. And T-738 under-stated itself: normalizeContentProvider matched exact keys, so "Microsoft Azure" became microsoftazure and fed getContentPublicPath, building public URLs no route serves — for the normal case, since multi-word provider fields are what documents actually carry.

Mutation testing found two tests that were green while checking nothing, and both are worth recording because the failure mode is invisible in review. The sandbox test compared each flag against the module's own constant — the same one a regression would edit — so --user 0:0 passed. The route-shadowing guard compared absolute paths to each other, so reintroducing /terraform/code did not fail, because it duplicates a relative child of /:provider, which is the actual shape of the bug. A third was found by updating it: the curated-image hook's tests were exercising the new fallback path rather than the new path, because the batched function was missing from the mock factory.

  • The architecture review, and 35 of its 62 findings fixed (T-701–T-762, 2026-08-28; #249 records, #250 remediates). Six specialist reviews, one per technology layer, run against merged main: Azure platform, Terraform IaC, backend Functions, frontend React, CI/CD, and the remaining ops surfaces (Cloudflare Worker, PowerShell scripts, Python harness, VPS agent). Every finding carries a file:line anchor; the review of record, with each finding's failure mode, recommendation and outcome, is wiki/Architecture-Review-2026-08.md.

All five Critical findings are closed. An editor could publish content live by enqueuing a job, because registerJobType defaulted the required role to editor and no job type declared one — so publish-content bypassed the publisher gate its HTTP twin enforces. Both PowerShell confirmation gates self-approved whenever stdin was not a TTY, which made bootstrap-terraform-oidc.ps1 -ElevateAccess < /dev/null reach tenant-root User Access Administrator unprompted; the code that then removes that grant reported "removed (verified)" over a grant that might still be live, because its read-back could not tell a failed call from an empty result. The T-526 cutover script's -WhatIf mutated the production Key Vault rather than simulating, and its "custom secret: set" line was unconditionally true — both fixed, so T-526 is now safe to run. And the production environment gates nothing while workflow_dispatch accepts any ref, which is now backstopped by a main-only guard in both deploy workflows.

Data durability. Media storage moves LRS → RA-GRS: ADR 0018 accepted LRS only "while the Firebase source retains the authoritative copy", and ADR 0023 removed that copy. RA-GRS rather than ZRS deliberately — the risk is account and regional loss, which zone redundancy does not cover, and LRS→ZRS is not Terraform-expressible. Cosmos backup moves to the 30-day tier, and prevent_destroy now covers the database and containers, which are generated from a spec file with immutable partition keys.

Also fixed: jobs stranded in running forever, a 2,000-point-read fan-out reachable from any Telegram message, missing timeouts on every outbound call in the change-feed path, a Telegram send failure that produced duplicate publish jobs, an SSRF gap on scraped image URLs, an alert window with no ingestion-lag headroom, and a 297 kB reduction in what every page preloads.

Two of the review's own findings were wrong, and that is recorded rather than quietly dropped. T-750 asked for CORS origins to be derived from variables; doing so broke a guard that reads those literals as text to keep the platform and application allowlists in sync — the drift it prevents is a dated outage. T-715 correctly diagnosed a 456 kB bundle on the critical path but prescribed a fix that rolldown ignores; measurement found what actually worked.

One finding exposed something larger than itself. T-747 — the lowest-value item in the review, "add a directory to Dependabot, zero impact today" — made Dependabot re-validate its config, which failed: an unsupported property on the terraform entry had invalidated the whole file, so no ecosystem had been receiving dependency updates at all. Latent because Dependabot re-validates only when the file changes.

  • Five of the Blog Machine's six backlog items landed after the program closed (2026-08-28; #243–#248). Recorded here on 2026-08-28 for the same reason as the entry below: they shipped without changelog entries. The backlog was written as "unique and cool, deliberately unscheduled", and then most of it turned out to be small once the program's machinery existed — which is the useful fact, not the features individually.

Four of them were dead frontend calls, not new ideas. The admin UI had been posting at generateSocialCaption, generatePreviewImages, generateCuratedArticleImage, generateReviewHeroImage, triggerAiImageGeneration and generateReviewerDigestManual since the import, and every one returned a 404. Each is now implemented against the contract the caller already sends, and each shares an existing core rather than growing a second implementation: the four image RPCs call ai-cover's extracted generateCoversForContent, so a manual hero and an automatic one are indistinguishable in the document; the digest RPC calls the 07:00 timer's own snapshot with generatedBy: 'manual', merging into the same workflow_digests/{date} document. Backlog item 3 also carried a wrong guess about its own contract — the "Run Now" tile is the reviewer queue snapshot, not the newsletter job — and the correction is recorded on the wiki page rather than quietly implemented.

Publishing now queues its own social post. A first live publish arms socialCaptionTrigger once per document (a republish never posts again, and staging posts nothing), the change feed generates a practitioner-voiced caption and bulk-schedules it in Publer after a delay that defaults to 60 minutes — deliberately, as the undo window. It is switched entirely by the owner-seeded admin_config/social_autopost: absent or disabled means no model call at all, and Publer unconfigured degrades to a Social Hub draft.

The forge proposes its own interlinking. After post-processing but before grading, so the grader sees the real article, every published post is scored against the draft with Jaccard over title and keyTopics tokens, reusing the dedupe gate's own tokenizer. The top three above a 0.15 floor become a deletable "related reading" module plus relatedContentIds series metadata. Only posts with a resolvable public URL are ever proposed — a link the reader cannot follow is worse than no link — and no score ceiling is needed, because true duplicates never reach this code: the dedupe gate already refused to forge them.

And the grader gained an advisory SEO lint (lib/content/seo-lint.js, no model call): summary length against the 70–160 meta-description window, slug/key-topic alignment against the real slugify the publisher uses, and heading hierarchy with code fences and module tags stripped first. Strictly advisory — findings never move the overall grade, the staging decision or any publish gate. They surface where the approve decision is actually made: as a bullet list in the preview banner and a one-line count in the forge_ready Telegram note.

Backlog item 4, analytics-informed topic weighting, is not landed and is not an engineering task: no engagement data source exists anywhere in the system. It needs an owner to pick an analytics provider before it is buildable at all.

  • Reachability gets an alert, from the edge rather than from Azure (T-519 repository half, 2026-08-27; #234, ADR 0024). Recorded here on 2026-08-28 — it shipped without a changelog entry, which is the gap this documentation pass found. Reachability was the one signal with nothing behind it, and it is the signal that matters most: every other alert needs the app healthy enough to emit telemetry, so it is the only one that survives the app being completely down. Azure's own availability test could not provide it. Cloudflare's Bot Fight Mode answers datacenter clients — which is exactly what Azure's availability agents are — with a 403 interstitial on https://api-azure.<domain>/api/health, and the WAF skip rule written against it was built, applied and confirmed inert, because Bot Fight Mode does not run on the Ruleset Engine and is not configurable on the free plan.

ADR 0024 routes around it instead of waiting on it. edge/availability-probe is a Cloudflare Worker on a five-minute cron whose same-zone subrequest is never challenged; it reports every attempt to Application Insights, and edge_probe_availability counts successes and fires below two in a fifteen-minute window. Counting successes rather than failures is the point: it makes a dead probe and an unreachable API the same incident, where a failure-counting rule would go quiet on both. Both the alert and the original standard web test stay gated (availability_probe_alert_enabled and availability_test_enabled, each defaulting to false), so the repository ships armed-but-inert; deploying the Worker and seeding its connection string are owner actions, and the procedure is wiki/Availability-Probe.md. The web test is kept in Terraform, disarmed, for the day the Cloudflare plan is upgraded.

  • The Blog Machine (T-601–T-607 closed, 2026-08-28; #236–#242 + the close-out PR). One initiative, seven phases, each one PR, turning the admin portal into a content engine around the already-working forge pipeline: check posts in the queue or paste a URL, and a professional, visually rich post in the owner's voice comes out the other end — announced on Telegram with a signed staging link, approved with one reply, published live through the full gated pipeline. The program of record with per-phase as-built notes is wiki/Blog-Machine.md.

What landed, phase by phase: the forge's only entry point fixed (its payload key had always been wrong — the test asserted the job type, not the key; both are pinned now) and modular posts regained syntax highlighting (T-601, #236). Paste-a-URL: generateArticleDraft HTTP-direct plus the unattended forge-from-url job, with the voice/format prompt extracted to one voice.js builder (T-602, #237). "Forge Selected (n)" on the review queue with select-all and grade/provenance badges (T-603, #238). Forge Studio — wordSoup, weighted interest areas, master prompt, banned phrases, style rules, publish threshold, autoForge — over new getForgeConfig/ updateForgeConfig RPCs, with voice calibration as accept/dismiss suggestions never merged automatically; retired the manual-Cosmos-seeding requirement (T-604, #239). Five rich module types (pull_quote, stat_board, comparison, timeline, callout) across the shared grammar, validation/repair, renderers, editor and forge prompt, fixing the design-module round-trip data loss on the way; MAX_MODULES 10→14 with list-contract tests on both sides (T-605, #240). The approval loop: the HMAC-signed /api/public/preview/{id}?t=… route (72 h, byte-identical 404 for every refusal) with the /preview/:id page, the forge_ready rising-edge Telegram notification, /approve publishing through the injected processPublishContent (every gate applies) and /reject running the extracted state-machine core — plus curated default heroes when AI cover generation fails (T-606, #241). And throughput: one forge_stats.today ledger enforcing autoForge.dailyLimit across scheduled and manual forging alike, interest-weighted candidate ranking, failure-only job notifications via a new jobs onComplete hook, grade sorting and a forged-today meter on the queue, and a staged-for-approval list in the bot's /queue (T-607, #242 + close-out).

Two live bugs fixed in passing, both of the same shape — a call that looked right and a test that asserted the wrong half: the bot's /ack//resolve had never persisted (alertId passed as patchDoc's updates argument), and /forge had never enqueued a runnable job.

Activation is owner-gated and deliberate: T-526 (webhook re-registration — the loop is silent until then), T-518 (timer arming), PREVIEW-SIGNING-SECRET and REPLICATE-API-KEY seeding, and the ~8 default-hero uploads. The code ships dark and arms itself as each gate opens; nothing publishes without the owner's explicit /approve from the verified chat.

  • The alert fabric can now be verified against what is actually deployed. .github/workflows/verify-alert-state.yml reads the three workload rules through ARM with the existing OIDC deployment identity and prints autoMitigate, mute duration, frequency, window and severity into the job summary; a stateless or missing rule fails the run and is named in the table.

It closes a gap that only became visible while trying to confirm the previous entry had taken effect. Applies run in HCP Terraform Cloud on a human's confirmation, and a green TFC run proves ARM accepted the change — not that the deployed rule behaves differently. Those come apart exactly at autoMitigate, the attribute that decides whether a firing rule mails once or every five minutes, and which is invisible from the repository, from CI and from the TFC run list. Nothing short of asking ARM answers it.

The identity's new grant is Monitoring Reader on rg-web-site-prod-cus (infra/oidc.tf), chosen over Reader deliberately: both carry the */read the operation needs, and Monitoring Reader does not carry listKeys on the workspace — so the identity cannot read ingestion keys and cannot forge or drown the telemetry the rules evaluate. That is the same reasoning that chose Log Analytics Reader for the alert rules' own identities in ADR 0022 decision 4. Scope is the resource group rather than the individual rules, so renaming a rule cannot silently break the verification path.

Two constraints shaped it. The job declares no environment: — doing so changes the OIDC subject GitHub presents from ref: to environment:, which the branch credential cannot match (the AADSTS700213 failure documented in infra/oidc.tf). And it is workflow_dispatch only: this reads production, so a human decides when.

Note the sequencing — the workflow cannot report until the apply carrying its own role assignment has landed. Until then it fails at azure/login or on the read, which is the self-arming behaviour the vars.CLIENT_ID gate already models elsewhere.

  • Listen & Learn (T-411). One study podcast per weighted skill area of a certification's official study guide, with the videos worth watching next. Ported from Site-Main functions/listen-and-learn/ (088f458) with the pipeline intact — study guide → skill areas → videos per area → dialogue → MP3 → draft — and three deliberate departures:

Audio is Gemini TTS, on the key the site already holds. Upstream authenticated to Cloud Text-to-Speech with Application Default Credentials, a GCP identity this Function App cannot hold — the same reason Vertex was dropped from the AI router. The replacement is the Gemini API's own multi-speaker TTS through GEMINI_API_KEY, which is the capability the feature was actually for: a two-host deep dive read from source material is what a NotebookLM audio overview is, and these are the models that produce it. It costs no new service, no new resource and no new credential. Its contract shapes the module in three ways: multi-speaker takes at most two speakers, which is exactly the number here; the dialogue is a prompt rather than markup, so the transcript's speaker labels must match the speech_config names or a label is read aloud instead of switching voice; and a 32k-token session comfortably holds a whole 9,000-byte script, so an episode is one request with nothing to chunk.

Episode audio is encoded to MP3, and that is not cosmetic. Gemini returns headerless 24 kHz 16-bit mono PCM with no format option — 48 KB per second, so a nine-minute episode is about 26 MB. readBlobForDelivery buffers a whole blob into memory and the media route returns it as one body with no range support, so that would hold 26 MB per concurrent listener, bill Function execution for the entire transfer, and make the player wait for the whole file before starting. At 64 kbps mono the same episode is about 4.5 MB (measured 5.8x on a 24 kHz tone). @breezystack/lamejs is the encoder: pure JavaScript, no native build, no dependencies of its own; it is LGPL-3.0 and imported unmodified, and it is the only copyleft dependency in the package.

Azure AI Speech is kept as the fallback, not deleted. Every Gemini TTS model is a preview model, and preview endpoints get retired on notice. Azure Speech is GA, and having the second path written and tested is the difference between a model retirement being a config change and being an outage. It needs a Cognitive Services resource, which is a spend decision, so nothing assumes one exists and an unseeded AZURE_SPEECH_KEY simply means the provider is not offered. Its own hazard is pinned by tests: that REST API silently TRUNCATES at ten minutes of audio rather than erroring, so its dialogue chunking estimates duration with a slow voice — being wrong the other way deletes the end of an episode and looks like a complete one.

Provider selection follows the AI router's rule — a key makes a provider possible, and the first configured one in preference order runs — with LISTEN_AND_LEARN_TTS_PROVIDER to pin one outright. A pin that is not configured FAILS rather than falling through, because falling through would produce episodes in a voice nobody chose. Both providers return MP3, so the blob path, the stored contentType and the <audio> element are identical whichever ran. Each episode records speechProvider, speechModel and durationSeconds — provenance for AI-generated study content, and the only thing that answers "why does this one sound different" after a model change.

The script no longer asserts the hosts' gender. Upstream's prompt said both hosts were women because the Google voices it used were documented female. The Gemini voice list publishes a descriptor per voice and no gender at all, so the claim is not one the audio can keep. The default pairing is by descriptor and follows the roles the prompt already assigns — Kore (Firm) leads and frames, Leda (Youthful) asks the question a learner would ask — and both are overridable per host.

Generation is a job, not a request. An Azure Functions HTTP response is bounded at 230 seconds by the load balancer and one certification is five model calls, five syntheses and five uploads, so the admin page enqueues generate-listen-and-learn and polls, as the RSS ingest does. The run still saves area by area, so a timeout leaves the finished episodes behind.

A missing speech key degrades; a broken one fails. Upstream treated any synthesis failure as a failed area. SpeechNotConfiguredError — no provider configured — now still saves the episode with its transcript, takeaways and videos and records audioError, which the admin page renders in place of the player. Every other synthesis failure still fails the area, because those are faults to fix rather than a state to ship in. The feature is useful the day it deploys and gains audio the day the key lands.

Approval is unchanged and is the point: episodes are AI-written summaries of a paid exam's objectives, generated as drafts, and GET /api/public/listen-and-learn filters on status === 'published' with an equality test — an unrecognised status stays hidden. listen_and_learn and listen_and_learn_episodes already existed in the Cosmos container spec; listenandlearn is a new private blob container served through the media route. GitHub exams are enabled alongside Azure and AWS, since they are hosted on Microsoft Learn and parse with the same adapter. - draw.io hotspot tooling (T-410). lib/drawio/parseDrawio.js and lib/drawio/hotspotGeometry.js port from Site-Main unchanged, with their test and fixture. A hotspot now stores a draw.io shape id and its position is derived from the diagram XML on every render (useResolvedHotspots), so re-uploading an edited diagram moves every pin with its shape instead of stranding it. DiagramSourcePanel replaces typing x/y percentages into two number inputs and eyeballing the result. Nothing renders the .drawio file — vendor stencils are most of what makes a cloud diagram readable, and the exported image stays what visitors see. Hand-positioned hotspots, the only kind this repository could write before, pass through untouched, and the admin preview now resolves through the same code as the public page.

  • The data-migration workflow can no longer publish production data. The repository is public, and migrate-data.yml uploaded scripts/reports/ — document ids and 240-character field samples — as a workflow artifact, while the import dry-run printed document samples to the job log. Every migration script now writes a *.summary.json (counts, container names, warning tallies) beside its full report; only summaries are uploaded, with 1-day retention; the export lives in $RUNNER_TEMP and dies with the runner; MIGRATION_CI=1 makes --show-samples an error; and the upload step refuses any non-summary JSON it finds.
  • No stored credential on either cloud for the migration. Firestore and GCS reads authenticate through Workload Identity Federation (google-github-actions/auth + applicationDefault()); connectFirestore() refuses a service_account key file in CI, connectCosmos() refuses to start if COSMOS_KEY is set, and FIREBASE_SERVICE_ACCOUNT_JSON is retired before it was ever provisioned. The azcopy storage script — whose GCS source accepts only a downloaded key — was replaced by a Node copier on the same federated credentials.
  • Production Cosmos is locked by RBAC, not by a YAML guard. The deploy identity's database-scope Data Contributor and blob-write roles on production exist only behind migration_writer_enabled (default false); the workflow's refusal of target=production for write modes is the second lock, not the only one.

  • Every GitHub Actions reference pinned to a commit SHA — 35 uses: lines across all 12 workflows. A tag is a mutable pointer: whoever controls the action's repository can move @v4 to different code at any time, and the March 2026 Aqua incident that broke this repository's Trivy step was exactly that. CodeQL's Actions pack raises one alert per unpinned third-party reference, so this closes that class of finding as well as the real supply-chain exposure. Each pin carries the version it was cut from as a trailing comment (@ff2f1c6… # v4), because a bare 40-character hex string tells a reviewer nothing; Dependabot's github-actions ecosystem was already configured and keeps SHA pins current the same way it keeps tags current. The repository standard already required this — one actions/checkout reference had been pinned and the other eleven had not.

  • Anonymously submitted HTML is sanitized on ingest. overviewHtml arrives through the anonymous submission endpoint, is stored, and is eventually rendered with dangerouslySetInnerHTML on a public template — with a single client-side DOMPurify.sanitize() call standing between those two facts. Sanitizing at ingest makes safety a property of the stored data rather than of one component's rendering choice; the client-side call stays, because two layers is the point. No dependency was added: the sanitizer uses cheerio, already in the tree, so it parses rather than pattern-matching markup. (TODO.md T-408)

  • The Content-Security-Policy stopped granting the Firebase/GCP surface, and started granting Entra. connect-src still allowed *.googleapis.com, *.firebaseio.com, *.cloudfunctions.net, *.run.app and wss://*.firebaseio.com long after the last Firebase import was deleted, plus *.documents.azure.com, which contradicts the rule that the browser never holds a Cosmos client. More consequential in the other direction: login.microsoftonline.com was absent from connect-src and frame-src, so admin sign-in and MSAL's silent token renewal could not have worked at all. Pinned by csp.test.js, since a CSP failure appears only in a browser console on a deployed site. (TODO.md T-404)
  • Key Vault failures are no longer indistinguishable from a missing secret. Throttled, RBAC-denied and unreachable all returned null, the same value as "this secret does not exist" — and the one caller turns null into an error message naming the wrong cause. null now means absent and everything else throws, carrying the real reason. Reads are cached for five minutes, so a hot path no longer spends the vault's request budget on a value that changes approximately never. (TODO.md T-405)
  • GET /api/health stopped reporting the runtime. It returned process.version and the deployment name to anyone — an unauthenticated inventory of exactly what an attacker enumerates first, and of no use to a liveness probe. (TODO.md T-402)
  • The role cache is bounded. It is only reachable after a token verifies, so an anonymous caller could not grow it, but it had no eviction at all and grew with every distinct principal that ever signed in. (TODO.md T-408)
  • Authorization-denial auditing is pinned by a test. The admin_audit_logs writer already existed in the guard's production composition — the finding was stale — but nothing failed if the line were deleted. Now something does. (TODO.md T-406)
  • The Cosmos account primary key is out of app settings. COSMOS_CONNECTION_STRING carried it — readable by anyone with Contributor on the resource group, and present in Terraform state — for the sole benefit of a change-feed trigger binding whose two handlers were empty TODOs that nonetheless ran continuously and billed lease-container RU. The handlers, the registrations and the setting are all gone, and the route-inventory test now asserts zero change-feed registrations so reinstating one is a visible decision. It was also masking a real risk: the binding kept working off the key while cosmos-client.js, which uses managed identity, would have returned 403 on every call if its role assignment were wrong — a half-working app is harder to diagnose than a uniformly broken one. (TODO.md T-315)
  • The Labs VPS agent no longer holds a database credential. It ran on a third-party host with a Cosmos account primary key — read/write over all 71 containers. It now authenticates to the Functions API with an Entra certificate and can reach three endpoints, each constrained server-side: claims are limited to the job types its registry document lists, results can only be written for jobs it currently holds, and cancelled is not a status it may report. Revocation is a field on the registry document and takes effect on the next call, with no cache in between. The rejected alternative and what still needs provisioning are recorded in REVIEW.md §0.4. (TODO.md T-401)
  • CORS applied to every route. lib/auth/http-route.js is now the single registration helper for all 59 HTTP routes: it registers OPTIONS, evaluates CORS before the handler runs, and merges the headers onto every response including errors. Previously cors.evaluate was called by one route of fifty-eight, and the advertised method list predated the REST surface, so a browser preflighting any of the fourteen PUT/PATCH/DELETE routes would have refused to send. (TODO.md T-102)
  • Route-inventory test added — the replacement for the firestore.rules default-deny catch-all that Azure has no equivalent of, and the test require-role.js declared in its header and never had. Every registration must be guarded or named in an explicit eight-entry public allowlist, must accept OPTIONS, and must evaluate CORS. Verified by mutation: an unguarded route and a raw app.http registration both fail it. (TODO.md T-103)
  • Dependency advisories cleared — dompurify to ^3.4.13 (moderate: XSS via detached subtree after IN_PLACE hook removal; ships in the app bundle), nanoid override ^3.3.18 (high), js-yaml override ^4.3.1 (high). Both packages report zero advisories. (#67)
  • Anonymous write path closed — public submissions now pass server-side validation and quota enforcement instead of writing directly to the content collection from the browser. (#45, #66)
  • oauthToken made write-only on every read path for mcp_servers. (#47, #62)
  • Upload path hardened — container allowlist, traversal-resistant blob path validation, and a decoded size cap enforced before storage is touched. (#62)
  • Snapshot endpoint allowlisted to certifications and speakerevents so it cannot become a generic container read. (#45)
  • speakerevents snapshots no longer publish admin emails or hidden events. SANITIZERS had a certifications entry and none for speakerevents, so raw rows were written into _snapshots/speakerevents and served anonymously — including createdBy/updatedBy, which carry the email of every admin who touched an event, and display: false records whose only filter was client-side. A positive field allowlist now governs what is published, and getSnapshot strips internal fields inside items[] rather than on the wrapper alone. A test asserts every snapshot collection has a sanitizer. Takes effect on the next publishSnapshot run — an already-published snapshot keeps its contents until then. (TODO.md T-201)
  • Soft-deleted podcasts, cache documents and AI insights are no longer served anonymously. listPodcasts and getFeed applied no deletion filter, and ai_insights was filtered on active !== false only — so a soft-deleted insight still reached the news feed. isSoftDeleted was extracted as the portion of isPublicDocument that applies to collections with no editorial workflow, and both handlers now use it. The full predicate was deliberately not applied: these three collections carry no publication status, so it would have emptied the podcasts page, the news feed and the insights panel. (TODO.md T-202)
  • The anonymous feed endpoint is bounded. getFeed ran SELECT * FROM c WHERE c.provider = @provider against both rss_cache and ai_insights with no ceiling, and queryDocs calls .fetchAll(). Both now cap at 200 documents — a runaway guard, not a page size: one rss_cache document is one feed, so sizing the bound to the 30 items the client renders would have dropped whole feeds. Items within a document remain unbounded, tracked as T-319. (TODO.md T-203)
  • Point reads against the four non-/id containers now fail loudly. readDoc/patchDoc/deleteDoc/replaceDocIfMatch defaulted the partition key to the document id, which for content_versions, image_prompt_sets_prompts, image_prompts_sets and listen_and_learn_episodes reads the wrong logical partition and returns nothing — surfacing as a permanent null. They now throw unless given an explicit key, and a test keeps the map in step with infra/cosmos-containers.json. (TODO.md T-313)
  • putConfig no longer deletes stored OAuth tokens. It is a full replace and reads never return oauthToken, so any read-modify-write round trip from an edit form would have wiped it. The token is carried forward unless explicitly supplied; an explicit empty string still revokes. The read-side hasOauthToken boolean is stripped from incoming bodies. (TODO.md T-314)
  • cms/content list rejects a malformed limit. ?limit=abc produced TOP NaN — a 500 carrying raw Cosmos error text — and ?limit=0 produced a silently empty list. Clamped like its four siblings, and error.message no longer reaches the client on any of the file's 500 paths. (TODO.md T-310)
  • deleteSetArtifacts queries one logical partition instead of fanning out across all of them; queryDocs gained an optional partitionKey. (TODO.md T-312)
  • Uploads no longer accept an arbitrary content type. contentType was taken verbatim from the body and stored as the blob's Content-Type, which the media route serves back: an editor could host evil.html as text/html on an org-owned domain. Six image types are now allowed, each having to agree with the path's extension — badge.png declared text/html and evil.html declared image/png are both refused. image/svg+xml is accepted only into containers the anonymous route does not serve, since an SVG on a public URL is a scriptable document in the storage origin and nosniff does not address a type that was declared rather than guessed. (TODO.md T-307)
  • A caller-chosen upload path can no longer replace a live asset. Uploads from the admin route are conditioned on If-None-Match: * and answer 409 instead of overwriting; uploadBlob's default is unchanged, so the paths that rewrite deterministic keys on purpose still do. The condition is asserted against a mocked SDK rather than only against the handler's fake storage — that fake is what let T-104 stay green while every real upload threw. (TODO.md T-307)
  • Upload size is checked before memory is committed. The 413 came after a full JSON parse, a full base64 string and a full Buffer decode — roughly a 250 MB peak for a 100 MB body on a 2048 MB instance. Content-Length is now checked before the body is read and dataBase64.length before it is decoded, with the decoded count still the final authority. There is no http.maxRequestBodySize in host.json to complement this; the v2+ extensions.http schema has no such key, so the anonymous submissions parse still needs its own check. (TODO.md T-306)

Infrastructure

  • Storage: Storage Blob Delegator role assignment for user-delegation SAS; media containers declared private, matching the account-level override that already made them so.

Authored but never applied — no Terraform validate, plan, or apply had run from any session (recorded at the time in REVIEW.md §1.1, since retired).

  • Cosmos DB serverless container specification (71 containers).
  • Flex Consumption plan and pricing work. (#38, #41, #42)
  • Container Apps Job definition for the CI runner. (#48)

Notes

  • Work merged before the SOP was adopted has been reconstructed from pull request history; entries reference PR numbers rather than release tags.
  • Nothing in this file has been verified against a deployed environment.