Infrastructure

• Introduction & Scope
• Chapter 1 — Introduction & Scope
• Infrastructure Overview
• Chapter 2 — Infrastructure Overview
• Network Architecture
• Chapter 3 — Network Architecture
• DNS & Domain Architecture
• Chapter 4 — DNS & Domain Architecture
• TLS & Edge Security
• Chapter 5 — TLS & Edge Security
• Compute & Application Layer
• Chapter 6 — Compute & Application Layer
• Storage & Data Layer
• Chapter 7 — Storage & Data Layer
• Secrets Management
• Chapter 8 — Secrets Management
• Identity & Access
• Chapter 9 — Identity & Access
• Infrastructure as Code
• Chapter 10 — Infrastructure as Code
• Supply-Chain Controls
• Chapter 11 — Supply-Chain Controls
• Observability
• Chapter 12 — Observability
• Audit Infrastructure
• Chapter 13 — Audit Infrastructure
• Backup & Disaster Recovery
• Chapter 14 — Backup & Disaster Recovery
• Capacity & Scaling
• Chapter 15 — Capacity & Scaling
• Maintenance & Patching
• Chapter 17 — Maintenance & Patching
• Appendices
• Chapter 18 — Appendices

Introduction & Scope

Chapter 1 — Introduction & Scope

1.1 Document Purpose

This document (LH-SBS-INF-001) is the authoritative infrastructure specification for The Dash — SBS's Multi-Gated AI Orchestrator Platform, designed by Ledger Hub Networks exclusively for SBS. It defines the complete DigitalOcean hosting environment on which the Platform operates: account and project structure, network topology, DNS and domain architecture, TLS and edge security, compute and storage layers, secrets management, identity integration, infrastructure-as-code practice, observability, audit infrastructure, and backup and disaster recovery.

The specification serves four functions. First, it is the build reference: every infrastructure resource provisioned for the Platform must conform to the configurations stated herein, and any deviation requires a documented change under the governance forms in effect for the engagement (LH-SBS-CO- series). Second, it is the verification baseline: the acceptance gates defined in LH-SBS-INST-001 (V1–V22) test the environment against this document. Third, it carries the Build Register (Appendix, Chapter 18): the tracked checklist of provisioning items, security controls, and open nonconformances between the design target and the as-built state, with per-item completion status. Fourth, it is the operating reference for the IT execution owner (Nelson Santos, IT Director) and any personnel subsequently authorized to administer the environment.

Where the as-built environment deviates from the design target, this document states both: the target configuration (normative) and the current state (informative, dated), with the gap tracked in the Build Register. A statement of current state is not acceptance of it.

This document is 100% technical in content. Commercial terms, fees, and engagement governance are out of scope and are addressed in their respective controlling documents.

1.2 Platform Context

The Dash is a zero-trust, identity-aware platform that runs SBS's routine corporate workflows through isolated, per-function Compartments. Inside each Compartment, three classes of AI agents operate under supervision — Workflow Agents perform the first-pass work, Guardian Agents enforce policy and Risk Tiers, and Supervisor Agents route Medium/High/Critical decisions to named SBS staff. The domain is publicly resolvable but serves a private audience behind SSO and phishing-resistant MFA; every agent decision is written to a tamper-evident, hash-chained audit log.

The Platform is hosted on DigitalOcean under a dedicated SBS account, organized into two DigitalOcean Projects that define the environment topology:

All internal service-to-service traffic is required to remain on private VPC networking. Public exposure is restricted to hardened entry points in two classes.

Tenant surfaces (three-subdomain tenant model):

Operational surfaces:

Operational surfaces carry no tenant data and sit behind the same TLS, reverse-proxy, and firewall controls as the tenant surfaces, with access scoped to administrative roles only under the least-privilege model of the IAM control family.

Each of the ten functional Compartments is isolated across eight layers — frontend, authentication, application, data, storage, secrets, network, and audit — and this specification defines how each isolation layer is realized in infrastructure: firewall rule sets and VPC segmentation (network); per-Compartment separation across the PostgreSQL, MongoDB, and Redis data services (data, storage); Vault namespace separation (secrets); and hash-chained log capture (audit). The infrastructure supporting the audit log — its storage, integrity chain, and retention — is specified in Chapter 13.

The environment follows a zero-trust posture per NIST SP 800-207 and application controls aligned to OWASP ASVS L2, with AI-specific controls addressing the OWASP LLM Top 10 (including LLM01 prompt injection) as carried in LH-SBS-SEC-001.

1.3 Referenced Documents

Where this specification and LH-SBS-SEC-001 address the same control, LH-SBS-SEC-001 states the requirement and this document states the implementation. In the event of conflict, LH-SBS-SEC-001 controls and this document shall be corrected.

Infrastructure Overview

Chapter 2 — Infrastructure Overview

2.1 DigitalOcean Account Structure

The Platform operates under a single dedicated DigitalOcean account owned by SBS (the "SBS Account"). A dedicated account — rather than a shared or Ledger Hub-owned account — keeps billing, resource ownership, and access control unambiguously with SBS and satisfies the tenant-separation intent of the IAM control family at the infrastructure root.

Account-level configuration (target):

• Account ownership — SBS-controlled email alias (not an individual mailbox), so ownership survives personnel changes. Execution owner: Nelson Santos (IT Director).
• Account MFA — phishing-resistant MFA enforced on the account root and on every team member, consistent with IAM-03. No shared credentials.
• Team access — DigitalOcean Teams with role-scoped membership. Ledger Hub Networks platform engineering joins as team members with the minimum role required for build activities; access is reviewed under IAM-06 and revocable by SBS without Ledger Hub involvement.
• API tokens — issued per-purpose (IaC pipeline, monitoring integrations), scoped read or write as required, stored exclusively in Vault (Chapter 8), rotated per the secrets lifecycle. No tokens in repositories, plaintext CI variables, or local files — per INF-02 and INF-03.
• Billing alerts — configured at account level so anomalous resource creation (a common compromise indicator) is surfaced to the account owner immediately.

Nonconformance — credential handling (CRITICAL). As of 2026-07-17 the operational changelog carried plaintext credentials: a shared root password stated to cover all project droplets, the GitHub account, the wiki admin account, and the Grafana admin account. All four credential sets are treated as compromised. Remediation (Build Register item 15): rotate all four immediately; migrate to Vault upon its deployment (Chapter 8) with interim storage in an approved password manager; strip the credentials section from the changelog and purge it from any version-control history; enforce distinct per-service credentials. No shared passwords are permitted for any host or service.

2.2 Project Topology

Resources are organized into two DigitalOcean Projects:

2.2.1 Droplet Inventory (As Built, 2026-07-18)

⚠ Two inventory nonconformances: (a) sbs-wiki region (§2.3); (b) Dev-Gitlab-server exists outside the two-project inventory yet holds production DNS — see §2.2.2.

2.2.2 Nonconformance — Production DNS Target (CRITICAL)

The apex record sbsdash.com → 138.197.222.154 currently resolves to Dev-Gitlab-server, not to sbsdash-server-prod (107.170.72.145). The primary tenant domain therefore points at an uninventoried development host. Required action (Build Register item 16), one of:

• (a) Repoint — move the apex A record to sbsdash-server-prod once its proxy/TLS stack is ready (Chapter 5), and either decommission Dev-Gitlab-server or inventory it properly under the Dev project with its own firewall posture; or
• (b) Hold — if the GitLab box is intentionally serving a temporary landing page, record it in this inventory, apply the Dev firewall posture, and schedule the repoint as a dated cutover.

Until resolved, Dev-Gitlab-server is inside the security boundary of the engagement and subject to all INF-family controls.

2.2.3 Project Topology Rules

• Promotion path — configurations are built and verified in Dev, then promoted to Prod exclusively through the infrastructure-as-code workflow (Chapter 10). No manual, console-driven resource creation in Prod; console access to Prod is break-glass only and logged.
• No shared resources (target) — no droplet, database, volume, or network resource is a member of both environments, and Dev and Prod reside on separate VPC networks with no peering. As built, this rule is not met: Dev, Prod, and monitor droplets all share default-nyc2 (10.100.0.0/20). See §3.1 for the remediation decision.
• No production data in Dev — Dev operates on synthetic or sanitized data only. Tenant data, Compartment data, and audit-log content never leave the Prod boundary.
• Naming convention — new resources carry environment-prefixed names (dev- / prod-). Existing droplet names are grandfathered in the Build Register; renaming may occur at a maintenance window (renames are cosmetic and zero-downtime).

2.3 Region Selection

Primary region: NYC2 (New York).

Rationale:

• Latency to operating footprint — SBS headquarters is in Miami with operations concentrated on the U.S. East Coast and Puerto Rico; the New York regions are the lowest-latency full-service option for the Miami–New York–San Juan corridor.
• Data residency — all tenant data, Compartment data, and audit-log content remain within the continental United States.
• Single-region baseline — both environments deploy in NYC2. Backup archives replicate to SFO3 for geographic redundancy (Chapter 14); no live workload runs outside NYC2.

Nonconformance — sbs-wiki (NYC1). DigitalOcean VPC networks are region-scoped: the NYC1 wiki droplet cannot join any NYC2 VPC. As built, the monitor scrapes the wiki's node_exporter over the public internet, unencrypted, admitted by a ufw rule scoped to the monitor's public IP. Metric content is low-sensitivity (CPU/RAM/disk), so this is an accepted interim state — but it is eliminated, not mitigated, by the wiki rebuild into NYC2 (Build Register item, carried from prior revision). Sequencing: rebuild before VPC enrollment and firewall standardization (Chapter 3); rebuilding after DNS/TLS/proxy work would force rework of all three.

Managed-service availability. NYC2 predates some newer DigitalOcean managed-service capacity. Availability of Managed Databases (PostgreSQL, MongoDB, Redis) in NYC2 is to be confirmed before Chapter 7 is finalized; the fallback design is self-hosted data services under Docker Compose on dedicated droplets within the Prod VPC.

2.4 Environment Topology

Toolchain versions are pinned identically across both environments (Chapter 11) so that Dev verification is meaningful for Prod promotion — version drift between environments is treated as a build defect. The current OS mix (24.04 vs 26.04) is a tracked instance of exactly this class of drift.

2.5 Server Users and Host Access

Target (INF-04): named user accounts per individual; no shared accounts; no direct root login (PermitRootLogin no); SSH key authentication only, password authentication disabled at the daemon; privilege elevation via sudo, logged, without blanket NOPASSWD; authoritative user list maintained in the IaC repository (Chapter 10); administrative access paths and IdP enforcement per Chapter 9.

As built (2026-07-17/18) — nonconformant:

• Root SSH login is enabled fleet-wide and in active use.
• A single SSH keypair (id_ed25519, one operator workstation) is authorized for both root and sbsRoot on all droplets — one shared identity for the entire fleet.
• Shared sbsRoot account (uid 1000) exists on prod, dev, and monitor with NOPASSWD:ALL sudo and docker group membership (docker group membership is root-equivalent).
• Prod is reachable only via jump host through monitor-servers (§3.3) — the one element of current practice retained in the target design.

Remediation (Build Register item 17): create per-person named users with individually generated keys; disable root SSH; replace blanket NOPASSWD with scoped sudo (or passworded sudo); retire or restrict sbsRoot to break-glass with vaulted credentials; capture the authorized-user list in IaC. Until complete, individual attribution of host actions is impossible, which also degrades MON-01 (unified audit trail).

Network Architecture

Chapter 3 — Network Architecture

3.1 VPC Design

Target: two dedicated VPC networks, one per environment, both in NYC2, no peering:

As built: neither dedicated VPC exists. Current membership:

Constraint driving the decision: a DigitalOcean droplet's VPC assignment is fixed at creation — moving a droplet to a new VPC requires snapshot-and-rebuild. Two paths (Build Register item 18):

• (a) Rebuild into dedicated VPCs — recommended. At the current fleet size (4–5 droplets, pre-data-services, pre-tenant-data) rebuilds are cheap: snapshot, re-create in target VPC, re-run configuration. Cost grows sharply once databases and Compartments are live. This path restores the "no shared network" rule of §2.2.3 as a structural guarantee. The wiki NYC2 rebuild is already mandatory, so path (a) folds it into one rebuild wave.
• (b) Accept shared default-nyc2 and compensate with strict host firewalls on every droplet. Dev↔Prod isolation then rests entirely on firewall rulesets rather than network non-adjacency — a weaker claim, and a permanent audit caveat against INF-01.

This specification proceeds on path (a); §3.2 onward describes the target VPCs. If path (b) is elected, §3.1–3.2 are amended by Change Order.

DigitalOcean VPC model notes (unchanged): a VPC is a single flat private network per region — no user-defined subnets, route tables, or security groups; intra-VPC segmentation is enforced by Cloud Firewalls (tag-scoped) plus host firewalls (§3.5, §3.6). VPCs are region-scoped, which is the technical basis for the wiki rebuild requirement.

3.2 Address Plan

Private IPs are recorded in the Build Register per droplet at VPC enrollment; the authoritative address inventory lives in IaC state (Chapter 10), not in this document, to avoid drift.

3.3 Internal Routing and Service Communication

All service-to-service traffic between Prod droplets transits the VPC private interface (eth1), never public IPs.

• Application → data services (target) — the application stack on sbsdash-server-prod reaches PostgreSQL, MongoDB, and Redis exclusively on VPC-private addresses; data services bind to the private interface only, with no public listener (verified at LH-SBS-INST-001 network gates).
• Prometheus scrape paths (deployed, verified 2026-07-17) — monitor-servers scrapes node_exporter on prod (10.100.0.3:9100) and dev (10.100.0.2:9100) over private networking; exporters bind private-only and are unreachable publicly. On the monitor itself, Prometheus (9090), Grafana (3000), and local node_exporter (9100) bind 127.0.0.1 only, fronted by nginx TLS — confirmed by ss -tlnp verification. This matches the target design and is marked verified for the NYC2 members.
• Wiki scrape (interim nonconformance) — sbs-wiki (NYC1) is scraped on its public IP (167.172.135.88:9100), unencrypted, admitted only from the monitor's public IP by ufw rule. Accepted temporarily (metrics only, low sensitivity); eliminated by the NYC2 rebuild, after which the wiki exporter rebinds to its private address like the rest of the fleet.
• Jump-host / bastion pattern (as built, retained in design) — sbsdash-server-prod accepts SSH only via ProxyJump through monitor-servers (ssh -J). This makes the monitor a de facto bastion. Consequences now normative: the monitor carries the tightest SSH admission (admin allowlist, §3.5); its compromise radius includes prod SSH reachability, so it receives priority hardening, patching, and log scrutiny (MON-01/MON-02); bastion SSH sessions are logged. Whether the bastion role remains on the monitor or moves to a dedicated minimal droplet is revisited at the rebuild wave.
• DNS resolution — droplets use systemd-resolved against DigitalOcean resolvers; internal addressing uses IaC-managed /etc/hosts entries (small fixed fleet). Revisit if the fleet exceeds ~10 hosts.

Inter-Compartment note (unchanged): Compartment network isolation is realized within sbsdash-server-prod at the Docker Compose network level — one dedicated compose network per Compartment, no shared bridges, cross-Compartment traffic denied by non-membership. Docker 29.1.3 / Compose 2.40.3 are now present on prod, dev, and monitor (2026-07-18), so this layer is buildable. Container-level detail in Chapter 6.

3.4 Egress Control

DigitalOcean provides no managed NAT/egress gateway; egress control is enforced at the host firewall layer:

• Restricted-egress allowlist at the host firewall on each Prod droplet, permitting: TCP 443 to Anthropic API endpoints; TCP 443 to GitHub (deploy pulls, pinned to published ranges, refreshed by IaC); TCP 443 to OS package mirrors and container registries enumerated in Chapter 11; TCP/UDP 123 (NTP); TCP 443 to Let's Encrypt ACME endpoints; DNS 53 to DigitalOcean resolvers only.
• All other outbound denied and logged — egress-deny lines ship to the observability pipeline (Chapter 12) as a MON-02 anomaly input. Unexpected egress from a Compartment container is a primary indicator for AI-02 (tool allowlisting) and LLM01 (prompt-injection exfiltration) violations.
• Compartment containers inherit a narrower egress set than the host: Anthropic API plus explicitly allowlisted per-Compartment integration endpoints, enforced via per-compose-network firewall chains, IaC-managed and change-controlled.

As built: no egress control exists on any droplet (all outbound open). Egress enforcement lands with the host-firewall baseline (§3.6, Build Register item 20).

3.5 Cloud Firewall Rules (DigitalOcean Cloud Firewalls, tag-scoped)

As built: no Cloud Firewalls exist. Host-level ufw is active on monitor and wiki only; prod and dev have no active firewall at any layer — their public interfaces are protected solely by service bind addresses. This is the most exposed element of the current state and is remediated first in the firewall workstream (Build Register item 19).

Target — four firewalls, attached by droplet tag:

FW prod-app → tag prod (sbsdash-server-prod)

FW prod-monitor → tag grafana (monitor-servers)

FW prod-wiki → tag wiki (sbs-wiki, post-rebuild)

FW dev-app → tag develop — SSH from admin allowlist; web surfaces admit admin allowlist only (no production DNS points at Dev — enforcement of this depends on resolving §2.2.2).

Deviation from prior draft, deliberate: direct public SSH to prod-app is removed in favor of bastion-only SSH, codifying the as-built jump-host pattern — it is the stronger posture. Rule hygiene unchanged: firewalls are IaC-defined; console edits are drift and reverted; the admin SSH allowlist is a named IaC variable reviewed at each IAM-06 access review.

3.6 Host Firewall

Cloud Firewalls filter at the platform edge; a host firewall on each droplet provides defense-in-depth and the per-container egress allowlist (§3.4), which Cloud Firewalls cannot express.

As built: ufw on monitor (22/80/443 in) and wiki (22/80/443 + 9100-from-monitor in); ufw inactive on prod and dev; no egress rules anywhere; no nftables rulesets deployed.

Target: nftables on all droplets — required because per-Compartment egress chains (one chain per compose network) and logged default-deny outbound are beyond ufw's expressible policy. Migration order: (1) activate baseline host firewall on prod and dev immediately (ufw acceptable as a stopgap — inbound mirror of the Cloud Firewall admit set); (2) replace fleet-wide with IaC-rendered nftables rulesets at the rebuild wave; (3) add per-Compartment chains when Compartment compose networks are created (Chapter 6). Base policy: inbound default-deny mirroring the cloud firewall, outbound default-deny with the §3.4 allowlist, established/related admitted. Rulesets are rendered from IaC templates and versioned; the live ruleset hash is a monitored value (MON-01) so tampering surfaces as drift.

3.7 Verification

Network gates (LH-SBS-INST-001): public-listener scan — no data-service or Prometheus ports on public interfaces (passes today on monitor per ss -tlnp evidence; wiki exporter public listener is a tracked exception until rebuild); Dev↔Prod isolation probe — no private path between environments (fails today: shared default-nyc2; passes after rebuild wave); egress-allowlist test — denied-domain attempt logged (not testable today: no egress control); per-Compartment cross-network denial test (pending Chapter 6 build); bastion path test — prod SSH unreachable except via monitor. INF-01 is the controlling SEC-001 requirement throughout.

DNS & Domain Architecture

Chapter 4 — DNS & Domain Architecture

4.1 Domain and Registrar

sbsdash.com is the sole Platform domain. Registrar-level state is a security control surface in its own right (DNS-01):

• Ownership (target) — the domain is registered to SBS (org-level contact, SBS-controlled email alias — same survivability rationale as the DigitalOcean account, §2.1). The Build Register carries the domain transfer item ("Transfer SBSDASH.COM") until registrar ownership sits with SBS; the current registration under the interim account is a tracked transitional state.
• Registrar lock — clientTransferProhibited enabled; transfer-out requires an authenticated unlock action. Domain-transfer attempts are a high-severity alert.
• Registrar MFA — phishing-resistant MFA on the registrar account. IAM-03 applies to every control plane, registrar included: registrar compromise is functionally equivalent to full platform compromise, since it permits nameserver redelegation.
• Renewal — auto-renew enabled, multi-year registration, expiry monitored (an expired domain is simultaneously an availability failure and a takeover opportunity).

4.2 Authoritative DNS

As built (2026-07-18): the zone is hosted on GoDaddy nameservers. GoDaddy does not provide DNSSEC signing for externally manageable zones in this configuration, records are console-managed rather than IaC-managed, and no API-driven change control is in place. This state satisfies neither DNS-01 (DNSSEC) nor the IaC management rule (Chapter 10) and is a tracked nonconformance (Build Register item 9).

Target: the authoritative zone is hosted at a DNSSEC-capable, API-managed provider. Approved options, in preference order:

1. Cloudflare DNS (free tier sufficient) — one-click DNSSEC signing, DS handoff to the registrar, full API for IaC management, native CAA/CT tooling.
2. deSEC — DNSSEC-by-default, API-managed, nonprofit.
3. Registrar-hosted DNS only if the registrar signs zones and exposes a management API.

Zone cutover sequence (Nelson Santos, execution owner): replicate the current record set at the new provider → verify resolution parity against §4.3 → lower TTLs to 300 → update NS delegation at the registrar → confirm propagation → enable DNSSEC and publish DS (§4.5) → raise TTLs. The zone is thereafter managed as code (Chapter 10): records are defined in the IaC repository and applied via provider API; console edits are drift and reverted.

4.3 Record Inventory

As built (2026-07-18):

Target:

Notes:

• Tenant surfaces share one A target; host-header routing at the reverse proxy (Chapter 5) separates them. No wildcard record exists — *.sbsdash.com is unresolvable by design (FE family: no surface enumeration; a wildcard would also pollute the CT-monitoring signal, §4.6).
• The apex repoint (138.197.222.154 → 107.170.72.145) is executed only after the prod proxy and TLS stack pass their Chapter 5 gates — repointing to an unready origin is an availability failure. Until then the current apex target remains inside the engagement security boundary (§2.2.2).
• Dev has no records in the public zone. The as-built apex record violates this rule today; it is cured by the repoint. Dev access is by IP against the admin allowlist.
• IPv6: AAAA records are added only when droplet IPv6 is enabled and firewall rulesets carry mirrored v6 rules — publishing AAAA without v6 firewall parity is a firewall bypass, so the two land together or not at all. (As built there are no v6 firewall rules anywhere; no AAAA records are published.)

4.4 Mail Posture (DNS-03)

sbsdash.com sends and receives no mail. Null-mail posture, published explicitly so the domain cannot be used as a spoofed sender:

These records are publishable on GoDaddy today — they do not depend on the zone cutover and should not wait for it. DMARC aggregate reports route to an SBS security mailbox on an existing SBS mail domain (not sbsdash.com), making spoof attempts observable — a MON-02 anomaly input, not merely a static record.

One caution: the Grafana admin account and the changelog reference admin@sbsdash.com and a Yahoo mailbox as service identities. No mailbox exists or will exist on sbsdash.com under the null-mail posture; service accounts must use SBS-controlled aliases on SBS's mail domain. Folded into the credential remediation (Build Register item 15).

4.5 DNSSEC (DNS-01)

As built: unsigned — no signing path exists on the current GoDaddy hosting (§4.2). DNS-01 is unmet until zone cutover completes.

Target, unchanged in substance:

• Zone signed at the DNS provider (ECDSA P-256, algorithm 13 — compact signatures, universal resolver support).
• DS record published at the registrar — signing without registrar DS publication provides no validation path. End-to-end verification: dig +dnssec sbsdash.com returns the AD flag from a validating resolver, and DNSViz shows an unbroken chain. This verification is a named Build Register gate (item 10).
• Key rollovers are provider-automated (CDS/CDNSKEY where the registrar supports it); manual DS changes are change-controlled.
• Failure-mode note: DNSSEC misconfiguration causes hard resolution failure, not degraded security — DS/keyset changes execute inside the 300-TTL window and are verified before TTLs are raised.

4.6 Certificate and Registration Surveillance

• CAA (DNS-02): sbsdash.com CAA 0 issue "letsencrypt.org" plus CAA 0 iodef "mailto:<SBS security alias>". Only Let's Encrypt may issue for the domain — consistent with the deployed ACME/certbot practice already live on monitor.sbsdash.com (Let's Encrypt via certbot, auto-renewal by systemd timer, per changelog 2026-07-17). Any other CA refusing on CAA is the control working. issuewild is intentionally absent (inherits issue; acceptable — no wildcard certificates are used). Publishable on GoDaddy today; does not wait for cutover.
• CT monitoring (DNS-04): subscription against Certificate Transparency logs for sbsdash.com (Cloudflare CT alerts or crt.sh RSS into the alert pipeline). Baseline note: certificates for monitor.sbsdash.com already appear in CT from the 2026-07-17 issuance — expected and legitimate. Any CT appearance not originating from the platform's ACME automation is a high-severity incident (CAA bypass or issuance compromise).
• Typosquat monitoring (DNS-05): dnstwist scheduled weekly from monitor-servers against a generated permutation set (homoglyphs, transpositions, TLD swaps); newly registered lookalikes feed MON-02 alerting. No defensive registrations by default; escalation case-by-case via Alexandra Del Rey (security-compliance routing).

4.7 Change Control and Verification

Until zone cutover, GoDaddy console changes are restricted to: the three mail records (§4.4), the CAA record (§4.6), and the apex repoint (§4.3) — each logged in SERVER_CHANGELOG.md with date and operator. After cutover, zone changes follow the IaC path (Chapter 10) with the same drift rules as firewalls.

Verification gates: NS delegation at registrar points to the DNSSEC-capable provider; DNSSEC chain validates (AD flag, DNSViz clean); null-MX/SPF/DMARC present; CAA present; apex and all subdomain records resolve per the §4.3 target table (apex → 107.170.72.145, not 138.197.222.154); no wildcard resolves; no Dev host referenced anywhere in the zone. Controls covered: DNS-01 through DNS-05.

---

Build Register deltas from this chapter: item 9 restated as GoDaddy→provider cutover with sequence; item 16 apex repoint gated on Chapter 5 readiness; new item 23 — publish null-mail + CAA records on GoDaddy immediately (pre-cutover); item 15 extended to cover the admin@sbsdash.com / Yahoo service-identity cleanup. Chapter 5 (TLS & Edge Security) next — the monitor's live nginx/certbot stack gives it a verified as-built baseline to write against.

TLS & Edge Security

Chapter 5 — TLS & Edge Security

5.1 Edge Model

Every public surface is fronted by an nginx reverse proxy terminating TLS on the droplet that serves it. No CDN or external proxy layer is in scope at this fleet size; origin concealment (TLS-04) is addressed in §5.6 within that constraint.

Host-header routing on sbsdash-server-prod separates the three tenant surfaces behind one IP. A default server block returns 444 (connection close, no response) for any request whose Host/SNI matches no configured surface — direct-to-IP scans and unrecognized hostnames get nothing, including no certificate hint (see §5.2 default cert note).

5.2 Certificate Management (TLS-02)

• CA and protocol: Let's Encrypt via ACME. Issuance and renewal by certbot with the nginx plugin, auto-renewal by certbot.timer (systemd) — the pattern already live and verified on the monitor is the fleet standard.
• One certificate per hostname. No wildcard certificates (consistent with §4.6 CAA — issuewild not authorized) and no multi-SAN certificates spanning surfaces: a SAN list enumerating admin. and clients. on the apex cert would leak the surface inventory to any TLS client, defeating the FE-family enumeration controls. Per-hostname certs keep CT disclosure to exactly the hostname being served.
• Challenge type: HTTP-01 on port 80 per surface (matches the §3.5 firewall admits). DNS-01 challenges become available post zone cutover (§4.2) but are not required — HTTP-01 avoids granting the ACME client DNS-write credentials, a smaller blast radius.
• Renewal monitoring: certificate expiry is a Prometheus-alerted metric (blackbox exporter probe per surface, Chapter 12). A cert inside 14 days of expiry with renewals failing is a paged alert — certbot.timer failure is otherwise silent.
• Default/fallback certificate: the 444 default block carries a self-signed placeholder cert (nginx requires one to complete the handshake before it can close); it discloses no real hostname.
• Key hygiene: private keys remain on the serving droplet at /etc/letsencrypt/live/<host>/, root-owned 0600, never copied off-host, never committed. Keys are not vaulted — reissuance via ACME is cheaper and safer than key escrow.

5.3 Protocol Floor (TLS-01)

Fleet-standard nginx TLS parameters, IaC-templated (Chapter 10), identical on every surface:

• ssl_protocols TLSv1.2 TLSv1.3; — nothing below 1.2, ever. 1.3 preferred by client negotiation.
• TLS 1.2 cipher set restricted to AEAD/ECDHE (ECDHE-ECDSA-AES128-GCM-SHA256, ECDHE-RSA-AES128-GCM-SHA256, ECDHE-*-AES256-GCM-*, CHACHA20-POLY1305); no CBC, no RSA key exchange, no 3DES/RC4.
• ssl_prefer_server_ciphers off; (correct posture for a modern AEAD-only set — client picks its fastest).
• ECDSA P-256 leaf keys preferred (smaller handshakes); RSA-2048 acceptable where tooling requires.
• OCSP stapling on (ssl_stapling on; ssl_stapling_verify on;).
• Session tickets off (ssl_session_tickets off;) — avoids ticket-key rotation burden; session cache local, 10 m.
• Verification: testssl.sh run per surface at each gate; target grade A on SSL Labs semantics with no protocol/cipher findings. The monitor's live config is brought to this template at the next maintenance window (certbot's default bundle admits TLS 1.0/1.1 on some distro snippets — to be confirmed and corrected; Build Register).

5.4 HSTS (TLS-03)

• Port 80 exists solely for ACME HTTP-01 (/.well-known/acme-challenge/) and a 301 to HTTPS; no content is ever served over HTTP.
• Rollout in two steps: (1) at surface go-live, Strict-Transport-Security: max-age=86400 — one day, recoverable if anything mis-serves; (2) after seven clean days, raise to max-age=31536000; includeSubDomains; preload fleet-wide and submit sbsdash.com to the Chrome preload list.
• includeSubDomains is safe only because the null-mail/no-other-services posture (§4.4) guarantees no plaintext-HTTP subdomain will ever exist. Preload submission is one-way in practice (removal takes months); it is executed as a named Build Register gate after Alexandra Del Rey's sign-off, since it permanently commits every future *.sbsdash.com service to TLS.

5.5 Security Headers (APP-02) and Frontend Exposure Controls (FE-01…FE-06)

Fleet-standard header set, applied at nginx (add_header ... always;), IaC-templated; per-surface CSP tightening happens at the application layer:

• robots.txt (FE-02): every surface serves User-agent: * / Disallow: /. Defense-in-depth with FE-01 — robots.txt alone is advisory; the header is the enforcement.
• No social cards (FE-03): no OpenGraph/Twitter meta tags in any served HTML; link unfurlers (Slack, iMessage, crawlers) get no title, description, or preview image. Grafana and BookStack template overrides to strip their defaults are Build Register items.
• Neutral metadata (FE-06): server_tokens off; (no nginx version); HTML <title> values are neutral ("Sign in" — not "SBS …"); no meta description; error pages are generic nginx defaults with no branding. The platform's public fingerprint is deliberately information-free: resolvable domain, TLS cert, sign-in page, nothing else.
• Grafana behind the proxy requires proxy_set_header Host + websocket upgrade headers on /api/live/ (already configured); BookStack requires correct X-Forwarded-Proto to generate https URLs.

5.6 Origin Concealment (TLS-04)

Scoped honestly for a no-CDN architecture: the A records necessarily disclose droplet IPs, so concealment here means the origin exposes nothing but the intended service:

• Direct-to-IP requests hit the 444 default block — no content, no redirect, no certificate naming a real host.
• No service banner, version string, or debug endpoint on any public port (§5.5 neutral metadata; ss -tlnp gate confirms only 80/443/22 public).
• Non-public services (Grafana upstream 3000, Prometheus 9090, exporters 9100, future data services) bind 127.0.0.1 or VPC-private only — verified as-built on the monitor.
• If DDoS or scraping pressure later justifies it, fronting with Cloudflare proxy (post zone cutover) upgrades TLS-04 to true IP concealment; that decision is deferred and would arrive as a Change Order touching §4.2, §5.2 (cert model), and §3.5 (firewall admits narrowed to Cloudflare ranges).

5.7 Admin Surface Isolation (APP-03, edge layer)

admin.sbsdash.com receives edge-layer restrictions beyond the tenant surfaces, ahead of the application-layer controls in Chapter 9:

• Separate nginx server block, separate upstream container, separate access log.
• IP allowlist at the edge (allow/deny from the IaC admin-allowlist variable) — the admin console is not globally reachable even pre-authentication, unlike sbsdash.com/clients..
• Stricter rate limit (limit_req zone: 5 r/s vs 20 r/s tenant baseline) and client_max_body_size scoped to actual admin upload needs.
• All four proxies (three tenant + admin) log in a common structured format shipped to the observability pipeline (Chapter 12) — access logs are a MON-01/MON-02 input.

5.8 Verification

Per-surface gate set (LH-SBS-INST-001): testssl.sh clean, protocol floor confirmed (no 1.0/1.1 handshake accepted); cert issued/renewing with expiry probe green; HTTP→HTTPS 301 with ACME path excluded; full header set present (curl -sI diff against template); robots.txt served; direct-IP request returns 444; no OG/Twitter tags in HTML; server_tokens absent; admin surface unreachable from a non-allowlisted IP; upstream ports absent from public ss -tlnp. The monitor surface — already live — is retro-fitted to the full template and re-verified as the reference implementation before the prod surfaces build against it.

Compute & Application Layer

Chapter 6 — Compute & Application Layer

6.1 Compute Model

The Platform runs on a small fixed droplet fleet with containerized workloads under Docker Compose — no Kubernetes, no managed PaaS. Rationale: the fleet is ≤5 hosts with a single-application production profile; Compose delivers the required per-Compartment network isolation (§6.4) with a fraction of the operational surface of an orchestrator, and its state is fully expressible in versioned files (IaC, Chapter 10). Revisit only if the Compartment count or droplet fleet grows materially (Chapter 16 thresholds).

As-built baseline: Docker 29.1.3 + Compose v2.40.3 installed from Ubuntu official repos (no third-party apt sources — consistent with the supply-chain posture, Chapter 11) on sbsdash-server-prod, Dev-SBS-server, and monitor-servers (2026-07-18); docker service enabled at boot. The wiki droplet intentionally has no Docker — BookStack runs as a system service and stays that way until the NYC2 rebuild, where containerization is decided (§6.6).

6.2 Droplet Roles and Sizing

Rule: the monitor and wiki are single-purpose hosts. Application or data workloads never deploy to them — the monitor because it is the bastion and alerting authority (compromise radius, §3.3), the wiki because it is an operational surface with no tenant data (§1.2).

6.3 Runtime Toolchain

Target toolchain on prod and dev (Build Register items; not yet installed as of 2026-07-18):

Two rules carried from §2.4: versions are identical on prod and dev (drift = build defect), and runtimes exist on the host primarily for build/operational tooling — application code executes inside containers, whose images pin their own runtime versions (Ch. 11). Host-level Node/Rust/Python are not a second execution path for platform services.

6.4 Per-Compartment Isolation at the Compute Layer

This section realizes the application/network/data slices of the eight-layer isolation model (§1.2) in Compose terms. Each of the ten Compartments deploys as an isolated stack:

• One Compose project per Compartment (compartment-01-hr … compartment-10-pr), each with its own compose file rendered from a common IaC template — no shared "mega-compose."
• One dedicated bridge network per Compartment (net-c01 … net-c10), internal: true where the Compartment needs no direct egress (egress instead via its declared integration endpoints per §3.4 chains). No container joins two Compartment networks. Cross-Compartment traffic is denied by non-membership — there is no route to filter because there is no shared segment.
• Edge attachment: each Compartment's frontend container additionally joins a shared net-edge network reachable only by nginx. nginx is the sole multi-network process; Compartment-to-Compartment via the edge network is impossible because only frontends and nginx sit on it and frontends accept only proxied requests (verified by header check at the app layer, APP-05).
• Per-Compartment resource limits: mem_limit and cpus set per service so one Compartment's runaway agent load cannot starve the other nine (availability isolation, not just confidentiality).
• No privileged containers, no docker.sock mounts, no-new-privileges: true, read-only root filesystems where the app allows — container escape hardening baseline.
• Secrets: injected at runtime from Vault (Ch. 8) via env-file rendered at deploy, never baked into images, never in compose files committed to the repo (INF-02). Per-Compartment Vault namespaces mean Compartment 03's containers can never read Compartment 08's credentials (AI-03 credential isolation at the infra layer).
• AnythingMCP deploys as a per-Compartment sidecar container on that Compartment's network — MCP tool servers are Compartment-scoped, which is the infrastructure half of AI-02 (tool allowlisting): a Compartment can only reach the MCP tools deployed on its own network, and each MCP container's outbound is bound by that Compartment's §3.4 egress chain.

6.5 Data Services Placement (Interim)

Pending the NYC2 Managed Database availability check (§2.3), PostgreSQL, MongoDB, and Redis run as containers on sbsdash-server-prod:

• Each data service on its own compose network (net-data-pg, etc.); Compartment application containers join only the data networks they are entitled to, and per-Compartment isolation inside each database engine (schemas/databases/ACLs per Compartment) is specified in Chapter 7.
• Data services bind container ports to the VPC-private interface or localhost only — never 0.0.0.0 (public-listener gate, §3.7).
• Volumes on the droplet's block storage with the backup regime of Chapter 14.
• If Managed Databases are confirmed in NYC2, migration moves data off-droplet and this section is amended by Change Order; the entitlement model (which Compartment reaches which database) is unchanged either way.

6.6 Wiki and Monitor Stacks

• monitor-servers: Prometheus/Grafana remain apt-installed system services (as built, verified) — containerizing the monitoring stack adds a Docker dependency to the thing that watches Docker; current form is deliberate. Docker exists on this host solely for operational tooling (e.g., blackbox exporter if containerized later).
• sbs-wiki: at NYC2 rebuild, BookStack is redeployed either as system service (snapshot restore, fastest) or containerized (compose template, consistent with fleet). Decision at rebuild; either way it stays a single-purpose host on the Prod VPC with private-bound exporter.

6.7 Deployment Flow

Images are built in CI from the GitHub repository (Ch. 10/11), pushed to the registry enumerated in the supply-chain allowlist, and pulled by prod over the §3.4 egress path. Deploys are docker compose up -d against IaC-rendered files, executed by the deploy user (per-person or CI identity — INF-04 remediation applies, §2.5); manual docker run on prod is prohibited and surfaces as drift (running-container set vs. compose state is a monitored comparison, MON-01). Rollback = redeploy previous pinned image tag; images are immutable and tagged by digest.

6.8 Verification

Gates (LH-SBS-INST-001): Compose project per Compartment present with dedicated network; cross-Compartment connectivity probe fails between any two Compartment networks; frontend reachable only via nginx (direct container port probe fails); no privileged container, no docker.sock mount, no-new-privileges set (docker inspect sweep); data-service ports absent from public interface; resource limits present on every Compartment service; running containers match compose state (drift check); toolchain versions identical prod↔dev (node -v, rustc -V, python3 -V diff).

Storage & Data Layer

Chapter 7 — Storage & Data Layer

7.1 Data Layer Overview

Three engines serve the Platform, each with a distinct role; nothing else is introduced without a Change Order:

As of 2026-07-18 none of the three is installed — this chapter is fully normative, with the §6.5 interim placement (containers on sbsdash-server-prod) as the build path and managed services as the preferred end state.

7.2 Managed vs. Self-Hosted Decision

• Preferred: DigitalOcean Managed Databases for PostgreSQL, MongoDB, and Redis (Valkey) — provider-handled failover, automated backups, patch management, and TLS-enforced connections reduce the operational load on a small team.
• Blocking check (Build Register, carried from §2.3): Managed Database availability in NYC2 is unconfirmed — newer DO capacity often lands in NYC1/NYC3 first. The check is: can all three engines be provisioned in NYC2 attached to the target Prod VPC (prod-vpc-nyc2, §3.1)? Cross-region managed databases are disqualified — data would transit public networking, violating INF-01, the same class of defect as the wiki scrape (§3.3).
• Fallback (interim, buildable now): self-hosted containers per §6.5. The remainder of this chapter is written engine-by-engine to be valid under either deployment, with deltas flagged.
• Migration posture: if the build starts self-hosted and Managed becomes available, migration is dump/restore per engine during a maintenance window, amended by Change Order. The isolation model (§7.4) is deployment-invariant, so no application change is required — only connection endpoints (Vault-managed, Ch. 8).

7.3 Object Storage

DigitalOcean Spaces (S3-compatible), region NYC3 (Spaces is not offered in NYC2; object storage over TLS to a neighboring region is acceptable — unlike database traffic, it is not latency- or VPC-bound):

Rules: all buckets private (no public-read ever — FE family applies to storage too); access exclusively via scoped Spaces keys stored in Vault, one keypair per purpose, never a shared account-wide key (INF-02); TLS enforced on every operation; bucket inventory and lifecycle policies defined in IaC (Ch. 10). CDN feature on Spaces stays off — these are private buckets, and enabling the CDN endpoint would create an unauthenticated public URL surface.

7.4 Data Isolation Boundaries (per-Compartment)

The data and storage slices of the eight-layer model (§1.2), stated per engine. The principle throughout: isolation is enforced by the engine's own privilege system, not by application discipline — a compromised Compartment 03 credential must be structurally unable to read Compartment 08 data.

• PostgreSQL — one database per Compartment (c01_hr … c10_pr) in the cluster; one role per Compartment with CONNECT on its own database only; PUBLIC connect revoked on every database; no cross-database query paths (no postgres_fdw, no dblink extensions installed); platform-shared configuration lives in a separate platform database with its own role. Superuser credentials exist only in Vault, used only by migration tooling.
• MongoDB — one database per Compartment; per-Compartment user with readWrite scoped to its own database; authentication mandatory (--auth), authorization enabled from first boot — never an open dev instance that later gets locked down.
• Redis — Redis ACLs (v6+) with one ACL user per Compartment, key-pattern-restricted (~c01:*), dangerous commands (FLUSHALL, CONFIG, KEYS, SCRIPT) denied to all Compartment users; default user disabled. If workload interference ever matters more than memory efficiency, escalation path is numbered logical databases or separate instances per Compose network — Change Order.
• Connection path (self-hosted interim): Compartment containers join only the data networks they are entitled to (§6.5), so isolation is double-walled — network non-membership and engine ACL. Under Managed Databases the network wall is the VPC + per-database credentials; the engine ACL layer is identical.
• Object storage: per-Compartment prefix (c01/…) with per-Compartment Spaces keys whose policy is prefix-scoped. AI-03 (credential isolation) extends here: an agent's runtime credentials reach exactly its Compartment's database, its Redis keyspace, and its Spaces prefix — nothing else.

7.5 Encryption

• At rest, block storage: DigitalOcean encrypts droplet disks and Volumes at rest (AES-256) at the platform layer. This is accepted as the baseline for the interim self-hosted deployment; no LUKS layer is added on top — key management for host-level LUKS on cloud VMs adds operational fragility (unattended reboot problem) for marginal gain against the relevant threat model (DO platform compromise is out of scope; stolen-disk scenarios are covered by DO's layer).
• At rest, engine level: PostgreSQL — no TDE in community builds; compensating controls are the DO layer plus strict dump handling (§7.6). MongoDB Community — same posture. Redis — ephemeral data, AOF file inherits disk encryption.
• At rest, object storage: Spaces server-side encryption; audit archives additionally carry their own integrity chain (hash-chained, Ch. 13) so tampering is detectable independent of encryption.
• In transit: TLS on every database connection including VPC-internal (zero-trust, NIST SP 800-207 — the private network is not a trust boundary): PostgreSQL ssl=on with sslmode=verify-full in clients; MongoDB tls=true; Redis TLS listener. Internal certs from a platform-internal CA generated and rotated by IaC (not Let's Encrypt — these hosts have no public names). Managed Databases enforce TLS natively.
• Backups/dumps: encrypted client-side (age or GPG, key in Vault) before upload to sbs-dash-backups — a leaked bucket key must not equal readable data. Detail in Ch. 14.

7.6 Operational Rules

• No production data leaves the Prod boundary (§2.2.3): Dev databases are seeded synthetically or from sanitized dumps produced by a reviewed script (PII/tenant content stripped), never raw prod dumps.
• Ad-hoc human access to prod data engines is break-glass only, via bastion (§3.3), with individually attributed credentials (post INF-04 remediation, §2.5), logged as a MON-01 event. Routine inspection happens through dashboards and read-only replicas if/when provisioned.
• Retention: Compartment operational data per SBS function-level policy captured at Discovery (LH-SBS-DISC-001/002 outputs); audit log 7-year minimum (Ch. 13); Redis nothing beyond declared AOF keyspaces.
• Volume/disk capacity is a Prometheus-alerted metric (Ch. 12); databases run on dedicated DO Volumes (not the droplet root disk) so storage scales and migrates independently of the droplet — Volume creation is part of the data-services Build Register item.

7.7 Verification

Gates (LH-SBS-INST-001): cross-Compartment access probe per engine fails (c03 role → c08 database denied in PostgreSQL, MongoDB, Redis ACL, and Spaces prefix); no public listener on any engine port (ss -tlnp + external scan); TLS handshake required on every engine (plaintext connection refused); PUBLIC/default access revoked (engine ACL dump review); dumps encrypted before leaving host (backup pipeline inspection); Dev contains no production data (sampled content check); Spaces buckets private (unauthenticated GET fails); audit bucket writer key cannot DELETE (negative test).

Secrets Management

Chapter 8 — Secrets Management

8.1 Current State and Scope

As of 2026-07-18 no secrets manager is deployed. Credential state is the Chapter 2 CRITICAL nonconformance: plaintext passwords in the operational changelog, a shared fleet-wide root password, and service identities on non-SBS mailboxes (Build Register item 15). This chapter defines the target system that item 15 migrates into, covering: platform secrets (API tokens, DB credentials, Spaces keys, internal-CA material, webhook URLs), per-Compartment agent credentials (AI-03), and deploy-time injection into containers (§6.4). Out of scope: end-user passwords (IdP domain, Chapter 9) and TLS private keys for public surfaces (deliberately host-local, §5.2).

8.2 Vault Deployment

HashiCorp Vault (OSS) on a dedicated minimal droplet:

Placement rationale: not on sbsdash-server-prod (the secrets store must survive and be trustworthy independent of the host running the workloads that consume it — a prod-droplet compromise should not equal immediate Vault-storage compromise), and not on monitor-servers (bastion + alerting authority already concentrates enough; adding secrets makes it a single point of total compromise). A 1 GB droplet is sufficient — Vault's footprint at this scale is trivial.

Unseal model: DigitalOcean offers no KMS, so auto-unseal via cloud KMS is unavailable. Shamir key shares: 5 shares, threshold 3, held by named individuals (Ledger Hub platform engineering ×2, Nelson Santos, Alexandra Del Rey, one SBS-designated alternate) — no individual can unseal alone; any 3 can recover. Shares are stored in each holder's personal password manager, never together, never in the changelog or repo. Unseal is required at Vault restart only; restarts are rare, scheduled events. Root token is revoked after initial setup (vault operator generate-root recreates one under quorum if ever needed) — routine administration uses named admin tokens under policy.

Availability posture, stated honestly: single-node Vault means Vault downtime blocks deploys and credential rotation, not running workloads (containers hold their injected credentials until restart, §8.5). Accepted at this fleet size; mitigation is Raft snapshot every 6 h to sbs-dash-backups (encrypted client-side per §7.5), restore-tested quarterly (Ch. 14). HA (3-node Raft) is a Chapter 16 growth trigger, not a Day-1 requirement.

8.3 Namespace and Policy Layout

Vault OSS lacks Enterprise namespaces; the equivalent is mount-path + policy separation, which is sufficient because policies are default-deny:

secret/platform/        # DO API tokens, GitHub deploy keys, Spaces keys,
                        # registrar/DNS API tokens, alerting webhooks
secret/infra/ca/        # internal CA key material (§7.5)
secret/db/platform/     # migration-tool superuser credentials (§7.4)
secret/c01/ … secret/c10/   # per-Compartment: DB role, Redis ACL user,
                            # Spaces prefix keys, Anthropic API key,
                            # per-Compartment integration credentials
secret/dev/…            # dev-environment mirror, synthetic/dev-scoped values only

Policies, one per consumer class: policy-c01 grants read on secret/c01/* and nothing else — the AI-03 guarantee at the secrets layer, matching the network (§6.4) and engine-ACL (§7.4) walls: Compartment 03's runtime identity cannot read secret/c08/* even if every other control fails. policy-deploy reads what the deploy pipeline renders; policy-admin is held by named humans; no policy grants secret/* wildcard. Dev identities can read secret/dev/* only — prod paths are unreachable from Dev (the promotion path carries templates, not values, §2.2.3).

8.4 Secret Lifecycle and Rotation Policy

Rules: every secret has a named owner and a recorded rotation date (KV metadata); KV v2 versioning on all mounts so rotation keeps the prior version recoverable for rollback; no secret is ever emailed, chat-pasted, or written to the changelog — the changelog records that a rotation happened, never the value. Audit device (file → shipped to Ch. 12 pipeline) logs every read/write with the requesting identity: secret access is itself a MON-01 event stream, and reads outside deploy windows or from unexpected identities are MON-02 anomaly inputs.

8.5 Application Integration

Injection model (carried from §6.4), chosen for minimal moving parts:

1. AppRole per consumer — one AppRole per Compartment (role-c01 … bound to policy-c01) and one for the deploy pipeline. RoleID is IaC-managed config; SecretID is delivered response-wrapped at deploy time with short TTL and use-limit 1 — a leaked wrapped token is worthless after first use or expiry.
2. Deploy-time render — the deploy step authenticates via AppRole, reads the Compartment's paths, renders the env-file consumed by docker compose (§6.7), sets mode 0600, and deletes the env-file after container start (compose has already captured the environment). Values never enter images, compose files, the repo, or CI logs (INF-02, INF-03).
3. Refresh — containers hold credentials until redeploy. With dynamic DB credentials (§8.4), lease TTLs are set ≥ the deploy cadence so normal deploys renew credentials as a side effect; an emergency revocation is: revoke lease in Vault → redeploy Compartment (minutes, and only the affected Compartment restarts — per-Compartment stacks pay off here).
4. Prohibited patterns — no Vault agent sidecars for now (added complexity before it's needed), no secrets in docker inspect-visible plain environment: keys in committed files, no shared "platform god token" consumed by all Compartments.

Bootstrap note (the classic first-secret problem): the deploy host must hold one credential to start — the RoleID (low sensitivity, useless without SecretID) on disk, and SecretID issuance gated by the CI/operator identity. This is the accepted trust anchor and is documented in the wiki runbook rather than hidden.

8.6 Migration of Existing Credentials (Item 15 Execution Order)

1. Deploy prod-vault, initialize, distribute shares, revoke root token.
2. Rotate the four exposed credential sets (GitHub, droplet root password → abolished entirely in favor of keys per §2.5, wiki admin, Grafana admin) — rotate then store; never vault a burned value.
3. Write rotated values + all new secrets created since (DO tokens, Spaces keys as provisioned) into their §8.3 paths.
4. Strip the changelog credentials section; purge from any git history (git filter-repo) if the file was ever committed; add pre-commit secret-scanning (gitleaks) to the repo (Ch. 10/11) so re-introduction is blocked mechanically, not procedurally.
5. Re-point service identities to SBS-controlled aliases (§4.4 note) as each is touched.

8.7 Verification

Gates (LH-SBS-INST-001): Vault reachable from prod over VPC TLS, unreachable from any public interface (external scan); policy probe — role-c03 token reads secret/c03/*, denied on secret/c08/* and secret/platform/*; response-wrapped SecretID single-use confirmed (second unwrap fails); env-file absent post-deploy (filesystem check); no secret string present in repo, images, compose files, or CI logs (gitleaks sweep); audit device shipping to observability pipeline; Raft snapshot present in sbs-dash-backups, encrypted, restore-tested; changelog contains no credential values (item 15 closure evidence); rekey ceremony documented with current holder list.

Identity & Access

Chapter 9 — Identity & Access

9.1 Current State and Scope

As of 2026-07-18 no IdP integration exists: Grafana and BookStack run on local admin accounts (both credentials exposed in the changelog — item 15 applies), the platform application is pre-deployment, and host access is the shared-key nonconformance of §2.5. This chapter defines the target identity architecture covering the IAM control family: IAM-01 (identity-aware proxy), IAM-02 (SSO federation), IAM-03 (phishing-resistant MFA), IAM-04 (tenant separation), IAM-05 (SCIM lifecycle), IAM-06 (access reviews). Human identity only — machine/agent credentials are Chapter 8; per-Compartment application RBAC internals (APP-05) are application-layer and specified in the platform application documentation, but their identity source is defined here.

9.2 Identity Provider

Single source of truth: SBS's corporate IdP. The Platform maintains no local user store — no local passwords on any surface (the two as-built local admins are migrated then reduced to break-glass, §9.6). Users exist on the Platform only as federated identities projected from the SBS directory.

Discovery dependency (blocking): SBS's incumbent IdP product (Entra ID, Okta, Google Workspace, or other) is unconfirmed — it is a named LH-SBS-DISC-002 IT-function question and a gate for this chapter's build items. The architecture below is written IdP-agnostic against capabilities every mainstream enterprise IdP provides: OIDC, SAML 2.0, SCIM 2.0, group claims, WebAuthn/FIDO2 MFA enforcement. Nelson Santos is execution owner for the IdP-side configuration (app registrations, group creation, SCIM token issuance).

Federation protocol: OIDC (Authorization Code + PKCE) everywhere it is supported — Grafana (native OIDC), the platform application (OIDC middleware), the identity-aware proxy (§9.3). SAML only where a component supports nothing better. Token hygiene: short-lived access tokens (≤1 h), refresh handled server-side, email_verified + group claims required in the ID token.

MFA (IAM-03): enforced at the IdP, not per-application — phishing-resistant methods only (WebAuthn/FIDO2 hardware or platform authenticators) for all Platform-mapped groups; TOTP acceptable only as a documented exception with expiry; SMS/voice prohibited. Conditional-access (where the IdP supports it): Platform app registrations require MFA on every authentication, no "remembered device" exceeding 12 h for admin roles.

9.3 Identity-Aware Proxy (IAM-01)

Enforcement point for surfaces that lack robust native OIDC or need pre-application gating. Implementation: oauth2-proxy deployed alongside nginx on the serving droplet, wired via auth_request:

oauth2-proxy session cookies: __Host- prefix, Secure, HttpOnly, SameSite=Lax, secret from Vault (secret/platform/), session lifetime ≤ 8 h with IdP re-auth. The proxy passes identity to upstreams via signed headers; upstreams reject unsigned/absent identity headers so the proxy cannot be bypassed by direct container access (pairs with the net-edge design, §6.4).

9.4 SCIM Provisioning (IAM-05)

Lifecycle automation — the control that makes offboarding real:

• Direction: IdP → Platform. SBS directory group membership drives Platform access; the Platform never invites users directly.
• Targets: the platform application (SCIM 2.0 endpoint, Compartment-role mapping per §9.5) and Grafana (SCIM or team-sync per licensing; fallback is OIDC group-claim mapping at login — acceptable because login-time mapping plus short sessions bounds staleness to the session length).
• Deprovisioning SLA: IdP deactivation propagates to Platform deactivation within 15 minutes (SCIM push) and in the worst case at next token expiry (≤ 1 h). A user disabled in the SBS directory holds no live Platform session beyond that hour — this is the tested metric, not the config's existence.
• SCIM tokens live in Vault (secret/platform/scim/), rotate on the 90-day static schedule (§8.4), and their use is logged (MON-01).
• Joiner/mover/leaver flows are exercised end-to-end as a named gate: create test user → group add → verify access; group move → verify role change; deactivate → verify lockout inside SLA.

9.5 Least-Privilege Model and Group Mapping (IAM-04 / APP-05 boundary)

Access is group-derived, never user-granted. SBS directory groups map to Platform roles; the group list is the single audit surface for IAM-06:

Rules: no wildcard "all-Compartments" user group exists — cross-Compartment humans (e.g., the COO) hold the exec-dashboard role, which reads reporting outputs, not Compartment interiors; platform-admin grants administration, and its members' actions are fully logged (MON-01) — admin ≠ unobserved. Group membership changes are IdP-audited events; the quarterly IAM-06 access review walks every dash-* group against current staffing (Alexandra Del Rey coordinates sign-off; Nelson Santos executes directory changes), and the review artifact is filed in the wiki with date and reviewer — access certification (MON-04) consumes the same artifact.

9.6 Admin Access Paths

Consolidated statement of every privileged path and its chain, replacing ad-hoc practice:

The break-glass row is the honest residual of the "no local accounts" rule: total IdP outage must not equal total platform lockout, so exactly one dormant local admin per critical surface survives — vaulted, alarmed on use, rotated after.

9.7 Verification

Gates (LH-SBS-INST-001): unauthenticated request to every gated surface yields IdP redirect, zero application bytes (curl sweep); direct-to-container bypass attempt rejected (unsigned identity header); MFA challenge presented on fresh session for admin group (manual gate); SCIM lifecycle test — provision/move/deactivate with deactivation lockout ≤ 15 min (timed); group-mapping probe — dash-c03-user reaches c03 surfaces, denied on c08 (IAM-04 human-layer test mirroring §7.7); local password login disabled on Grafana/BookStack (negative login test); break-glass procedure executed in drill, rotation confirmed; IAM-06 review artifact exists with current date.

Infrastructure as Code

Chapter 10 — Infrastructure as Code

10.1 Current State and Scope

As of 2026-07-18 no IaC exists: all resources to date (droplets, DNS records, ufw rules, Grafana/Prometheus configs) were created by console or shell, documented after the fact in SERVER_CHANGELOG.md. That changelog discipline is the right instinct executed at the wrong layer — this chapter moves the source of truth from prose describing what was done to code that does it, with the changelog demoted to narrative context. Scope: DigitalOcean resources, DNS zone (post-cutover, §4.2), host configuration, firewall rulesets, Vault policy/mount layout, compose templates, and monitoring config. Out of scope: application source code (its CI/CD is platform-repo concern; supply-chain controls shared with Chapter 11).

10.2 Tooling

Two tools, sharply divided by responsibility — resisting the single-tool temptation because each is poor at the other's job:

Supporting: gitleaks (pre-commit + CI, §8.6), promtool/nginx -t/vault policy fmt as validation steps in CI, Infracost not adopted (fleet too small to justify).

State backend: OpenTofu state in the sbs-dash-backups Spaces bucket (S3-compatible backend) with state locking via DO Spaces conditional writes; state contains secrets-adjacent values (IPs, IDs — not credentials, which stay in Vault per §8.5) and the bucket is private + encrypted (§7.3). Ansible is stateless by design; its "state" is the repo.

10.3 Repository Structure

One infrastructure repository (sbs-dash-infra) in the GitHub organization — separate from application code so infra reviewers and app reviewers are distinct CODEOWNERS populations:

sbs-dash-infra/
├── tofu/
│   ├── environments/
│   │   ├── prod/          # prod VPC, droplets, firewalls, volumes
│   │   └── dev/           # dev VPC, droplet
│   ├── modules/
│   │   ├── droplet/       # standard droplet: VPC member, tags, monitoring
│   │   ├── firewall/      # tag-scoped Cloud Firewall pattern (§3.5)
│   │   └── spaces/        # bucket + scoped key pattern (§7.3)
│   ├── dns/               # zone records post-cutover (§4.3 target table)
│   └── globals/           # admin SSH allowlist, region, naming — the named
│                          #   variables referenced throughout this spec
├── ansible/
│   ├── inventory/         # generated FROM tofu output — single source for IPs
│   ├── roles/
│   │   ├── base/          # users, sshd, patching (§2.5 target, INF-05)
│   │   ├── nftables/      # §3.6 templates incl. per-Compartment chains
│   │   ├── nginx-edge/    # §5.3 TLS params + §5.5 header template
│   │   ├── monitoring/    # prometheus jobs, exporter binds, dashboards
│   │   ├── vault/         # §8.2 server config
│   │   └── compose/       # §6.4 Compartment template renderer
│   └── playbooks/
├── vault/policies/        # §8.3 policy HCL, applied via CI
├── .gitleaks.toml
└── CODEOWNERS

Repository controls (INF-03, shared baseline with Ch. 11): default branch protected — PR required, ≥1 review from CODEOWNERS, no force-push, no direct commit, signed commits required, CI green as merge condition; admin bypass disabled; the GitHub org and repo live under SBS ownership with Ledger Hub engineers as members (same revocability principle as §2.1).

10.4 Change Workflow

Every infrastructure change follows one path — the §2.2.3 promotion rule made concrete:

1. Branch + PR against sbs-dash-infra. PR description states intent and links the Build Register item or Change Order where applicable (LH-SBS-CO- series for scope-level changes; routine parameter changes need only the PR).
2. CI validation on PR: tofu validate + tofu plan (both environments, plan output posted to the PR), ansible-lint, template syntax checks (nginx -t in container, promtool check config), gitleaks sweep. A plan showing destruction of stateful resources (Volumes, databases, Spaces buckets) requires an explicit destructive-change label + second reviewer.
3. Review by CODEOWNERS — for firewall, Vault-policy, or IAM-adjacent paths, the reviewer set includes the security owner (Alexandra Del Rey routing).
4. Apply to Dev on merge: CI runs tofu apply (dev workspace) + Ansible against dev inventory automatically.
5. Apply to Prod as a manually approved CI job (GitHub environment protection, named approvers) — never from an operator laptop. The CI identity authenticates to Vault via its own AppRole (§8.5) for any secret-touching render; the DO token it uses is the pipeline-scoped token of §2.1.
6. Emergency path: break-glass console/shell changes are permitted to restore service (MON-05 incident context), and must be back-ported to the repo within 24 h — the next drift run (§10.5) enforces this mechanically by flagging anything that wasn't.

10.5 Drift Detection

Drift — divergence between repo-declared state and live state — is the failure mode this whole chapter exists to prevent, and every preceding chapter's "console edits are drift and reverted" clauses land here as mechanism:

• Provisioning drift: scheduled CI job runs tofu plan -detailed-exitcode nightly per environment. Exit code 2 (changes present) → alert to the ops Slack channel (Ch. 12 routing) with the plan diff. Triage outcomes: revert the live change (apply), or adopt it (PR that codifies it) — silent coexistence is not an outcome.
• Configuration drift: nightly ansible-playbook --check --diff against full inventory; any changed-task output alerts identically. Ansible runs are also idempotent enforcement — the weekly scheduled real run re-asserts declared state, so unrecorded manual edits have a maximum lifetime of seven days.
• Ruleset integrity: live nftables ruleset hash and running-container-vs-compose comparison (§3.6, §6.7) are exported as Prometheus metrics; mismatch alerts in minutes rather than nightly — these two are the fastest-moving tamper indicators (MON-01 inputs).
• DNS drift (post-cutover): nightly diff of provider-API zone dump vs. tofu/dns/ — catches registrar-console edits (§4.7).
• Drift alerts are not auto-remediated on Prod (auto-apply of a malicious or mistaken repo state is its own attack path); human triage stands between detection and correction, with the weekly enforcement run as the backstop.

10.6 Bootstrap and Migration of As-Built Resources

The existing fleet was built by hand; it is imported, not rebuilt, except where the §3.1 rebuild wave already mandates recreation:

1. Stand up repo, branch protection, CI skeleton, gitleaks (already Build Register item 47).
2. tofu import current resources: 4–5 droplets, existing ufw-era firewall facts recorded as-is, Spaces buckets when created. Import produces an honest baseline whose first plan output is the formal nonconformance list — the delta between as-built and this specification, machine-generated.
3. Author target state per this spec (dedicated VPCs, Cloud Firewalls, prod-vault droplet); the §3.1 rebuild wave then executes as reviewed PRs — the rebuild becomes the first real exercise of the §10.4 workflow rather than another manual event.
4. Ansible baseline role applied fleet-wide, which is the execution vehicle for item 17 (per-person users, sshd hardening) and item 20 (host firewalls).
5. SERVER_CHANGELOG.md thereafter records narrative/context only; "what exists" questions are answered by the repo and its state, and the changelog's §1.3 role is updated accordingly.

10.7 Verification

Gates (LH-SBS-INST-001): repo exists under SBS org with branch protection verified (settings API dump); direct push to default branch rejected (negative test); PR without CI green cannot merge; tofu plan clean on both environments (zero drift at gate time); nightly drift jobs scheduled and alerting path tested (induced drift: manual test-tag change → alert received → reverted); Prod apply requires named approval (job history evidence); emergency-change backport rule exercised in drill; gitleaks blocks a seeded dummy secret in pre-commit and CI; inventory generated from tofu output matches live fleet.

Supply-Chain Controls

Chapter 11 — Supply-Chain Controls

11.1 Current State and Scope

As-built supply-chain posture is minimal but deliberately clean so far: everything installed to date (Docker, Compose, Prometheus, nginx, certbot) came from Ubuntu official repos or the official Grafana apt repo — no curl-pipe-bash installs, no third-party PPAs, no unpinned container images yet, because no application containers exist yet. This chapter sets the controls before the application build starts pulling npm/crates/PyPI dependencies and publishing images — retrofitting provenance onto an already-polluted dependency tree is an order of magnitude harder than starting gated. Scope: container images, OS packages, language dependencies (Node/pnpm, Rust/Cargo, Python), the artifact registry, and the build pipeline that connects them. Controlling SEC-001 requirement: INF-03 (repository controls) plus the supply-chain family imported into KO-001; AI-02 intersects where MCP tool containers are concerned (§6.4).

11.2 Registry Policy

GitHub Container Registry (GHCR) under the SBS GitHub organization is the sole artifact registry:

• One registry, private visibility, org-owned — same ownership/revocability principle as §2.1 and §10.3. No Docker Hub publishing; no images pulled from Docker Hub except the official library base images enumerated in the allowlist (§11.3).
• Pull access from droplets via a registry-scoped read-only token stored in Vault (secret/platform/ghcr/), rotated on the 90-day static schedule (§8.4). Push access belongs to CI only — no human pushes images; an image that wasn't built by the pipeline doesn't exist as far as deployment is concerned.
• The §3.4 egress allowlist admits exactly: GHCR endpoints, Ubuntu archive/security mirrors, the Grafana apt repo, and the language registries (registry.npmjs.org, crates.io/static.crates.io, pypi.org/files.pythonhosted.org) from build contexts only — production droplets pull finished images, never raw dependencies; the language registries are unreachable from prod runtime (a compromised container cannot pnpm add its way to a payload).

11.3 Image Provenance

• Base image allowlist, pinned by digest, reviewed quarterly: official node:<LTS>-slim, rust:<pinned>-slim (build stage only), python:3.x-slim, nginx:stable, postgres/mongo/redis official images for the §6.5 interim, hashicorp/vault. Nothing outside the list without a PR touching the allowlist file (CODEOWNERS: security reviewer per §10.4).
• Digest pinning everywhere: FROM node:22-slim@sha256:… in Dockerfiles; compose files reference deployed images by digest, not floating tags (:latest is prohibited and CI-linted). Tag-based human readability is preserved by also tagging, but the digest is what deploys — a registry-side tag repoint cannot swap what prod runs.
• Multi-stage builds — build stages (compilers, package managers) never ship; runtime stages carry the application and its production dependencies only. Runtime images run as non-root UID, no shell where the base permits (distroless-style trimming is a later optimization, not a Day-1 gate).
• Rebuild cadence: weekly CI rebuild of all images against updated base digests + apt upgrade layer, so CVE patching is a pipeline event, not an emergency (§INF-05 patching applies to images as much as hosts). Trivy scan (§11.6) gates the result.

11.4 Dependency Pinning

Per-ecosystem, all enforced in CI (a lockfile that isn't verified is documentation, not control):

Dependency updates are PRs like any other change (Renovate bot, grouped weekly, auto-PR but never auto-merge) — the update path goes through the same review + CI gauntlet, so a poisoned upstream release sits in a diff a human looks at, not in a silent nightly pull. New top-level dependencies require a one-line justification in the PR body; transitive bloat is reviewed via the lockfile diff.

11.5 Artifact Signing and Verification

• Commits: signed commits required on both repos (already a §10.3 branch-protection condition) — SSH-key signing acceptable; the key inventory rides the per-person identity work of item 17.
• Images: CI signs every pushed image with cosign keyless (Sigstore, GitHub OIDC identity) — the signature attests "built by this workflow, from this commit, in this repo," which is the provenance claim that matters. No long-lived signing key to protect or rotate (the §8 lifecycle problem cosign keyless exists to delete).
• Verification at deploy: the deploy step (§6.7 / §10.4) runs cosign verify against the expected workflow identity before compose up; an unsigned or wrongly-attested image fails the deploy. This closes the loop: registry compromise alone cannot inject a runnable image, because the attacker must also produce a valid Sigstore attestation from the protected CI workflow.
• SBOM: CI generates SPDX SBOMs (syft) per image, attached as a cosign attestation and archived to sbs-dash-backups — when the next log4shell-class event lands, "are we exposed" is a query against stored SBOMs, not an archaeology project.

11.6 Pipeline Integrity and Scanning

The build pipeline is itself supply chain:

• GitHub Actions pinned by action digest (not @v4 floating tags) — the same rule applied to the tooling as to the images; third-party actions from a reviewed allowlist only.
• Workflow permissions default contents: read; the push job alone gets packages: write + id-token: write (for cosign OIDC). No self-hosted runners at this fleet size — GitHub-hosted runners keep runner compromise off the threat sheet at the cost of trusting GitHub, which the design already does (repo, registry).
• Trivy scans every built image (CI gate: fail on CRITICAL with fix available; HIGH warns and files an issue) and re-scans the deployed digest set nightly against fresh CVE data — the nightly catches vulnerabilities published after the image shipped, feeding MON-06 (vulnerability scanning) with the §11.3 weekly rebuild as the remediation vehicle.
• gitleaks (already items 47/§10.4) covers the secrets half of pipeline hygiene.

11.7 Verification

Gates (LH-SBS-INST-001): base-image allowlist file exists, all Dockerfiles resolve FROM digests within it (lint); :latest absent from every compose/Dockerfile (grep gate); frozen-lockfile builds pass and a deliberately drifted lockfile fails (negative test); cosign verify passes on a pipeline image and fails on a manually-pushed unsigned test image (negative test); deploy refuses the unsigned image end-to-end; SBOM attestation present for every deployed digest; Trivy CI gate demonstrated on a seeded-CVE test image; language registries unreachable from prod runtime (egress probe, §3.4); Actions digests pinned (workflow lint); prod running-set digests ⊆ registry signed-set (drift cross-check with §10.5).

Observability

Chapter 12 — Observability

12.1 Current State and Scope

Prometheus + Grafana 13.1.0 + nginx/TLS on monitor-servers, serving monitor.sbsdash.com; node_exporter on all four droplets (prod/dev private-bind at 10.100.0.3/.2:9100, wiki public-bind with ufw scope — the §3.3 interim, monitor local at 127.0.0.1); scrape job droplets with per-server labels; dashboards "Node Exporter Full" (ID 1860) and custom "SBS Servers Overview" (uid sbs-overview, provisioned as home dashboard); 30-second kiosk playlist. Verified 2026-07-17: 5 targets up, promtool clean, Grafana healthy.

What exists is metrics + visualization. This chapter completes the design: logging pipeline (absent), alerting (absent — dashboards without alerts is observability that only works when someone is looking), blackbox/health probes (absent), and the MON-family feeds every prior chapter has been promising (MON-01 unified audit trail inputs, MON-02 anomaly detection). Grafana local-admin → OIDC migration is already item 52 (Ch. 9); Prometheus job additions below fold into the Ansible monitoring role (§10.3) rather than remaining hand-edits.

12.2 Metrics (Prometheus)

Retained as deployed — apt-installed system service on monitor-servers, 127.0.0.1-bound behind nginx (§6.6 rationale: don't containerize the watcher). Extensions, all IaC-managed:

Retention: 30 days local TSDB (sized fine on 120 GB); longer-horizon capacity trending is served by the 30-day window plus quarterly-review screenshots — remote-write long-term storage is a Chapter 16 growth item, not Day-1. Scrape interval stays 15 s. Every new exporter binds VPC-private or localhost — the §3.7 public-listener gate applies to observability itself.

12.3 Logging Pipeline

Grafana Loki + promtail/alloy agents — chosen over an ELK stack deliberately: one vendor surface with Grafana (logs and metrics correlate in the same UI), index-light architecture that fits a 4 GB monitor droplet, and LogQL's label model matches the Prometheus labels already in place.

• Loki on monitor-servers, 127.0.0.1-bound, filesystem storage on a dedicated DO Volume; retention 90 days hot. Audit-relevant streams are additionally shipped to the Ch. 13 hash-chained archive — Loki is the operational search tier, not the evidentiary store (different integrity guarantees, different retention: 90 d vs 7 y).
• Agents on every droplet (Ansible role), shipping over the VPC (wiki joins post-rebuild; until then its logs ship over TLS to the monitor's public interface — same interim-acceptance logic as its scrape path, §3.3):

12.4 Alert Routing

Alertmanager on monitor-servers (the changelog's own "pendiente" item, now normative), 127.0.0.1-bound:

• Channels: ops Slack channel (webhook from Vault secret/platform/alerting/) for everything; email to named on-call for severity=critical as the Slack-independent second path. The per-Compartment Slack channel fabric (platform-level alerting, §5.2 of the SOW-side design) is application-layer and separate — this section is infrastructure alerting; the two fabrics share the Slack workspace but not routes.
• Severity model: critical = platform-down/data-risk (surface down ≥ 2 min, cert ≤ 7 d with renewal failing, disk ≥ 90 %, Vault sealed, backup age > 26 h, integrity-hash mismatch, target down ≥ 5 min); warning = degradation (disk ≥ 80 %, cert ≤ 14 d, CPU/RAM sustained ≥ 85 % 15 min, drift detected, Trivy nightly CRITICAL finding, egress-denial burst, failed-auth burst ≥ 10/5 min/IP); info = notable (deploys, rotation confirmations, weekly Ansible enforcement diff).
• Routing: group by alertname+instance, 4 h repeat for unresolved critical, inhibition (droplet-down inhibits its per-service alerts). Dead-man's switch: a permanently-firing Watchdog alert routes to a heartbeat receiver — if the watchdog goes silent, the monitoring stack itself is down; this is the alert that covers the monitor (§12.6 covers it structurally).
• Alert rules live in the repo (monitoring role), promtool-validated in CI (§10.4) — alert changes are PRs, not console edits.

12.5 Dashboards

Provisioned-by-file discipline (as already practiced — sbs-overview and 1860 are file-provisioned) becomes the rule: no console-authored production dashboards; Grafana UI is for exploration, dashboards graduate to the repo. Dashboard set:

12.6 Monitoring the Monitor

The stack's structural blind spot, addressed three ways: the dead-man's switch (§12.4); an external uptime check on https://monitor.sbsdash.com from outside DO (UptimeRobot free tier or equivalent — the one approved external service, alerting to the same email path); and the monitor droplet's own metrics riding the same droplets job with its alerts routed identically. Loki/Alertmanager/blackbox process health via systemd unit-state metrics on the integrity job. The bastion role (§3.3) already makes this host priority-patched; observability adds priority-watched.

12.7 Verification

Gates (LH-SBS-INST-001): all §12.2 jobs green with private binds confirmed (ss -tlnp sweep per droplet); blackbox probes cover all five URLs incl. cert-expiry metric present; induced-failure drill — stop a service, alert received on both channels inside SLA, inhibition correct; dead-man silence drill — stop Alertmanager, heartbeat receiver fires; Loki streams present per table incl. Vault audit and egress denials (LogQL sample per stream); Compartment log visibility probe — c03 group cannot read c08 stream (mirrors §7.7/§9.7); dashboard set file-provisioned, console-save on prod folders denied; alert rules pass promtool in CI, console-edit reverted by drift run; external uptime check firing path tested; retention conf 30 d TSDB / 90 d Loki confirmed; wiki agent path migrates to VPC at rebuild (carried gate).

Audit Infrastructure

Chapter 13 — Audit Infrastructure

13.1 Purpose and Distinction from Operational Logging

This chapter specifies the evidentiary tier: the tamper-evident, hash-chained audit record backing the Platform's compliance claims — every AI Agent decision tagged by Risk Tier, admissible for FCC and SOX review, retained seven years minimum (per the engagement's audit-log commitments). It is deliberately separate from Loki (§12.3), and the separation is the design:

As of 2026-07-18 nothing is built; nothing blocks it either — the storage substrate (sbs-dash-audit bucket, §7.3) and the event sources (§13.3) are already specified. This is the audit layer of the eight-layer Compartment isolation model (§1.2) and the infrastructure realization of MON-01 (unified audit trail), consumed by Guardian Agents as writers and by SBS/regulators as readers.

13.2 Hash-Chain Architecture

Design: per-Compartment append-only event chains, batched into signed, anchored segments.

• Event record. Every auditable event is a canonical JSON document: {seq, ts (RFC 3339 UTC), compartment, actor (agent id | human identity per §9), event_type, risk_tier, payload_hash, prev_hash, entry_hash} where entry_hash = SHA-256(canonical_serialization(record minus entry_hash)) and prev_hash is the previous event's entry_hash. Canonicalization is fixed (RFC 8785 JCS) — hash stability across implementations is what makes verification portable to an auditor's independent tooling.
• Payload separation. The chain entry carries the hash of the full decision payload, not the payload itself; full payloads (prompts, tool calls, outputs — potentially large, potentially sensitive) are stored adjacent in the same segment object, addressed by payload_hash. The chain stays compact and verifiable in isolation; payload disclosure decisions (e.g., what an auditor sees) are separable from integrity proof.
• Ten chains, not one. One chain per Compartment (c01…c10) plus a platform chain (identity events, break-glass use, Vault audit summaries, infra changes from §10.4). Per-Compartment chains preserve the isolation model in the evidence itself — Compartment 03's audit history is disclosable to an auditor without touching Compartment 08's — and keep verification parallelizable. Cross-chain ordering, where ever needed, is provided by the anchor layer, not by merging chains.
• Segments. The writer service (§13.3) closes a segment every hour or 10 000 events, whichever first: segment = {chain_id, seq_range, first_hash, last_hash, event_file, payload_file}, uploaded to sbs-dash-audit under {chain}/{yyyy}/{mm}/{dd}/{segment_id}.tar.zst with a manifest. Segment manifests are cosign-signed (keyless, the CI/service identity — same machinery as §11.5, no new key lifecycle).
• Anchoring (the external trust root). Each segment's last_hash is anchored outside the platform's own control so that even a full-platform compromise cannot silently rewrite history: (a) appended to a daily anchor digest — one line per chain per day — which is (b) cosign-signed and (c) submitted to the Sigstore Rekor public transparency log, whose inclusion proof is stored alongside. Rekor gives an independent, publicly verifiable timestamp for free, using tooling already in the stack. The daily anchor digest is additionally emailed to the SBS security alias (§4.4) — a low-tech second anchor in SBS-controlled custody that no platform credential can retract.

13.3 Write Path and Event Sources

• Audit writer service — a small dedicated service (one per environment, prod's on sbsdash-server-prod, its own compose network net-audit): receives events on a VPC/localhost-only endpoint, assigns seq/prev_hash/entry_hash, appends to the active segment on a dedicated DO Volume (write buffer), closes/uploads/anchors segments. It holds the only credential with PUT access to sbs-dash-audit (§7.3 write-once posture: PUT-only, no DELETE, no overwrite — bucket versioning on as belt-and-braces).
• Sources (deduplicated against §12.3 — same events, second delivery, different guarantee): Guardian Agent decision records (the primary stream — every agent action with Risk Tier, per SOW-side commitments); Supervisor routing/resolution events (Medium/High/Critical human decisions, §9.5 supervisor groups); identity events (§9.4 SCIM lifecycle, break-glass use §9.6); Vault audit device (§8.4); infrastructure changes (merged PRs, Prod applies, emergency backports §10.4); egress-denial summaries (§3.4).
• Delivery semantics, stated honestly: at-least-once from sources to writer with idempotent dedup on (source, source_seq); the writer's local buffer means a writer crash loses at most the unflushed tail (fsync per event for Guardian decisions — the stream where loss is least acceptable; batched fsync elsewhere). The chain structure means loss is always detectable (sequence gap) even where not preventable — a gap is an incident (MON-05), never silently renumbered.
• Backpressure rule: if the writer is unavailable, Guardian Agents queue locally and continue enforcing (availability of the platform does not hinge on the audit path), but a queue older than 15 minutes is a critical alert (§12.4) — extended audit outage is treated with the severity of a control failure, because it is one.

13.4 Retention and Lifecycle

• Seven years minimum, rolling, per the engagement's FCC/SOX-grade retention commitment; segments carry their close-date and become deletion-eligible only after close + 7y, and deletion is a manual, dual-controlled annual ceremony (SBS sign-off via Alexandra Del Rey routing + Ledger Hub execution), logged — there is no automated deletion path at all, which is the safest failure mode for an evidentiary store.
• Storage layout: hot segments in sbs-dash-audit (NYC3), replicated to the SFO3 backup bucket on the §14 schedule — two regions, plus the Rekor/email anchors which survive even dual-bucket loss (anchors prove what the data was; replication ensures the data survives — both are required, neither substitutes).
• Capacity honesty: at plausible agent volumes (10 Compartments × thousands of decisions/day), zstd-compressed JSON runs low single-digit GB/year for chains and tens of GB/year with payloads — Spaces-trivial; the 7-year cost concern is nil. The Volume write buffer is sized for 72 h of disconnected accumulation.
• Legal hold: a hold flag on any chain/date-range suspends deletion eligibility indefinitely; holds are themselves platform-chain events.

13.5 Integrity Verification

Verification is continuous and layered — an integrity system that is only checked when someone subpoenas it is not an integrity system:

• Continuous (automated): a verifier job on monitor-servers (deliberately a different host than the writer — the checker must not share the writer's compromise) re-downloads a rolling sample daily: yesterday's segments fully, plus a random historical segment per chain per week. It recomputes every entry_hash, walks prev_hash continuity across segment boundaries, checks cosign signatures, and verifies Rekor inclusion proofs. Results export as integrity job metrics (§12.2); any mismatch is critical (§12.4) and a MON-05 incident.
• Full-chain verification: quarterly, complete re-verification of all chains end-to-end (compute-cheap at these volumes), timed with the quarterly executive review cadence — the review receives a signed verification report, generated evidence rather than asserted health.
• Independent verification path (the point of the whole design): a standalone, dependency-light verifier CLI (single static binary, in the repo, itself cosign-signed) that an SBS auditor or regulator can run against exported segments without any Ledger Hub system access: input segments + the public Rekor log + the emailed anchor digests → PASS/FAIL per chain. Admissibility rests on this property — the evidence verifies against public roots, not against the operator's word.
• Tamper response: verification failure freezes the affected chain's deletion eligibility, triggers incident response (MON-05), and the last-good anchor bounds the affected window precisely — the chain structure turns "was anything altered?" into "these specific sequence ranges after this timestamp are suspect."

13.6 Verification (Build Gates)

Gates (LH-SBS-INST-001): writer reachable only on net-audit/VPC (public + cross-Compartment probe fails); Guardian test event → chained entry with correct prev_hash linkage (recompute check); segment close → bucket object + cosign signature + Rekor inclusion proof all present; PUT-only posture proven (writer credential DELETE attempt fails, §7.7 carried gate); induced-tamper drill — modify one byte in a staged segment copy, continuous verifier alerts within its cycle; sequence-gap drill — drop an event in staging, gap detected and alerted; independent verifier CLI produces PASS on clean export and FAIL on the tampered copy, run from a machine with zero platform access; queue-age alert fires in writer-outage drill; anchor email arriving at SBS alias confirmed; 7-year lifecycle rules + hold flag present in IaC with no automated deletion path (code review gate).

Backup & Disaster Recovery

Chapter 14 — Backup & Disaster Recovery

14.1 Current State and Scope

As of 2026-07-18, tier 1 (DigitalOcean Backups) is enabled on the droplet fleet — the first backup layer in production. Tiers 2 and 3 remain unbuilt: no database dumps (no databases yet), no repo mirrors, no Vault snapshots (no Vault yet), no offline custody artifacts. Single-droplet loss is now recoverable; application-consistent recovery, region loss, and account compromise are not yet covered. Scope of this chapter: everything required to reconstruct the Platform (data, configuration, state, secrets, evidence); the recovery procedures per failure class; and the test cadence that keeps "we have backups" meaning "we can restore." Chapter 13's audit anchors are the integrity counterpart; this chapter is survivability. Controlling family: the backup/DR control set imported into KO-001.

14.2 Backup Architecture — Three Tiers

The design principle: IaC-first recovery — most of the platform is rebuilt from code, not restored from images. Three complementary tiers, each covering a failure class the others cannot:

14.2.1 Tier 1 — DigitalOcean Backups (Live)

Enabled on the fleet as of 2026-07-18. Target configuration:

Open verification items against the live enablement (Build Register item 96 residual):

• Plan and settings audit — confirm which plan is active (usage-based, GA since Nov 2025 and priced by restorable file size with configurable frequency/retention, vs. legacy flat-percentage) and the actual per-droplet frequency selected (usage-based plans allow 4 h / 6 h / 12 h / daily / weekly). Align to the target table or amend it.
• Codify in tofu at the §10.6 import — enablement was performed via console; once imported, a droplet created without backups surfaces as a plan diff rather than a discovered gap. Console-enabled today is acceptable as bootstrap; console-managed permanently is drift.
• Dev-Gitlab-server — uninventoried (§2.2.2) but currently serving the apex record; take one backup/snapshot of it before the DNS repoint as cheap insurance, regardless of its final disposition.

Ongoing tier-1 mechanics: monthly, one Prod backup per droplet is converted to a snapshot and retained on a 12-month rolling basis (backups convert to snapshots for indefinite storage — the medium-term image tier). Restore paths are in-place restore or new-droplet-from-backup; creation runs roughly 2 min/GB used, keeping backup windows in minutes at our footprint.

Position in the design, stated plainly: tier 1 is the speed layer for single-droplet loss only. Disk images are crash-consistent — a mid-write PostgreSQL image can restore dirty, so databases recover from tier 2; and an image restores whatever existed at image time, drift and compromise included, so the IaC rebuild (§10) remains the clean-room path.

14.2.2 Tier 2 — Application Artifacts (What Code Cannot Regenerate)

All dumps are encrypted client-side before upload (age; key in Vault, with the age identity also held offline in SBS custody per the §9.6 break-glass pattern — an encrypted backup whose only key lives in the Vault it must survive is a circular dependency, broken here) and land in sbs-dash-backups (NYC3) with lifecycle replication to SFO3.

14.3 Schedule and Retention

Every tier-2 job emits completion + size + age metrics to the integrity job (§12.2); tier-1 backup age is polled from the DO API per droplet and exported to the same job — now actionable immediately and prioritized, since it is the only automated proof the live backups are actually completing. Any backup age > 26 h (tier-1 daily set or tier-2 nightly set) is critical (§12.4) — a silently failing backup is the canonical DR failure and is alerted, not discovered.

14.4 RPO / RTO Targets

Per failure class — one blended number would be fiction:

Until tier 2 exists, the honest fleet-wide posture is: RPO for any future data = undefined; recoverability = single-droplet mechanical failure only. This line is deleted from the specification when items 86–92 close — it is written here so the gap is a stated fact, not an implication. Database RPO ≤ 24 h is the accepted Day-1 posture once dumps run; the WAL-archiving upgrade to ≤ 5 min is triggered when the first Compartment carries production tenant data — not before (cost honesty), mandatorily not after.

14.5 Restore Procedures

Each is a wiki runbook (§1.4 publication target) with exact commands and a verification tail; summarized:

1. Single-droplet fast path (tier 1 — available today): restore image or create-from-backup → verify drift (tofu plan + Ansible --check clean against repo state once §10 exists; until then, manual diff against SERVER_CHANGELOG.md — the interim reality, stated) → rejoin monitoring → smoke test.
2. Database restore (tier 2): provision clean engine (IaC) → decrypt dump (age key from Vault, or offline custody copy if Vault is the casualty) → restore per-database → re-assert per-Compartment roles/ACLs from IaC before any application start (a restored database with default grants is an isolation breach, §7.4) → row-count comparison vs. manifest → smoke test.
3. Vault restore: IaC/image droplet → raft snapshot restore → quorum unseal → audit-device continuity check → rotate any credential issued after snapshot timestamp (derivable from the Vault audit stream).
4. Full-region DR: run-order: VPCs → Vault (secrets first — everything authenticates through it) → data services + restores → app droplets + deploys → monitor → DNS cutover (TTLs drop to 300 at DR declaration) → §12.7-style verification sweep before declaring recovery.
5. Restore-time isolation rule: Compartment agent workloads stopped during recovery — agents never run against partially restored data (Guardian policy state could lag data state); restart is gated on the verification tail and is itself an audit-chain event (§13.3).

14.6 Worst-Case: Account/Custody Compromise

The survivability floor. If the DO account and GitHub org are both lost or hostile: tier 1 is gone by definition (backups are not downloadable and live inside the account); in-account tier 2 is presumed gone. Reconstruction requires exactly the tier-3 set — repo mirrors (SFO3 replicas), encrypted dumps + offline age identity (SBS custody), Vault snapshot + Shamir shares (5 named holders, §8.2), and audit anchors (Rekor + SBS email, §13.2). Each lives outside the compromised trust domain by design. The runbook is the incident-response DR annex (MON-05 linkage); the §14.7 annual tabletop rehearses this scenario specifically.

14.7 DR Test Cadence

Untested backups are hypotheses. Each test produces a dated wiki report (MON-04/quarterly-review evidence):

14.8 Verification (Build Gates)

Gates (LH-SBS-INST-001): DO Backups confirmed active on all five droplets with plan/frequency audit closed and settings codified in tofu (item 96 residual); tier-1 backup-age visible via DO API and exported to integrity per droplet; first tier-1 restore drill passed; every §14.2.2 artifact present in sbs-dash-backups with age metric green; SFO3 replication object-count parity; dump decrypts with the offline-custody key copy (circular-dependency break proven, not just the Vault copy); weekly restore-verify green ≥ 2 consecutive cycles pre-Cutover; Vault snapshot restore drill passed; backup-age alert fires in induced-failure drill; monthly snapshot-conversion automation present in IaC; restore runbooks on wiki with last-tested dates; WAL-PITR trigger condition documented; region variable proven by clean tofu plan targeting SFO3; Dev-Gitlab-server pre-repoint backup taken.

Capacity & Scaling

Chapter 15 — Capacity & Scaling

15.1 Philosophy and Scope

Capacity planning at this fleet size is not autoscaling — it is knowing the numbers, naming the thresholds, and pre-writing the procedures so scaling is a scheduled maintenance action rather than an incident response. The Platform's load profile helps: Compartment agent workloads are queue-driven and tolerate minutes of latency (a traffic-log reconciliation that waits 5 minutes is not an incident), so the design scales for throughput over time, not burst absorption. The only latency-sensitive paths are the human web surfaces and the audit write path. Scope: droplet sizing, per-Compartment resource budgets, storage growth, the thresholds that trigger action, and the procedures per scaling event. Growth triggers named in earlier chapters (Vault HA §8.2, remote-write metrics §12.2, hot-standby DR §14.4, private DNS §3.3) consolidate here.

15.2 Baseline Sizing and Budgets

As-built fleet totals: 22 GB RAM / 11 vCPU / 500 GB disk across five droplets (§2.2.1, incl. prod-vault at creation). Per-droplet budgets:

sbsdash-server-prod (8 GB / 4 vCPU / 160 GB) — the contended host. Budget under the §6.5 interim (data services colocated):

Consequence, stated plainly: the 8 GB prod droplet supports ~3 concurrent Compartment stacks in the interim topology. That matches the engagement's minimum-Compartment starting footprint, but full ten-Compartment operation on this host requires either the data-service offload (Managed DBs, §7.2) or the resize/split ladder (§15.4) — this is a designed checkpoint, not a surprise.

Other hosts: monitor-servers (4 GB) — Prometheus 30 d TSDB ≈ 2–6 GB disk at current cardinality, Loki 90 d on its Volume; RAM adequate until Loki ingest grows with Compartment count (threshold below). prod-vault (1 GB) — trivial load, no growth coupling. sbs-wiki (2 GB) — static. Dev-SBS-server — mirrors prod budgets by design (§2.4).

15.3 Growth Thresholds (Named Triggers → Named Actions)

Thresholds are Prometheus alert rules (§12.4 warning tier unless noted), each mapped to a §15.4 procedure — a threshold without a pre-decided action is just anxiety:

15.4 Scaling Procedures

All procedures are IaC changes (§10.4 workflow) executed in maintenance windows; none is novel at execution time because each is pre-written as a wiki runbook:

• P1 — Droplet resize (vertical). tofu size change → PR → apply = powered-off resize (~1–3 min downtime, disk+CPU+RAM resize; reserve the disk-inclusive variant so storage grows too). Order of preference: resize prod before adding hosts — vertical is operationally free until §15.3's 16 GB ceiling. Tier-1 backup taken pre-resize (fast rollback = restore + revert PR).
• P2 — Data-service offload. The designed inflection at Compartment #4: Managed DBs available in NYC2 (item 36) → migrate per §7.2 (dump/restore per engine, maintenance window, Change Order) → frees ~3 GB on prod → Compartment ceiling rises to ~7–8 on the same droplet. If Managed DBs unavailable → dedicated prod-data droplet (8 GB) in the Prod VPC running the §6.5 compose stacks — same isolation model, same entitlement networks, one hop away.
• P3 — Storage grow. DO Volumes resize online (grow-only); filesystem expand follows in the same window. Root-disk pressure on droplets without Volumes → P1 disk-inclusive resize. Spaces is elastic — no procedure, only the cost line.
• P4 — Observability scaling. Monitor resize (P1 pattern) or retention reduction — an explicit reviewed trade, never silent data loss.
• P5 — Audit path tuning. Volume IOPS/size bump, fsync batching review (§13.3 semantics preserved — Guardian stream stays per-event), segment-close interval tuning.
• P6 — Horizontal split. When prod outgrows vertical: split along the already-drawn lines — data tier out first (P2's droplet variant), then Compartments 6–10 to a second app droplet (net-edge spans hosts via VPC; nginx upstreams update by IaC variable). The compose-per-Compartment design (§6.4) makes Compartment placement a scheduling decision, not a re-architecture — this is the payoff for refusing the mega-compose.
• Scale-down is the same procedures reversed, gated on 30 days below 50 % of the triggering threshold — flap-damping for capacity decisions.

15.5 Capacity Review Cadence

Quarterly (aligned with the §12.5/§13.5 quarterly review artifacts): dashboard-sourced trend review of RAM/CPU/disk/ingest slopes per host, Compartment-count forecast against the P2 checkpoint, cost line (droplets + Volumes + Spaces + DO Backups usage-based charges + Anthropic consumption trend from egress/billing data), and a written one-page capacity position filed to the wiki. The review's only mandatory output: confirmation that the next scaling event is known, named, and scheduled — or explicitly "none within horizon."

15.6 Verification (Build Gates)

Gates (LH-SBS-INST-001): §15.3 thresholds present as alert rules in the repo (promtool-validated, mapped action in annotation); §6.4 resource limits present on every Compartment service and OOM/throttle events exported as metrics; P1 resize drill executed once on Dev (timed, rollback path proven); Compartment-deploy RAM precondition check wired into the deploy pipeline (blocked-deploy negative test); P2 decision documented and item 36 closed before 4th-Compartment Workorder acceptance; capacity runbooks (P1–P6) published to wiki; first quarterly capacity position filed.

Maintenance & Patching

Chapter 17 — Maintenance & Patching

17.1 Current State and Scope

As-built patching is unmanaged: no unattended-upgrades confirmation, no defined windows, no reboot policy, and one already-tracked drift instance (OS mix 24.04/26.04, item 21). This chapter defines the patch regime for every layer — OS packages, kernel/reboots, container images, system services (nginx, Prometheus, Grafana, Vault, BookStack), data engines, and the DO platform events outside our control — plus the rollback path per layer. Controlling requirement: INF-05 (patching); MON-06 (vulnerability scanning, §11.6) is the detection feed this chapter remediates. The governing principle: patching is a pipeline event with a rollback, never an SSH session with apt upgrade and hope.

17.2 Patch Windows and Cadence

Ordering rule, always: Dev first, soak ≥ 24 h (weekly cycle) with monitoring green, then Prod. The 1:1 Dev sizing (§2.4) exists precisely so this soak is meaningful. Within Prod, order is: wiki → monitor → vault → prod-app — least-critical first, and the bastion/alerting host (monitor) is patched before the app host so a monitor regression is discovered while the app host is still stable, never both at once.

17.3 Patching by Layer

• OS security patches: unattended-upgrades enabled fleet-wide (Ansible base role, item 60) — security pocket only, no automatic reboots, mail/log output shipped to Loki. This makes security patching continuous and the windows about coordination, not application.
• Kernel/reboot: needrestart/reboot-required flag exported as an integrity metric (§12.2); pending-reboot age > 14 days = warning. Reboots execute in the monthly window, one host at a time, in the §17.2 order, with the bastion-loss interim SSH note (§14.4) pre-staged when monitor reboots. DO Backup taken pre-reboot on each host (tier-1 as rollback, §14.2.1).
• Container images: already fully specified — weekly CI rebuild against updated bases + Trivy gate (§11.3/§11.6); this chapter adds only the rollout rule: rebuilt images deploy to Dev on build, Prod in the monthly window or the emergency path. Rollback is redeploy-previous-digest (§6.7) — the strongest rollback in the whole stack, which is why the design pushes as much as possible into images.
• System services (nginx, Prometheus, Grafana, Loki, certbot): apt-managed on their hosts; minor versions ride the monthly window. Grafana major versions (13 → 14 class) get a Dev-first upgrade with dashboard/provisioning verification before Prod — Grafana majors have history of provisioning-format changes, and the §12.5 file-provisioning rule is what makes this testable.
• Vault: upgrade = new binary via Ansible, service restart, unseal quorum required (§8.2) — so Vault upgrades are scheduled with shareholders, monthly window only, never emergency-path unless the CVE is Vault-critical. Raft snapshot taken immediately pre-upgrade; rollback = binary revert + snapshot restore (§14.5.3).
• Data engines (interim containers, §6.5): minor/patch versions ride the image rebuild cycle. Major versions (PG 16→17 class) are Change-Order events with dump/restore rehearsal in Dev, never a window item. Under Managed DBs (§7.2), DO handles patch/minor; majors remain scheduled decisions.
• BookStack: monthly window, dump first (tier-2), standard upstream upgrade path.
• DO platform events (hypervisor migrations, mandatory maintenance): outside our scheduling — mitigations are the tier-1 backups, the alert stack (droplet-down alerts already routed), and reading DO's status/email notices, which route to the SBS-controlled account alias (§2.1).

17.4 Upgrade Procedure (Standard Template)

Every window execution follows one runbook shape (wiki-published, per-layer specifics as annexes):

1. Pre: confirm monitoring green fleet-wide; confirm last backup age green (tier-1 and applicable tier-2, §14.3 metrics); announce in ops channel (start/scope/expected end); freeze deploys for the window.
2. Execute: Dev-proven changes only (except emergency path); one host at a time in §17.2 order; between hosts, verify: services active, blackbox probes green (§12.2), no new alerts for 10 min before proceeding.
3. Verify: per-layer smoke (nginx -t + probe sweep; Grafana health + one dashboard render; Vault unsealed + read probe; app surfaces via blackbox; audit writer queue age nominal).
4. Close: announce complete; changelog entry (narrative layer, §10.6) with versions before/after; any deviation → Build Register item within 24 h.

17.5 Rollback

Rollback authority: the window executor rolls back without seeking approval when any post-step verification fails — the approval was for the change; reverting to known-good needs none. Paths per layer, worst-case first:

Rollback is itself verified (same §17.4.3 smoke set) and logged; a rolled-back change re-enters through Dev with a written cause, never retried directly on Prod — the failure taught something or it didn't; the PR discussion says which.

17.6 Verification (Build Gates)

Gates (LH-SBS-INST-001): unattended-upgrades active fleet-wide with security-pocket-only config (Ansible-asserted, drift-checked); reboot-required metric exported and alerting at 14 d; window calendar published to wiki with §17.2 order and freeze rules; standard runbook + per-layer annexes published; one full monthly-window execution completed end-to-end on the fleet with changelog evidence; one deliberate rollback drill on Dev (failed-verification simulation → digest revert → smoke green); Vault pre-upgrade snapshot step present in its annex and rehearsed with test shares (§14.7 linkage); OS-mix remediation (item 21) closed or scheduled through these windows — the mechanism that fixes the existing drift is the mechanism that prevents the next.

Appendices

Chapter 18 — Appendices

Appendix A — Resource Inventory

A.1 DigitalOcean Account & Projects

A.2 Droplets (As Built)

A.3 Network (Target vs. As Built)

A.4 DNS Zone (sbsdash.com — GoDaddy NS, cutover pending item 9)

A.5 Storage & Registry (Planned)

A.6 Software Baseline (As Built)

Docker 29.1.3 / Compose 2.40.3 (prod, dev, monitor) · Grafana 13.1.0 (grafana apt repo) · Prometheus + node_exporter ×4 (Ubuntu repo) · nginx 1.28.3 · certbot + LE cert live on monitor · BookStack (wiki). Pending install: Node LTS/pnpm, Rust/Cargo, Python venvs, PostgreSQL, MongoDB, Redis, Vault, Loki, Alertmanager, blackbox/cAdvisor/engine exporters, oauth2-proxy, AnythingMCP.

---

Appendix B — Firewall Rule Tables (Target, §3.5–3.6)

B.1 Cloud Firewalls (tag-scoped) — as built: none exist ⚠ (item 19)

B.2 Host Firewall (nftables target; as built: ufw on monitor/wiki only, none on prod/dev ⚠ item 20)

Baseline: inbound default-deny mirroring Cloud FW; outbound default-deny + allowlist — 443→Anthropic API, GitHub/GHCR, apt mirrors, LE ACME; 123→NTP; 53→DO resolvers. Per-Compartment chains: egress = Anthropic + that Compartment's declared integrations only (§3.4). Ruleset hash exported to integrity job (§12.2).

B.3 Egress Allowlist Summary

---

Appendix C — Build Register (Consolidated, items 1–115)

Status: ✅ done · ◐ partial · ○ open · ⛔ blocked/decision · numbering per chapter of origin.

Critical path: 15 → 20 → 57/58 → 18(a) rebuild wave (4, 19, 42, wiki NYC2) → 60 (17) → Ch. 5 prod edge (28) → 16 apex repoint → 9/10 DNSSEC cutover. Blocking decisions: 18, 21, 22, 36, 49.

---

Appendix D — Acronym Table

---

Appendix E — References