Chapter 14 — Backup & Disaster Recovery 14.1 Current State and Scope As of 2026-07-18, tier 1 (DigitalOcean Backups) is enabled on the droplet fleet — the first backup layer in production. Tiers 2 and 3 remain unbuilt: no database dumps (no databases yet), no repo mirrors, no Vault snapshots (no Vault yet), no offline custody artifacts. Single-droplet loss is now recoverable; application-consistent recovery, region loss, and account compromise are not yet covered. Scope of this chapter: everything required to reconstruct the Platform (data, configuration, state, secrets, evidence); the recovery procedures per failure class; and the test cadence that keeps "we have backups" meaning "we can restore." Chapter 13's audit anchors are the integrity counterpart; this chapter is survivability. Controlling family: the backup/DR control set imported into KO-001. 14.2 Backup Architecture — Three Tiers The design principle: IaC-first recovery — most of the platform is rebuilt from code, not restored from images. Three complementary tiers, each covering a failure class the others cannot: Tier Mechanism State Covers Cannot cover 1 — Disk images DigitalOcean Backups (native) Live Fast single-droplet restore Application consistency (crash-consistent only); account compromise (backups are not downloadable and live inside the DO control plane) 2 — Application artifacts Encrypted dumps + state archives to Spaces, replicated NYC3→SFO3 Unbuilt Application-consistent data recovery; region loss Hostile DO account — see tier 3 3 — Off-account survivability Repo mirrors, offline age identity (SBS custody), Vault Shamir shares, audit anchors (Rekor + SBS email) Unbuilt Total account/custody compromise (§14.6) — (this is the floor) 14.2.1 Tier 1 — DigitalOcean Backups (Live) Enabled on the fleet as of 2026-07-18. Target configuration: Droplet Frequency (target) Retention (target) sbsdash-server-prod Daily 7 days prod-vault (at creation) Daily 7 days monitor-servers Daily 7 days sbs-wiki (carries through NYC2 rebuild) Daily 7 days Dev-SBS-server Weekly 4 weeks Open verification items against the live enablement (Build Register item 96 residual): Plan and settings audit — confirm which plan is active (usage-based, GA since Nov 2025 and priced by restorable file size with configurable frequency/retention, vs. legacy flat-percentage) and the actual per-droplet frequency selected (usage-based plans allow 4 h / 6 h / 12 h / daily / weekly). Align to the target table or amend it. Codify in tofu at the §10.6 import — enablement was performed via console; once imported, a droplet created without backups surfaces as a plan diff rather than a discovered gap. Console-enabled today is acceptable as bootstrap; console-managed permanently is drift. Dev-Gitlab-server — uninventoried (§2.2.2) but currently serving the apex record; take one backup/snapshot of it before the DNS repoint as cheap insurance, regardless of its final disposition. Ongoing tier-1 mechanics: monthly, one Prod backup per droplet is converted to a snapshot and retained on a 12-month rolling basis (backups convert to snapshots for indefinite storage — the medium-term image tier). Restore paths are in-place restore or new-droplet-from-backup; creation runs roughly 2 min/GB used, keeping backup windows in minutes at our footprint. Position in the design, stated plainly: tier 1 is the speed layer for single-droplet loss only. Disk images are crash-consistent — a mid-write PostgreSQL image can restore dirty, so databases recover from tier 2; and an image restores whatever existed at image time, drift and compromise included, so the IaC rebuild (§10) remains the clean-room path. 14.2.2 Tier 2 — Application Artifacts (What Code Cannot Regenerate) Asset Reconstruction source Backup artifact Droplets (OS + services) Rebuilt — tofu + Ansible (§10); tier-1 image as fast path — Infra + app repos, IaC state GitHub (off-platform) + tofu state in Spaces (§10.2) Nightly git bundle mirror of both repos to sbs-dash-backups — GitHub org loss is inside the threat model (§2.1 revocability cuts both ways) PostgreSQL — pg_dump per-database (per-Compartment, §7.4) nightly + WAL archiving continuous once the PITR trigger fires (§14.4) MongoDB — mongodump per-database nightly Redis Declared AOF keyspaces only (§7.1) AOF-covered; nightly RDB copy off-host. Cache loss accepted by design Vault — Raft snapshot every 6 h (§8.2) — the single most critical artifact; losing Vault + its snapshots = re-keying the platform Audit segments Already in sbs-dash-audit Cross-region replication NYC3→SFO3 (§13.4); anchors survive dual-bucket loss Grafana dashboards, alert rules, nginx, compose, nftables Repo (file-provisioned, §12.5/§10.3) Covered by repo mirror — this is why §12.5 bans console-authored dashboards TLS keys Not backed up — reissued via ACME (§5.2) Deliberate: reissuance ≤ minutes; escrow risk > loss cost BookStack (wiki) — Nightly DB dump + uploads archive (runbooks are operational capital) All dumps are encrypted client-side before upload (age; key in Vault, with the age identity also held offline in SBS custody per the §9.6 break-glass pattern — an encrypted backup whose only key lives in the Vault it must survive is a circular dependency, broken here) and land in sbs-dash-backups (NYC3) with lifecycle replication to SFO3. 14.3 Schedule and Retention Artifact Frequency Retention DO Backups (disk images) — live Daily Prod / weekly Dev (target; audit pending), DO-scheduled 7 d / 4 w native; 12 monthly via snapshot conversion DB dumps (PG, Mongo, BookStack) Nightly 03:00 ET 14 nightly, 8 weekly, 12 monthly (GFS) WAL archive (when enabled) Continuous 14 days Vault Raft snapshot 6-hourly 28 snapshots (7 d) + 12 monthly Repo mirrors Nightly 30 days Audit replication On segment close (§13.2) 7 y (§13.4) Redis RDB Nightly 7 days Every tier-2 job emits completion + size + age metrics to the integrity job (§12.2); tier-1 backup age is polled from the DO API per droplet and exported to the same job — now actionable immediately and prioritized, since it is the only automated proof the live backups are actually completing. Any backup age > 26 h (tier-1 daily set or tier-2 nightly set) is critical (§12.4) — a silently failing backup is the canonical DR failure and is alerted, not discovered. 14.4 RPO / RTO Targets Per failure class — one blended number would be fiction: Failure class RPO RTO Recovery path Single service/container failure 0 ≤ 15 min Compose restart / redeploy pinned digest (§6.7) Single droplet loss (non-data) 0 (stateless) ≤ 30 min (tier-1 image restore) / ≤ 2 h (IaC clean-room rebuild) Image restore when the failure is mechanical; IaC rebuild when state is suspect Prod app droplet loss (interim data services, §6.5) ≤ 24 h (nightly dumps); ≤ 5 min once WAL PITR enabled ≤ 4 h Image-restore or rebuild droplet, restore tier-2 dumps, re-assert ACLs (§7.4), redeploy Vault droplet loss ≤ 6 h ≤ 2 h New droplet (IaC or image) + Raft snapshot restore + quorum unseal (§8.2) — RTO includes assembling 3 shareholders; that human dependency is the honest bottleneck Monitor/bastion loss 0 config / 30 d metrics history accepted lost ≤ 4 h Image restore or IaC rebuild; interim SSH path: temporary direct-SSH firewall rule to prod via emergency PR (§10.4), removed at bastion restoration Wiki loss ≤ 24 h ≤ 4 h Image restore or IaC rebuild + dump restore Region loss (NYC2) ≤ 24 h (SFO3 replicas; tier-1 images are region-bound and do not participate) ≤ 3 business days Full IaC re-deploy to alternate region (region is a tofu variable, §10.3 globals); DNS repoint (§4.3); restore from SFO3. Not hot-standby — stated cost decision at this scale, revisited at Ch. 16 triggers Account compromise (DO or GitHub org) Bounded by tier-3 artifacts Days, incident-governed §14.6 — tier 1 and in-account tier 2 presumed lost Until tier 2 exists, the honest fleet-wide posture is: RPO for any future data = undefined; recoverability = single-droplet mechanical failure only. This line is deleted from the specification when items 86–92 close — it is written here so the gap is a stated fact, not an implication. Database RPO ≤ 24 h is the accepted Day-1 posture once dumps run; the WAL-archiving upgrade to ≤ 5 min is triggered when the first Compartment carries production tenant data — not before (cost honesty), mandatorily not after. 14.5 Restore Procedures Each is a wiki runbook (§1.4 publication target) with exact commands and a verification tail; summarized: Single-droplet fast path (tier 1 — available today): restore image or create-from-backup → verify drift ( tofu plan + Ansible --check clean against repo state once §10 exists; until then, manual diff against SERVER_CHANGELOG.md — the interim reality, stated) → rejoin monitoring → smoke test. Database restore (tier 2): provision clean engine (IaC) → decrypt dump (age key from Vault, or offline custody copy if Vault is the casualty) → restore per-database → re-assert per-Compartment roles/ACLs from IaC before any application start (a restored database with default grants is an isolation breach, §7.4) → row-count comparison vs. manifest → smoke test. Vault restore: IaC/image droplet → raft snapshot restore → quorum unseal → audit-device continuity check → rotate any credential issued after snapshot timestamp (derivable from the Vault audit stream). Full-region DR: run-order: VPCs → Vault (secrets first — everything authenticates through it) → data services + restores → app droplets + deploys → monitor → DNS cutover (TTLs drop to 300 at DR declaration) → §12.7-style verification sweep before declaring recovery. Restore-time isolation rule: Compartment agent workloads stopped during recovery — agents never run against partially restored data (Guardian policy state could lag data state); restart is gated on the verification tail and is itself an audit-chain event (§13.3). 14.6 Worst-Case: Account/Custody Compromise The survivability floor. If the DO account and GitHub org are both lost or hostile: tier 1 is gone by definition (backups are not downloadable and live inside the account); in-account tier 2 is presumed gone. Reconstruction requires exactly the tier-3 set — repo mirrors (SFO3 replicas), encrypted dumps + offline age identity (SBS custody), Vault snapshot + Shamir shares (5 named holders, §8.2), and audit anchors (Rekor + SBS email, §13.2). Each lives outside the compromised trust domain by design. The runbook is the incident-response DR annex (MON-05 linkage); the §14.7 annual tabletop rehearses this scenario specifically. 14.7 DR Test Cadence Untested backups are hypotheses. Each test produces a dated wiki report (MON-04/quarterly-review evidence): Test Cadence Pass criterion Tier-1 restore drill — first execution scheduled now (the layer is live; prove it): restore latest backup of a non-critical droplet (wiki or dev) to a new droplet, boot, inspect, destroy First: immediate. Then quarterly within the rotation below Boots, services start, data present as of backup time Automated restore-verify: latest PG/Mongo dump into throwaway Dev container, row counts + ACL assertions Weekly, automated (CI), from tier-2 build Green integrity metric; failure = critical Single-droplet recovery drill, alternating quarterly between tier-1 image restore and IaC clean-room rebuild Quarterly RTO ≤ target; post-restore drift check clean Vault snapshot restore into isolated instance (test shares, no prod unseal) Quarterly Unseal + read probe passes Full-region tabletop + partial technical (Vault+DB restore into SFO3) Annually Runbook gaps filed as Build Register items Account-compromise tabletop (§14.6) Annually Custody chain confirmed current (share holders, offline key, alias access) 14.8 Verification (Build Gates) Gates (LH-SBS-INST-001): DO Backups confirmed active on all five droplets with plan/frequency audit closed and settings codified in tofu (item 96 residual); tier-1 backup-age visible via DO API and exported to integrity per droplet; first tier-1 restore drill passed; every §14.2.2 artifact present in sbs-dash-backups with age metric green; SFO3 replication object-count parity; dump decrypts with the offline-custody key copy (circular-dependency break proven, not just the Vault copy); weekly restore-verify green ≥ 2 consecutive cycles pre-Cutover; Vault snapshot restore drill passed; backup-age alert fires in induced-failure drill; monthly snapshot-conversion automation present in IaC; restore runbooks on wiki with last-tested dates; WAL-PITR trigger condition documented; region variable proven by clean tofu plan targeting SFO3; Dev-Gitlab-server pre-repoint backup taken.