Chapter 14 — Backup & Disaster Recovery
14.1 Current State and Scope
As of 2026-07-18, tier 1 (DigitalOcean Backups) is enabled on the droplet fleet — the first backup layer in production. Tiers 2 and 3 remain unbuilt: no database dumps (no databases yet), no repo mirrors, no Vault snapshots (no Vault yet), no offline custody artifacts. Single-droplet loss is now recoverable; application-consistent recovery, region loss, and account compromise are not yet covered. Scope of this chapter: everything required to reconstruct the Platform (data, configuration, state, secrets, evidence); the recovery procedures per failure class; and the test cadence that keeps "we have backups" meaning "we can restore." Chapter 13's audit anchors are the integrity counterpart; this chapter is survivability. Controlling family: the backup/DR control set imported into KO-001.
14.2 Backup Architecture — Three Tiers
The design principle: IaC-first recovery — most of the platform is rebuilt from code, not restored from images. Three complementary tiers, each covering a failure class the others cannot:
| Tier | Mechanism | State | Covers | Cannot cover |
|---|---|---|---|---|
| 1 — Disk images | DigitalOcean Backups (native) | Live | Fast single-droplet restore | Application consistency (crash-consistent only); account compromise (backups are not downloadable and live inside the DO control plane) |
| 2 — Application artifacts | Encrypted dumps + state archives to Spaces, replicated NYC3→SFO3 | Unbuilt | Application-consistent data recovery; region loss | Hostile DO account — see tier 3 |
| 3 — Off-account survivability | Repo mirrors, offline age identity (SBS custody), Vault Shamir shares, audit anchors (Rekor + SBS email) | Unbuilt | Total account/custody compromise (§14.6) | — (this is the floor) |
14.2.1 Tier 1 — DigitalOcean Backups (Live)
Enabled on the fleet as of 2026-07-18. Target configuration:
| Droplet | Frequency (target) | Retention (target) |
|---|---|---|
sbsdash-server-prod |
Daily | 7 days |
prod-vault (at creation) |
Daily | 7 days |
monitor-servers |
Daily | 7 days |
sbs-wiki (carries through NYC2 rebuild) |
Daily | 7 days |
Dev-SBS-server |
Weekly | 4 weeks |
Open verification items against the live enablement (Build Register item 96 residual):
- Plan and settings audit — confirm which plan is active (usage-based, GA since Nov 2025 and priced by restorable file size with configurable frequency/retention, vs. legacy flat-percentage) and the actual per-droplet frequency selected (usage-based plans allow 4 h / 6 h / 12 h / daily / weekly). Align to the target table or amend it.
- Codify in tofu at the §10.6 import — enablement was performed via console; once imported, a droplet created without backups surfaces as a plan diff rather than a discovered gap. Console-enabled today is acceptable as bootstrap; console-managed permanently is drift.
Dev-Gitlab-server— uninventoried (§2.2.2) but currently serving the apex record; take one backup/snapshot of it before the DNS repoint as cheap insurance, regardless of its final disposition.
Ongoing tier-1 mechanics: monthly, one Prod backup per droplet is converted to a snapshot and retained on a 12-month rolling basis (backups convert to snapshots for indefinite storage — the medium-term image tier). Restore paths are in-place restore or new-droplet-from-backup; creation runs roughly 2 min/GB used, keeping backup windows in minutes at our footprint.
Position in the design, stated plainly: tier 1 is the speed layer for single-droplet loss only. Disk images are crash-consistent — a mid-write PostgreSQL image can restore dirty, so databases recover from tier 2; and an image restores whatever existed at image time, drift and compromise included, so the IaC rebuild (§10) remains the clean-room path.
14.2.2 Tier 2 — Application Artifacts (What Code Cannot Regenerate)
| Asset | Reconstruction source | Backup artifact |
|---|---|---|
| Droplets (OS + services) | Rebuilt — tofu + Ansible (§10); tier-1 image as fast path | — |
| Infra + app repos, IaC state | GitHub (off-platform) + tofu state in Spaces (§10.2) | Nightly git bundle mirror of both repos to sbs-dash-backups — GitHub org loss is inside the threat model (§2.1 revocability cuts both ways) |
| PostgreSQL | — | pg_dump per-database (per-Compartment, §7.4) nightly + WAL archiving continuous once the PITR trigger fires (§14.4) |
| MongoDB | — | mongodump per-database nightly |
| Redis | Declared AOF keyspaces only (§7.1) | AOF-covered; nightly RDB copy off-host. Cache loss accepted by design |
| Vault | — | Raft snapshot every 6 h (§8.2) — the single most critical artifact; losing Vault + its snapshots = re-keying the platform |
| Audit segments | Already in sbs-dash-audit |
Cross-region replication NYC3→SFO3 (§13.4); anchors survive dual-bucket loss |
| Grafana dashboards, alert rules, nginx, compose, nftables | Repo (file-provisioned, §12.5/§10.3) | Covered by repo mirror — this is why §12.5 bans console-authored dashboards |
| TLS keys | Not backed up — reissued via ACME (§5.2) | Deliberate: reissuance ≤ minutes; escrow risk > loss cost |
| BookStack (wiki) | — | Nightly DB dump + uploads archive (runbooks are operational capital) |
All dumps are encrypted client-side before upload (age; key in Vault, with the age identity also held offline in SBS custody per the §9.6 break-glass pattern — an encrypted backup whose only key lives in the Vault it must survive is a circular dependency, broken here) and land in sbs-dash-backups (NYC3) with lifecycle replication to SFO3.
14.3 Schedule and Retention
| Artifact | Frequency | Retention |
|---|---|---|
| DO Backups (disk images) — live | Daily Prod / weekly Dev (target; audit pending), DO-scheduled | 7 d / 4 w native; 12 monthly via snapshot conversion |
| DB dumps (PG, Mongo, BookStack) | Nightly 03:00 ET | 14 nightly, 8 weekly, 12 monthly (GFS) |
| WAL archive (when enabled) | Continuous | 14 days |
| Vault Raft snapshot | 6-hourly | 28 snapshots (7 d) + 12 monthly |
| Repo mirrors | Nightly | 30 days |
| Audit replication | On segment close (§13.2) | 7 y (§13.4) |
| Redis RDB | Nightly | 7 days |
Every tier-2 job emits completion + size + age metrics to the integrity job (§12.2); tier-1 backup age is polled from the DO API per droplet and exported to the same job — now actionable immediately and prioritized, since it is the only automated proof the live backups are actually completing. Any backup age > 26 h (tier-1 daily set or tier-2 nightly set) is critical (§12.4) — a silently failing backup is the canonical DR failure and is alerted, not discovered.
14.4 RPO / RTO Targets
Per failure class — one blended number would be fiction:
| Failure class | RPO | RTO | Recovery path |
|---|---|---|---|
| Single service/container failure | 0 | ≤ 15 min | Compose restart / redeploy pinned digest (§6.7) |
| Single droplet loss (non-data) | 0 (stateless) | ≤ 30 min (tier-1 image restore) / ≤ 2 h (IaC clean-room rebuild) | Image restore when the failure is mechanical; IaC rebuild when state is suspect |
| Prod app droplet loss (interim data services, §6.5) | ≤ 24 h (nightly dumps); ≤ 5 min once WAL PITR enabled | ≤ 4 h | Image-restore or rebuild droplet, restore tier-2 dumps, re-assert ACLs (§7.4), redeploy |
| Vault droplet loss | ≤ 6 h | ≤ 2 h | New droplet (IaC or image) + Raft snapshot restore + quorum unseal (§8.2) — RTO includes assembling 3 shareholders; that human dependency is the honest bottleneck |
| Monitor/bastion loss | 0 config / 30 d metrics history accepted lost | ≤ 4 h | Image restore or IaC rebuild; interim SSH path: temporary direct-SSH firewall rule to prod via emergency PR (§10.4), removed at bastion restoration |
| Wiki loss | ≤ 24 h | ≤ 4 h | Image restore or IaC rebuild + dump restore |
| Region loss (NYC2) | ≤ 24 h (SFO3 replicas; tier-1 images are region-bound and do not participate) | ≤ 3 business days | Full IaC re-deploy to alternate region (region is a tofu variable, §10.3 globals); DNS repoint (§4.3); restore from SFO3. Not hot-standby — stated cost decision at this scale, revisited at Ch. 16 triggers |
| Account compromise (DO or GitHub org) | Bounded by tier-3 artifacts | Days, incident-governed | §14.6 — tier 1 and in-account tier 2 presumed lost |
Until tier 2 exists, the honest fleet-wide posture is: RPO for any future data = undefined; recoverability = single-droplet mechanical failure only. This line is deleted from the specification when items 86–92 close — it is written here so the gap is a stated fact, not an implication. Database RPO ≤ 24 h is the accepted Day-1 posture once dumps run; the WAL-archiving upgrade to ≤ 5 min is triggered when the first Compartment carries production tenant data — not before (cost honesty), mandatorily not after.
14.5 Restore Procedures
Each is a wiki runbook (§1.4 publication target) with exact commands and a verification tail; summarized:
- Single-droplet fast path (tier 1 — available today): restore image or create-from-backup → verify drift (
tofu plan+ Ansible--checkclean against repo state once §10 exists; until then, manual diff against SERVER_CHANGELOG.md — the interim reality, stated) → rejoin monitoring → smoke test. - Database restore (tier 2): provision clean engine (IaC) → decrypt dump (age key from Vault, or offline custody copy if Vault is the casualty) → restore per-database → re-assert per-Compartment roles/ACLs from IaC before any application start (a restored database with default grants is an isolation breach, §7.4) → row-count comparison vs. manifest → smoke test.
- Vault restore: IaC/image droplet →
raft snapshot restore→ quorum unseal → audit-device continuity check → rotate any credential issued after snapshot timestamp (derivable from the Vault audit stream). - Full-region DR: run-order: VPCs → Vault (secrets first — everything authenticates through it) → data services + restores → app droplets + deploys → monitor → DNS cutover (TTLs drop to 300 at DR declaration) → §12.7-style verification sweep before declaring recovery.
- Restore-time isolation rule: Compartment agent workloads stopped during recovery — agents never run against partially restored data (Guardian policy state could lag data state); restart is gated on the verification tail and is itself an audit-chain event (§13.3).
14.6 Worst-Case: Account/Custody Compromise
The survivability floor. If the DO account and GitHub org are both lost or hostile: tier 1 is gone by definition (backups are not downloadable and live inside the account); in-account tier 2 is presumed gone. Reconstruction requires exactly the tier-3 set — repo mirrors (SFO3 replicas), encrypted dumps + offline age identity (SBS custody), Vault snapshot + Shamir shares (5 named holders, §8.2), and audit anchors (Rekor + SBS email, §13.2). Each lives outside the compromised trust domain by design. The runbook is the incident-response DR annex (MON-05 linkage); the §14.7 annual tabletop rehearses this scenario specifically.
14.7 DR Test Cadence
Untested backups are hypotheses. Each test produces a dated wiki report (MON-04/quarterly-review evidence):
| Test | Cadence | Pass criterion |
|---|---|---|
| Tier-1 restore drill — first execution scheduled now (the layer is live; prove it): restore latest backup of a non-critical droplet (wiki or dev) to a new droplet, boot, inspect, destroy | First: immediate. Then quarterly within the rotation below | Boots, services start, data present as of backup time |
| Automated restore-verify: latest PG/Mongo dump into throwaway Dev container, row counts + ACL assertions | Weekly, automated (CI), from tier-2 build | Green integrity metric; failure = critical |
| Single-droplet recovery drill, alternating quarterly between tier-1 image restore and IaC clean-room rebuild | Quarterly | RTO ≤ target; post-restore drift check clean |
| Vault snapshot restore into isolated instance (test shares, no prod unseal) | Quarterly | Unseal + read probe passes |
| Full-region tabletop + partial technical (Vault+DB restore into SFO3) | Annually | Runbook gaps filed as Build Register items |
| Account-compromise tabletop (§14.6) | Annually | Custody chain confirmed current (share holders, offline key, alias access) |
14.8 Verification (Build Gates)
Gates (LH-SBS-INST-001): DO Backups confirmed active on all five droplets with plan/frequency audit closed and settings codified in tofu (item 96 residual); tier-1 backup-age visible via DO API and exported to integrity per droplet; first tier-1 restore drill passed; every §14.2.2 artifact present in sbs-dash-backups with age metric green; SFO3 replication object-count parity; dump decrypts with the offline-custody key copy (circular-dependency break proven, not just the Vault copy); weekly restore-verify green ≥ 2 consecutive cycles pre-Cutover; Vault snapshot restore drill passed; backup-age alert fires in induced-failure drill; monthly snapshot-conversion automation present in IaC; restore runbooks on wiki with last-tested dates; WAL-PITR trigger condition documented; region variable proven by clean tofu plan targeting SFO3; Dev-Gitlab-server pre-repoint backup taken.