# Chapter 8 — Secrets Management

##### **8.1 Current State and Scope**

As of 2026-07-18 **no secrets manager is deployed**. Credential state is the Chapter 2 CRITICAL nonconformance: plaintext passwords in the operational changelog, a shared fleet-wide root password, and service identities on non-SBS mailboxes (Build Register item 15). This chapter defines the target system that item 15 migrates into, covering: platform secrets (API tokens, DB credentials, Spaces keys, internal-CA material, webhook URLs), per-Compartment agent credentials (AI-03), and deploy-time injection into containers (§6.4). Out of scope: end-user passwords (IdP domain, Chapter 9) and TLS private keys for public surfaces (deliberately host-local, §5.2).

##### **8.2 Vault Deployment**

**HashiCorp Vault (OSS)** on a dedicated minimal droplet:

<div class="overflow-x-auto w-full px-2 mb-6" id="bkmrk-attribute-value-drop"><table class="min-w-full border-collapse text-sm leading-[1.7] whitespace-normal"><thead class="text-left"><tr><th class="text-text-100 border-b-0.5 border-[hsl(var(--border-300)/0.6)] py-2 pr-4 align-top font-bold" scope="col">Attribute</th><th class="text-text-100 border-b-0.5 border-[hsl(var(--border-300)/0.6)] py-2 pr-4 align-top font-bold" scope="col">Value</th></tr></thead><tbody><tr><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">Droplet</td><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">`prod-vault` — 1 GB / 1 vCPU, NYC2, member of `prod-vpc-nyc2` (created in the §3.1 rebuild wave)</td></tr><tr><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">Storage backend</td><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">Integrated Raft, single node, on a dedicated DO Volume</td></tr><tr><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">Listener</td><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">VPC-private interface **only**, TLS from the internal CA (§7.5) — no public interface listener at all; Vault has no DNS name in the public zone</td></tr><tr><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">Access paths</td><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">API from prod/dev droplets over VPC; human access via bastion (§3.3) with individually attributed tokens</td></tr><tr><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">Firewall</td><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">Cloud Firewall `prod-vault`: TCP 8200 from `sbsdash-server-prod`, `Dev-SBS-server` (for dev-namespace reads), and bastion private IPs only; SSH via bastion only</td></tr></tbody></table>

</div>Placement rationale: not on `sbsdash-server-prod` (the secrets store must survive and be trustworthy independent of the host running the workloads that consume it — a prod-droplet compromise should not equal immediate Vault-storage compromise), and not on `monitor-servers` (bastion + alerting authority already concentrates enough; adding secrets makes it a single point of total compromise). A 1 GB droplet is sufficient — Vault's footprint at this scale is trivial.

**Unseal model:** DigitalOcean offers no KMS, so auto-unseal via cloud KMS is unavailable. **Shamir key shares: 5 shares, threshold 3**, held by named individuals (Ledger Hub platform engineering ×2, Nelson Santos, Alexandra Del Rey, one SBS-designated alternate) — no individual can unseal alone; any 3 can recover. Shares are stored in each holder's personal password manager, never together, never in the changelog or repo. Unseal is required at Vault restart only; restarts are rare, scheduled events. Root token is revoked after initial setup (`vault operator generate-root` recreates one under quorum if ever needed) — routine administration uses named admin tokens under policy.

**Availability posture, stated honestly:** single-node Vault means Vault downtime blocks *deploys and credential rotation*, not running workloads (containers hold their injected credentials until restart, §8.5). Accepted at this fleet size; mitigation is Raft snapshot every 6 h to `sbs-dash-backups` (encrypted client-side per §7.5), restore-tested quarterly (Ch. 14). HA (3-node Raft) is a Chapter 16 growth trigger, not a Day-1 requirement.

##### **8.3 Namespace and Policy Layout**

Vault OSS lacks Enterprise namespaces; the equivalent is **mount-path + policy separation**, which is sufficient because policies are default-deny:

<div aria-label="Code" class="relative group/copy bg-bg-000/50 border-0.5 border-border-400 rounded-lg focus:outline-none focus-visible:ring-2 focus-visible:ring-accent-100" id="bkmrk-" role="group" tabindex="0"><div class="sticky opacity-0 group-hover/copy:opacity-100 group-focus-within/copy:opacity-100 top-2 py-2 h-12 w-0 float-right"><div class="absolute right-0 h-8 px-2 items-center inline-flex z-10">  
</div></div><div class="overflow-x-auto"></div></div>```
secret/platform/        # DO API tokens, GitHub deploy keys, Spaces keys,
                        # registrar/DNS API tokens, alerting webhooks
secret/infra/ca/        # internal CA key material (§7.5)
secret/db/platform/     # migration-tool superuser credentials (§7.4)
secret/c01/ … secret/c10/   # per-Compartment: DB role, Redis ACL user,
                            # Spaces prefix keys, Anthropic API key,
                            # per-Compartment integration credentials
secret/dev/…            # dev-environment mirror, synthetic/dev-scoped values only
```

Policies, one per consumer class: `policy-c01` grants read on `secret/c01/*` and **nothing else** — the AI-03 guarantee at the secrets layer, matching the network (§6.4) and engine-ACL (§7.4) walls: Compartment 03's runtime identity cannot read `secret/c08/*` even if every other control fails. `policy-deploy` reads what the deploy pipeline renders; `policy-admin` is held by named humans; no policy grants `secret/*` wildcard. Dev identities can read `secret/dev/*` only — prod paths are unreachable from Dev (the promotion path carries templates, not values, §2.2.3).

##### **8.4 Secret Lifecycle and Rotation Policy**

<div class="overflow-x-auto w-full px-2 mb-6" id="bkmrk-class-examples-rotat"><table class="min-w-full border-collapse text-sm leading-[1.7] whitespace-normal"><thead class="text-left"><tr><th class="text-text-100 border-b-0.5 border-[hsl(var(--border-300)/0.6)] py-2 pr-4 align-top font-bold" scope="col">Class</th><th class="text-text-100 border-b-0.5 border-[hsl(var(--border-300)/0.6)] py-2 pr-4 align-top font-bold" scope="col">Examples</th><th class="text-text-100 border-b-0.5 border-[hsl(var(--border-300)/0.6)] py-2 pr-4 align-top font-bold" scope="col">Rotation</th><th class="text-text-100 border-b-0.5 border-[hsl(var(--border-300)/0.6)] py-2 pr-4 align-top font-bold" scope="col">Mechanism</th></tr></thead><tbody><tr><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">Static, high-blast-radius</td><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">DO API token, GitHub deploy key, registrar/DNS API token, Spaces keys</td><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">90 days, and immediately on personnel change or suspected exposure</td><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">Manual rotation runbook (wiki), tracked as recurring Build Register/ops-calendar item; each rotation logged</td></tr><tr><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">Database credentials</td><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">Per-Compartment PostgreSQL / MongoDB / Redis users</td><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">30 days target</td><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">**Vault database secrets engine** (dynamic short-TTL credentials) once data services are up — Vault creates per-lease users, expiry is automatic, rotation ceases to be a human task. Until then: static creds at 90 days</td></tr><tr><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">Anthropic API keys</td><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">Per-Compartment keys</td><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">90 days; immediately on anomalous-usage alert (MON-02 / §3.4 egress signals)</td><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">Manual via Anthropic console, stored per-Compartment path</td></tr><tr><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">Internal CA leaf certs</td><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">DB TLS (§7.5)</td><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">90-day leaf TTL, auto-renewed by IaC</td><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">Vault PKI engine issues leaves; CA root offline in `secret/infra/ca/`</td></tr><tr><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">Unseal shares</td><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">—</td><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">On any holder change</td><td class="border-b-0.5 border-[hsl(var(--border-300)/0.3)] py-2 pr-4 align-top">Rekey ceremony (`vault operator rekey`, quorum)</td></tr></tbody></table>

</div>Rules: every secret has a named owner and a recorded rotation date (KV metadata); **KV v2 versioning** on all mounts so rotation keeps the prior version recoverable for rollback; no secret is ever emailed, chat-pasted, or written to the changelog — the changelog records *that* a rotation happened, never the value. Audit device (file → shipped to Ch. 12 pipeline) logs every read/write with the requesting identity: secret access is itself a MON-01 event stream, and reads outside deploy windows or from unexpected identities are MON-02 anomaly inputs.

##### **8.5 Application Integration**

Injection model (carried from §6.4), chosen for minimal moving parts:

1. **AppRole per consumer** — one AppRole per Compartment (`role-c01` … bound to `policy-c01`) and one for the deploy pipeline. RoleID is IaC-managed config; SecretID is delivered response-wrapped at deploy time with short TTL and use-limit 1 — a leaked wrapped token is worthless after first use or expiry.
2. **Deploy-time render** — the deploy step authenticates via AppRole, reads the Compartment's paths, renders the env-file consumed by `docker compose` (§6.7), sets mode 0600, and **deletes the env-file after container start** (compose has already captured the environment). Values never enter images, compose files, the repo, or CI logs (INF-02, INF-03).
3. **Refresh** — containers hold credentials until redeploy. With dynamic DB credentials (§8.4), lease TTLs are set ≥ the deploy cadence so normal deploys renew credentials as a side effect; an emergency revocation is: revoke lease in Vault → redeploy Compartment (minutes, and only the affected Compartment restarts — per-Compartment stacks pay off here).
4. **Prohibited patterns** — no Vault agent sidecars for now (added complexity before it's needed), no secrets in `docker inspect`-visible plain `environment:` keys in committed files, no shared "platform god token" consumed by all Compartments.

Bootstrap note (the classic first-secret problem): the deploy host must hold one credential to start — the RoleID (low sensitivity, useless without SecretID) on disk, and SecretID issuance gated by the CI/operator identity. This is the accepted trust anchor and is documented in the wiki runbook rather than hidden.

##### **8.6 Migration of Existing Credentials (Item 15 Execution Order)**

1. Deploy `prod-vault`, initialize, distribute shares, revoke root token.
2. **Rotate** the four exposed credential sets (GitHub, droplet root password → abolished entirely in favor of keys per §2.5, wiki admin, Grafana admin) — rotate *then* store; never vault a burned value.
3. Write rotated values + all new secrets created since (DO tokens, Spaces keys as provisioned) into their §8.3 paths.
4. Strip the changelog credentials section; purge from any git history (`git filter-repo`) if the file was ever committed; add pre-commit secret-scanning (gitleaks) to the repo (Ch. 10/11) so re-introduction is blocked mechanically, not procedurally.
5. Re-point service identities to SBS-controlled aliases (§4.4 note) as each is touched.

##### **8.7 Verification**

Gates (LH-SBS-INST-001): Vault reachable from prod over VPC TLS, unreachable from any public interface (external scan); policy probe — `role-c03` token reads `secret/c03/*`, denied on `secret/c08/*` and `secret/platform/*`; response-wrapped SecretID single-use confirmed (second unwrap fails); env-file absent post-deploy (filesystem check); no secret string present in repo, images, compose files, or CI logs (gitleaks sweep); audit device shipping to observability pipeline; Raft snapshot present in `sbs-dash-backups`, encrypted, restore-tested; changelog contains no credential values (item 15 closure evidence); rekey ceremony documented with current holder list.