Docs/Operate

#Secrets: envelope encryption design

Status: proposed, 2026-09-06 · Issue #37 (epic #24) · Binds ADR 0008. The cipher implementation is chosen by the #38 spike in ADR 0102. The storage schema and the Secrets API are #39; KMS providers are #40. Nothing here ships code.

#1. What is already decided

ADR 0008 fixes three things this document builds on and cannot revisit:

1. One database per tenant. The tenant's secrets, its migration history and its audit log live inside the tenant's own database, so a tenant backs up, restores, moves and drops as one unit.
2. A separate control database, with credentials no venture role holds, keeps the tenant registry and the global secrets store.
3. Two-tier secrets. Global secrets (tenant connection strings, platform keys) in the control database; tenant secrets in the tenant's database. Modules can only ever obtain a tenant store; the global store is unreachable from module code — by API and by credentials, so the separation is physical, not a rule in our own code.

This applies to the native runtime (phase 3). The Cloudflare Worker path is unchanged: one Worker, one D1, secrets arrive as wrangler environment bindings exactly as today (section 9).

#2. Key hierarchy

• KEK: one master key per environment (dev, staging, prod), held by the KMS, never exportable. Every store's DEK in that environment is wrapped by the environment's KEK. A KEK never leaves the KMS; the KMS performs wrap and unwrap operations on it.
• DEK: one 256-bit data key per store, generated locally from the OS RNG, wrapped by the KMS at creation, stored wrapped inside the database that the DEK protects. The global store has one; every tenant store has its own. Key ids (key_id) are ULIDs.
• Per-store keys mean: offboarding a tenant is a crypto-shred — delete the wrapped DEK row (or the whole database) and every ciphertext of that tenant is permanently undecryptable; a leaked tenant DEK exposes one tenant and no global secret; a tenant's backup is self-contained, and a tenant moved to another cluster takes its keys with its data.
• Rotation is cheap at both levels. Rotating the KEK re-wraps each DEK (one KMS call per store; no secret is re-encrypted). Rotating a DEK creates a new key_id, re-encrypts that store's secret versions under it, and retires the old row after a grace period.

#3. The database, not the KMS, holds the wrapped keys

Stated explicitly because it is the counter-intuitive half of envelope encryption: each database stores the wrapped DEK for the store inside it (a small key table: key_id, wrapped_dek, kek_id, cipher, created_at), and the KMS holds no per-store state at all.

This is safe because the wrapped DEK is useless without the KMS: it is an AEAD ciphertext under the KEK, and the KEK is non-exportable. A full database dump therefore yields ciphertexts plus a blob that cannot be unwrapped anywhere but the KMS. It is also what makes the properties in §2 hold — a backup is self-contained, a tenant move carries its keys, an offboarding shred is a local delete. If instead the KMS held the wrapped keys, a database dump would still be useless, but offboarding would require an out-of-band KMS operation, backups would need key export coordination, and the KMS would become a per-tenant state store.

#4. Cipher and nonce policy

The AEAD is XChaCha20-Poly1305, chosen by the #38 spike (ADR 0102): it is the only candidate that builds for wasm32-unknown-unknown, keeps keys zeroised, exposes AAD cleanly, adds ~46 KiB of binary, and needs no C toolchain. The alternative profile (AES-256-GCM, a FIPS-validated backend if a contractual FIPS requirement ever lands) is a backend swap at the port seam, not a redesign.

Nonce: 24 random bytes from the OS RNG per encryption (per secret version). No counters, no persistence, safe across processes: with a 192-bit nonce the birthday collision probability under one DEK is ≈ 2⁻¹²⁹ at 2³² encryptions — and a store performs encryptions only when secrets are written or rotated, so the real count is tiny.

The cipher is pinned per key id. The DEK row records the cipher it seals with. A backend change (the FIPS path above) provisions a new DEK version under the new cipher and re-encrypts on rotation; a store never mixes ciphers within one key_id. If AES-256-GCM is ever pinned for a store, its nonce policy is 12 random bytes with a hard ceiling of 2²⁰ encryptions per DEK (rotating well before the 96-bit birthday bound matters under a per-store key).

#5. What binds a ciphertext to its context

Every ciphertext is sealed with additional authenticated data derived from the row's identity — the detail home-grown schemes skip. The AAD is:

FZ-SECRETS-AAD-v1
  || len_le(store_id)   || store_id       // "global" | tenant ULID
  || len_le(secret_name) || secret_name   // "resend/api_key", …
  || len_le(u32)         || version_le    // secret version, monotonically increasing
  || len_le(key_id)      || key_id        // DEK that sealed this version

Length prefixes make the encoding injective — no two distinct contexts share an AAD whatever the character sets. The same bytes are recomputed on decrypt from the row the caller read. The spike proves each failure mode against every candidate (spikes/crypto/tests/roundtrip.rs):

A row that survives being moved, renamed or repointed is not a valid ciphertext anymore. Renaming a secret means writing a new version under the new name, not editing the row.

#The port

The trait is cratefield_kms::Kms (issue #40): provider(), key_ref(), wrap(), unwrap(), and nothing else, because wrapping is the only thing the KMS does for us. Errors split unavailable (retry) from denied (a human has to change something) from tampered (never retry, never fall back), since collapsing them makes a permissions mistake look like an outage.

LocalFileKms is the development provider and refuses to construct when the environment is production: its master key sits on a local disk, which is the property a KMS exists to remove. cratefield_kms:: conformance is the suite every future provider passes — round trip, a fresh nonce per wrap, every single-bit change to the wrapped blob refused, and a truncated or empty blob refused rather than panicking.

Managed vendor providers are not written yet: they need credentials and a nightly job against the real service, and an unexercised vendor integration here would be worse than an absent one. One production provider exists (WorkerSecretKms, ADR 0103): its KEK is a versioned set of Worker secrets, so it needs no vendor account — but the wiring that puts a KMS on the Worker path is a follow-up of #535, and §9's "Worker path unchanged" stands until it lands.

#The store

cratefield-secrets (issue #39) holds the two tables in every store — harness_secret_keys and harness_secrets — and the API modules call. Secrets::tenant() is what a module gets; Secrets::global() takes a token type only the harness can construct, so the control database's store is unreachable from module code however public the method looks.

SecretBytes zeroises on drop, prints as [redacted], and implements neither Display, Serialize nor Clone. Every method takes an Actor and records an AuditEvent before returning, on failure as well as success; the event type carries no value. Audit is the seam the append-only log (#41) fills, and until then accesses go to tracing.

One thing the schema settles that this document left implicit: the foreign key from a secret to its key row means a ciphertext cannot be orphaned. A key row cannot be deleted while secrets reference it, so the offboarding shred is dropping the tenant's database — which is what offboarding is under ADR 0008 anyway — rather than removing one row.

#The audit chain

harness_secret_audit, one per store, in the store's own database (issue #41). Every Secrets method writes exactly one row before returning — successes and refusals alike — and the store refuses the access when the sink cannot record it, because an unrecorded read is what the log exists to make impossible.

Three separate mechanisms, easily conflated:

Volume. A row is ~200 bytes with its two hashes. A venture that reads one secret per request would be unusable, so reads are per process start and per rotation, not per request: today's modules resolve a mailer key and a captcha secret at boot. At six tenants that is tens of rows a day per store, thousands a year — nothing. The number to watch is reads per tenant per day; partitioning is worth considering past roughly a million rows in one store's table, which at anything like the current shape is years away. Re-measure before assuming it.

#6. Operations

DEK lifecycle. Provision: generate 256 bits from the OS RNG, wrap under the KEK, store the wrapped blob in the store's database, warm the cache. Rotate: new key_id, re-encrypt secret versions, retire the old row once nothing references it — implemented as SecretStore::rotate_dek, with the runbook in KEY-ROTATION.md. Shred (offboarding): delete the wrapped DEK row with the database or independently — the ciphertexts become permanent noise. Crypto-shred is irreversible; a tenant returning starts with fresh keys.

Caching — designed, not built. The design below is what this section described in the present tense while SecretStore had no cache at all. It is written as a plan now, because a security document that describes an unbuilt control is worse than one that omits it: a reader planning capacity, or reading the threat model below, would believe a mitigation is in place.

What happens today. SecretStore::active_key and SecretStore::key call Kms::unwrap on every access. There is no map, no TTL and no eviction, so the KMS is called once per secret read rather than once per store per process per TTL — the calls scale with request volume, and the cost estimate that used to sit here was computed from a TTL that does not exist. Plaintext key material is wiped: Dek is a Zeroizing<Vec<u8>>, so a key is zeroised when it is dropped, which is at the end of the operation that unwrapped it.

What is planned. Unwrapped DEKs in process memory in a small map keyed by (store_id, key_id) with a TTL of 5 minutes (configurable), entries zeroised on eviction as well as on drop, and the KMS called on cache miss only — once per store per process per TTL at worst, because a hit refreshes nothing: the entry simply expires. Indicative AWS KMS pricing for that shape: a Decrypt call is ~$0.03 per 10,000; at 25 stores across 4 processes with a 300 s TTL the worst case is ~29k calls/day ≈ $0.09/day. The TTL exists to bound plaintext-key exposure, not to protect the KMS bill — which is also why the absence of a cache is not a security regression, only a cost and a latency one.

KMS unreachable. No process can decrypt anything until the KMS answers — no unwrap, no plaintext, no fallback. With the cache above built, a warm process would keep serving from it; without it, every read needs the KMS, so the dependency is total rather than cold-start-only. This is an accepted availability dependency on the KMS for every store, and it is alarmed, not merely documented:

• alarm on KMS error rate (unwrap failures) above threshold for 5 minutes;
• alarm on cold-process decrypt failures (KMS-unreachable errors reaching request handling) above zero sustained;
• degrade loudly: requests that need a secret fail with a distinct problem slug (secrets/kms-unavailable), never with a fallback to storing or sending plaintext. Warm caches make the steady-state blast radius of a short KMS outage approximately zero; only restarts during the outage hurt, which the second alarm catches.

The harness reads the global store only to resolve infrastructure — connecting a tenant database, provisioning a module — never on behalf of a tenant request, and never from module code (ADR 0008's reachability rule).

#7. Threat model

#8. Every secret in use today, assigned to a tier

On the Worker path today every value below arrives as a wrangler environment binding (or D1 binding). The tier column is where the value lives once the stores ship (#39) on the native runtime.

Assignment rule, so no future secret needs a design review: if the secret exists because Factory Zero operates infrastructure, it is global; if it exists because a venture runs its business, it is tenant. Factory Zero's own venture is an ordinary tenant (ADR 0008).

#9. The Worker path is unchanged

ADR 0008 scopes two-tier secrets to the native runtime. On Cloudflare, one venture = one Worker: secrets stay wrangler bindings, D1 stays the venture's single database, and nothing in this document adds a KMS dependency to the Worker deploy. The value of wasm-cleanliness for the crypto layer (ADR 0102) is that the primitive can be shared by both paths — not that the Worker path adopts the stores.

#10. Out of scope

Storage schema and the Secrets API (#39). KMS provider crates and the port (#40). Key ceremonies. Secret-versioning UIs. Benchmarking (the spike sanity-checked performance only).

#11. Open questions

Mirrored from docs/SECRETS-DESIGN.md at b6f50cd. The harness is the source of truth: a correction goes there first.