[vibe](../../README.md) > [Guidebooks](../README.md) > [Lab ecosystem](README.md) > **Secrets & Vault** # Secrets & Vault > **Status**: ๐ŸŸข Active > **Last Updated**: 2026-06-23 > **Related**: [Lab ecosystem](README.md) ยท [Tools brick](02-tools.md) ยท [Storage & recovery](storage-and-recovery.md) ยท [Naming conventions](naming-conventions.md) > **Decision**: [ADR 0001 โ€” Safe, production-like environment](../../ADR/0001-safe-prod-like-environment.md) ## TL;DR **HashiCorp Vault is the single source of truth for every secret in the lab.** There is no sops, no age, no secret files in git โ€” if a credential exists, Vault either stores it or mints it on demand. Two parties consume secrets, and each authenticates a different way: **pods** use the Kubernetes auth backend (via the Vault Secrets Operator), and **CI / OpenTofu** use Gitea OIDC JWT (one role `gitea_cicd_` per app). Vault holds static config in KV, encryption keys in transit, and issues **short-lived, dynamic** PostgreSQL credentials so no long-lived DB password is ever written down. The trade-off: Vault is sealed on every restart and must be **manually unsealed** (1 key, threshold 1) before anything that needs a secret can come back. ## Why Vault, and only Vault The lab made a deliberate choice: **one** secret store, accessed over the network, rather than encrypted secret files scattered through the repos. The consequences are structuring: - **No secret material in git.** Charts and OpenTofu reference Vault *paths*, never values. A leaked repo leaks no credentials. - **One revocation point.** Rotating or revoking a credential happens in Vault; consumers pick up the change on their next read or lease renewal. - **Dynamic over static.** Where a backend supports it (Postgres), Vault issues a fresh, time-boxed credential per consumer instead of a shared static password. Vault itself runs as the `hashicorp-vault` chart in the **tools** namespace. Its full configuration โ€” engines, auth backends, policies, the per-app role/policy modules โ€” lives in the tools repo; see the [Tools brick](02-tools.md) for the deployment context. ## What Vault mounts | Mount | Type | Purpose | | --- | --- | --- | | `kvv2/` | KV v2 (versioned) | Application static config, e.g. `kvv2//config`. Versioned so a bad write can be rolled back. | | KV v1 | KV v1 (unversioned) | Flat secrets that don't need history. | | `transit/` | Transit | Encryption-as-a-service: encrypt/decrypt and sign without exposing the key. | | `postgres/` | Database (dynamic) | Issues **short-lived** PostgreSQL credentials on demand: `postgres/creds/` hands out a fresh login user, granted `_role`, with a lease that expires. | The `` slug threads through every one of these paths โ€” `kvv2//config`, `postgres/creds/` โ€” exactly as described in [Naming conventions](naming-conventions.md). ## The two auth backends Vault doesn't trust callers by static token. Each class of consumer proves its identity through a backend matched to where it runs: - **Kubernetes auth** โ€” for **pods**. The Vault Secrets Operator (VSO) and workloads present their Kubernetes ServiceAccount token; Vault validates it against the cluster's API and maps the SA to the Vault role ``, which carries the runtime policy ``. - **Gitea OIDC / JWT auth** โ€” for **CI and OpenTofu**. A Gitea Actions workflow obtains an OIDC token; Vault validates it and maps it to the JWT role `gitea_cicd_`, which carries the CI/ops policy `-ops`. This is how `tofu apply` in CI reads and writes the secrets it manages without any pre-shared Vault token. The split matters: pods get only what they need at runtime (the `` policy), while CI gets the broader provisioning rights (`-ops`) needed to *create* the very secrets the pods will later read. ## How VSO delivers secrets to pods Inside the cluster, the **Vault Secrets Operator** is the bridge between Vault and Kubernetes. It watches two CRDs: - **`VaultAuth`** โ€” declares *how* to authenticate to Vault (the Kubernetes auth mount + the `` role). - **`VaultDynamicSecret`** (and `VaultStaticSecret`) โ€” declares *what* to fetch (e.g. `postgres/creds/`) and which Kubernetes Secret to materialise it into. For dynamic secrets, VSO also **renews the lease** and rotates the Secret before it expires. The pod then mounts the resulting Kubernetes Secret as it would any other โ€” it never speaks to Vault directly, and never sees a static DB password. ## The secret flow, end to end ```mermaid %%{init: {'theme':'base'}}%% flowchart LR subgraph CI["CI / Provisioning path"] GHA["Gitea Actions
workflow"]:::src TOFU["OpenTofu
tofu apply"]:::proc end subgraph RT["Runtime path (in-cluster)"] VSO["Vault Secrets
Operator (VSO)"]:::proc POD["App pod
(ServiceAccount <app>)"]:::proc end VAULT["Vault
KV v1/v2 ยท transit ยท postgres dynamic"]:::store GHA -->|"OIDC JWT
role gitea_cicd_<app>"| VAULT VAULT -->|"policy <app>-ops
read/write secrets"| TOFU TOFU -->|"writes config to
kvv2/<app>/config"| VAULT VSO -->|"k8s auth
role <app> (SA token)"| VAULT VAULT -->|"dynamic creds
postgres/creds/<app>"| VSO VSO -->|"materialises +
renews K8s Secret"| POD classDef src fill:#2563eb,stroke:#1e40af,color:#fff classDef proc fill:#059669,stroke:#047857,color:#fff classDef store fill:#7c3aed,stroke:#6d28d9,color:#fff ``` 1. **CI path:** a Gitea Actions workflow requests an OIDC JWT and presents it to Vault under the role `gitea_cicd_`. Vault validates the token and grants the `-ops` policy. 2. With that policy, OpenTofu (`tofu apply`, running in CI) reads the secrets it needs and writes the app's static config back to `kvv2//config`. No pre-shared Vault token is ever stored โ€” the trust is established per-run via OIDC. 3. **Runtime path:** in the cluster, the Vault Secrets Operator authenticates with the Kubernetes auth backend, presenting the app's ServiceAccount token mapped to the Vault role ``. 4. Vault issues a **short-lived, dynamic** PostgreSQL credential from `postgres/creds/` back to VSO. 5. VSO materialises that credential into a Kubernetes Secret in the app's namespace, then **renews the lease** and rotates the Secret before it expires. 6. The app pod mounts the Kubernetes Secret like any other โ€” it never talks to Vault, and never holds a long-lived database password. ## The unseal model Vault encrypts its storage with a master key that is **never persisted in usable form**. On every start โ€” a fresh deploy, a pod reschedule, or a full cluster recovery โ€” Vault comes up **sealed** and refuses every request until it is unsealed. - **Shamir config:** 1 unseal key, threshold 1 (a single-operator lab, so no key-splitting ceremony). - **Where the key lives:** on the control node (the MacBook), at `~/.arcodange/cluster-keys.json`. It is *not* in git, *not* in Kubernetes, *not* in Vault. - **Operational consequence:** **nothing that needs a secret recovers until a human unseals Vault.** This is the chokepoint baked into the recovery order โ€” VSO cannot re-auth, dynamic DB creds cannot be issued, and dependent apps cannot start, until the unseal happens. See [Storage & recovery](storage-and-recovery.md) for where unseal sits in the tested startup sequence. > [!CAUTION] > If `~/.arcodange/cluster-keys.json` is lost, Vault's data is **unrecoverable** โ€” there is no second copy of the unseal key and no key-recovery path. Treat that file as the most critical secret in the lab. ## Sandbox implications A production-like sandbox does **not** share the production Vault. It runs its **own** Vault instance with its **own** unseal key and its **own** policies, so that exercising secret flows, rotating credentials, or testing a broken unseal cannot touch production secrets. Because the `` join key is environment-relative (see [Naming conventions](naming-conventions.md)), the sandbox can keep identical role and policy names โ€” `gitea_cicd_`, ``, `-ops` โ€” while remaining fully isolated. The rationale for that separate-Vault, separate-unseal posture is recorded in [ADR 0001 โ€” Safe, production-like environment](../../ADR/0001-safe-prod-like-environment.md). ## See also - [Tools brick](02-tools.md) โ€” where the `hashicorp-vault` chart, VSO, and the per-app Vault IaC modules are deployed. - [Storage & recovery](storage-and-recovery.md) โ€” Vault unseal as a step in the tested power-cut recovery order. - [Naming conventions](naming-conventions.md) โ€” how `gitea_cicd_`, ``, and `-ops` derive from the join key. - [ADR 0001 โ€” Safe, production-like environment](../../ADR/0001-safe-prod-like-environment.md) โ€” the sandbox's separate-Vault decision.