Use OpenBao as the secret authority

Context

The monorepo is the production source. Runtime secrets need one authority outside Git. The current OpenBao deployment uses one Raft member and a static GCP service-account JSON key for Cloud KMS auto-unseal.

SOPS currently protects secrets in Git. Retaining both SOPS and OpenBao as runtime authorities leaves two workflows and an unclear source of truth.

Decision

Use OpenBao as the authority for managed secrets. Applications receive secrets through External Secrets. After migration, do not store secret values in Git.

Run three OpenBao Raft members on three nodes on different Proxmox hosts. Use one strict-local Longhorn replica per member and required pod anti-affinity. OpenBao Raft, not Longhorn, owns replication for this data.

Use GCP Cloud KMS for auto-unseal with a static GCP service-account key. The service account can only use the unseal KMS key. Keep the key in Bitwarden, with a copy in Bitwarden cloud, and update both when the key rotates. The key does not depend on the cluster identity, so a rebuilt cluster can unseal OpenBao without a change in GCP.

Keep older KMS key versions enabled while live data or retained snapshots need them. The key rotates on a schedule (terraform/modules/openbao_gcp/main.tf:19), and each rotation adds a version.

The Terraform foundation root does not read OpenBao. It reads its credentials from Bitwarden, because it must work while OpenBao is down (ADR 002).

In production, External Secrets authenticates with OpenBao's kubernetes auth method using local TokenReview. OpenBao uses its pod's mounted reviewer token and CA, with matching RBAC, and pins no issuer, token, or CA. That way a rebuilt cluster with a new identity still works.

Staging reads its secrets from production OpenBao through JWT auth. This is the permanent design. Staging gets no OpenBao of its own for applications. The OpenBao on staging exists only for restore rehearsals. OpenBao runs outside staging and checks staging's tokens through JWKS, so it needs no TokenReview access to staging's API server.

OpenBao gets staging's signing keys from its JWKS endpoint, through jwks_url or oidc_discovery_url, whichever OpenBao can reach. It does not pin terraform/files/talos-staging-service-account.pub. Staging's API server must allow anonymous reads of the discovery endpoints (system:service-account-issuer-discovery), and OpenBao must reach it. Rename the mount from jwt-staging-grafana to jwt-staging, and migrate its roles and the SecretStore paths, before staging moves to a VIP. The mount pins the issuer https://10.0.20.204:6443, and Talos derives the issuer from the cluster endpoint.

The Terraform app root logs in to OpenBao from GitHub Actions with the job's OIDC token through a JWT auth mount:

• The read-only tf-plan role accepts any branch of igresc/homelab.
• The tf-apply role accepts only refs/heads/main and event_name=push.
• Both roles also bind the repository, the audience, and the Terraform workflow file (workflow_ref).

The vault provider sets skip_child_token = true in CI, so the read-only role does not need rights to create tokens. The first apply that creates these roles runs locally. ADR 011 covers the CI side.

Recovery roots are the deliberate exception to central secret storage. Keep OpenBao recovery shares, B2 read access, age identities, Talos recovery material, and break-glass GCP access in Bitwarden, with copies in Bitwarden cloud and the offsite recovery bundle.

Use an encrypted OpenBao Raft snapshot as the authoritative OpenBao backup. Longhorn volume backups are a secondary recovery option because separate volume backups do not form a coordinated Raft snapshot.

After important key changes, take an encrypted off-cluster Raft snapshot before applications depend on the new keys. The scheduled snapshot remains required. For irreplaceable data, identify which data backups and secret versions must recover together. Retain decryption-key versions while retained data or backups still need them.

Keep an independent encrypted recovery copy of application decryption keys whose loss would make personal data unreadable. Access to that copy must not depend on OpenBao or its GCP KMS seal key. Include it in the offsite recovery arrangement from ADR 006: Backups and rebuild. This is recovery-only escrow, not a second runtime secret authority. Keep an inventory of the applications that need this escrow, starting with Grafana's GF_SECURITY_SECRET_KEY. Test recovery before moving the affected data to Talos. If a key cannot be exported, establish a tested independent data recovery method before using it for irreplaceable data.

Implementation status

Staging reads secrets through External Secrets and the jwt-staging-grafana mount, which pins a public key and the issuer. OpenBao still runs as one member on k3s. The three-member cluster, the kubernetes auth method, the jwt-staging mount with JWKS checks, the GitHub OIDC roles, and the escrow inventory are future work.

Consequences

Applications have one runtime secret authority and the repository has no secret values. Recovery still depends on GCP Cloud KMS and the unseal key in Bitwarden and its cloud copy.

A static key is a long-lived credential. Its scope is limited to one KMS key, and rotating it means updating Bitwarden and its cloud copy together.

Staging depends on production OpenBao for its secrets. If production OpenBao is down, staging cannot fetch new secrets either.

Recovery shares cannot replace a permanently lost GCP KMS seal key. OpenBao snapshots alone do not protect against that loss. Independent application-key copies protect the corresponding data, not the OpenBao cluster itself. These key-protection requirements remain unverified until their recovery tests pass.

Three Raft members survive one member loss without extra storage copies. A lost strict-local member must rejoin with clean storage and copy state from the surviving quorum.

OpenBao 2.7 moves gcpckms to an external plugin. Pin and test that plugin in staging before upgrading from OpenBao 2.6.

References

• OpenBao GCP Cloud KMS seal
• OpenBao integrated storage
• OpenBao Kubernetes auth method
• OpenBao JWT auth method
• OpenBao auto-unseal runbook
• ADR 006: Backups and rebuild

Rollout gates

• On staging, three Raft members start and unseal with the KMS-scoped key. open
• A snapshot restore onto a cluster with a new identity has passed, without a secret from Git. open
• The operator can retrieve B2 and GCP recovery access from outside OpenBao. open
• Applications receive their migrated secrets through External Secrets. open
• Production External Secrets authenticates through local kubernetes auth. open
• Staging logins verify through JWKS on the jwt-staging mount, and still work after OpenBao moves to production Talos. open
• Each application on the escrow inventory has a tested key recovery. open

docs/decisions/009-openbao-secret-authority.md