Skip to content

OpenBao auto-unseal with GCP Cloud KMS

Use this runbook to configure and recover the OpenBao auto-unseal setup in this homelab.

Implementation status

The current production overlay runs one standalone Raft member and mounts a GCP service-account JSON key. ADR 009 keeps the JSON key and moves OpenBao to three Raft members. The three-member cluster is not implemented yet.

Overview

OpenBao encrypts its storage with a master key. Without auto-unseal, OpenBao splits that master key into Shamir unseal keys. You must manually unseal OpenBao after each restart.

With GCP Cloud KMS auto-unseal, OpenBao stores its encrypted master key in its Raft data. During startup, OpenBao asks Cloud KMS to decrypt the wrapping key. The pod can then restart without a manual unseal.

The setup has three parts:

  • Terraform creates the GCP Cloud KMS key and IAM permissions.
  • Kubernetes gives OpenBao GCP credentials.
  • The OpenBao Helm values configure the gcpckms seal stanza.

The target production setup on Talos runs three OpenBao Raft members. Each member uses one strict-local Longhorn replica. Required pod anti-affinity places the members on three different workers.

Changing the current standalone release into a three-member Raft release needs a tested migration procedure. Do not treat it as a replica-count change. The migration procedure is a production blocker and is not yet implemented.

Terraform resources

Terraform is in:

terraform

The prod Terraform root manages the following resources:

  • cloudkms.googleapis.com API enablement
  • KMS key ring: openbao
  • KMS crypto key: openbao-unseal
  • Service account: openbao-unseal@homelab-igresc.iam.gserviceaccount.com
  • IAM binding granting roles/cloudkms.cryptoKeyEncrypterDecrypter on only the OpenBao key
  • IAM binding granting roles/cloudkms.viewer on the same key so OpenBao can read key metadata

The key has prevent_destroy = true. Losing the key can make OpenBao data unrecoverable.

Apply changes as follows:

cd terraform
tofu init
tofu plan
tofu apply

Use these outputs:

tofu output openbao_gcp_kms_project
tofu output openbao_gcp_kms_location
tofu output openbao_gcp_kms_key_ring
tofu output openbao_gcp_kms_crypto_key
tofu output openbao_gcp_service_account_email

OpenBao configuration

The OpenBao production overlay is in this repository:

kubernetes/apps/production/openbao/values-patch.yaml

The OpenBao server config includes this seal stanza:

seal "gcpckms" {
  project    = "homelab-igresc"
  region     = "europe-west1"
  key_ring   = "openbao"
  crypto_key = "openbao-unseal"
}

The Helm values also mount a Kubernetes secret. They point Google client libraries to that secret:

server:
  extraEnvironmentVars:
    GOOGLE_APPLICATION_CREDENTIALS: /openbao/userconfig/openbao-gcp-kms/credentials.json
  extraVolumes:
    - type: secret
      name: openbao-gcp-kms

The openbao-gcp-kms Kubernetes secret must not be committed to Git.

Service-account key

OpenBao authenticates to GCP with a service-account JSON key stored as a Kubernetes secret. The key can only use the OpenBao unseal key. It does not depend on the cluster identity, so a rebuilt cluster can unseal without a change in GCP.

Create the key in a private temporary directory:

umask 077
credential_dir="$(mktemp -d)"
gcloud iam service-accounts keys create "${credential_dir}/openbao-gcp-kms.json" \
  --iam-account=openbao-unseal@homelab-igresc.iam.gserviceaccount.com

Create the Kubernetes secret:

kubectl -n openbao create secret generic openbao-gcp-kms \
  --from-file=credentials.json="${credential_dir}/openbao-gcp-kms.json"

After you verify the secret, store the JSON file in Bitwarden and its cloud copy. When the key rotates, update both. Then remove the temporary copy and directory:

rm -f "${credential_dir}/openbao-gcp-kms.json"
rmdir "${credential_dir}"
unset credential_dir

Never commit this JSON key, a Kubernetes Secret containing it, or a decrypted copy of it.

Install a new OpenBao cluster

For a fresh OpenBao install, do the following:

  1. Apply Terraform so that the KMS key and service account exist.
  2. Create the openbao-gcp-kms Kubernetes secret in the openbao namespace.
  3. Deploy/reconcile the OpenBao GitOps manifests.
  4. Initialize OpenBao.

Initialize:

kubectl -n openbao exec -it openbao-0 -- bao operator init

With auto-unseal enabled, OpenBao should return recovery keys rather than normal unseal keys. Store the recovery keys and initial root token in a password manager or an offline backup.

Recovery keys do not replace the GCP KMS key. A restored OpenBao cluster still needs the original KMS key and the service-account key to unseal.

Migrate an existing Shamir seal

If OpenBao was initialized before you added the gcpckms stanza, migrate the existing seal. This operation requires downtime. Keep the old Shamir shares and the new GCP KMS seal available until migration and restart checks pass.

  1. Schedule downtime. Stop writes to OpenBao.
  2. Create a verified OpenBao Raft snapshot. Copy the snapshot outside the cluster.
  3. Record the active KMS key version. Confirm that the original Shamir shares are available.
  4. Confirm that the GCP KMS resources and required IAM permissions exist.
  5. Create the openbao-gcp-kms Kubernetes secret.
  6. Deploy or reconcile the updated OpenBao Helm values.
  7. Restart the OpenBao pod so that it loads the new seal configuration.
  8. Run migration unseal with the existing Shamir unseal keys.

The OpenBao Helm chart uses an OnDelete StatefulSet update strategy. A ConfigMap or StatefulSet update does not automatically restart openbao-0. If bao operator unseal -migrate returns no migration seal found, the pod usually still runs the old configuration.

Restart the pod:

kubectl -n openbao delete pod openbao-0

Before you enter unseal keys, wait for the replacement pod to run:

kubectl -n openbao get pod openbao-0 -w

Run this once per required unseal key share until the threshold is met:

kubectl -n openbao exec -it openbao-0 -- bao operator unseal -migrate

After migration, restart the pod. Verify that it unseals automatically.

Verification

Check the pod status:

kubectl -n openbao get pods

Check the OpenBao seal status:

kubectl -n openbao exec -it openbao-0 -- bao status

After successful auto-unseal, expect the following:

  • Initialized is true
  • Sealed is false
  • seal type references GCP KMS or recovery seal behavior

Test a restart as follows:

kubectl -n openbao delete pod openbao-0
kubectl -n openbao wait --for=condition=Ready pod/openbao-0 --timeout=5m
kubectl -n openbao exec -it openbao-0 -- bao status

The pod becomes ready without manual bao operator unseal.

Recovery notes

Keep the following items backed up outside the repository:

  • OpenBao recovery keys
  • Initial root token until replaced/revoked according to normal procedure
  • GCP service-account JSON key, in Bitwarden and its cloud copy
  • Break-glass access to the GCP project and KMS key
  • Age identity and B2 read credential for the OpenBao backup
  • OpenBao Raft snapshots

Do not destroy or rotate away the active KMS key until you confirm that OpenBao can still decrypt its seal data. The Terraform key resource uses prevent_destroy to reduce this risk.

Use the OpenBao Raft snapshot as the authoritative backup. Separate Longhorn backups of the three member volumes are not a coordinated database snapshot.

If one strict-local member loses its worker and volume, keep the surviving quorum. Remove the failed peer and create a clean replacement member. Restore the whole OpenBao cluster from a Raft snapshot only after quorum is lost.

Protect keys and match recovery points

These checks are required before irreplaceable data moves to Talos. The application-key inventory, recovery copies, and tests remain unverified.

  1. Identify keys whose loss would make personal data unreadable. Record which application-data backups need each key version, without recording key values in Git. Distinguish these keys from passwords that can be reset.
  2. After important key changes, take an encrypted off-cluster OpenBao Raft snapshot before applications depend on the new keys. Verify the artifact's integrity and retrieval. Keep the scheduled snapshot as well.
  3. Keep an independent encrypted recovery copy of each irreplaceable application decryption key. Retain versions while retained data or backups need them. Access must not require OpenBao or its GCP KMS seal key. Use the offsite recovery bundle. If a key cannot be exported, test an independent data-recovery method before using that key for irreplaceable data.
  4. In isolated staging, test an OpenBao restore and decrypt representative data with the independent key copy. Test the latter without OpenBao or its KMS seal key. Do not disable or delete the production KMS key to simulate loss.
  5. Before enabling restored applications, match the data recovery point to its required secret versions. If restored passwords differ from a database or surviving external service, reset them through that service's recovery procedure. Record validation results, never secret values.

Recovery-only copies do not become another runtime secret authority. They protect the corresponding application data, not OpenBao itself: recovery shares and Raft snapshots cannot recover OpenBao after permanent loss of its seal key.

Security tradeoff

Auto-unseal removes manual unseal work but mounts a long-lived GCP service-account JSON key in the OpenBao pod. Anyone who can read the openbao-gcp-kms Kubernetes secret can use the service account to call the KMS key. The IAM bindings limit the account to that one key.

Do not store OpenBao recovery keys in Git.

Upgrade warning

OpenBao 2.7 moves gcpckms from the built-in seal set to an external openbao-plugins plugin. Before you upgrade to 2.7 or later, deploy and test the plugin path in staging. Do not treat that upgrade as an image-only change.

Diagnose unknown wrapper: gcpckms

If an OpenBao 2.7 pod enters CrashLoopBackOff, check its image and previous container logs:

kubectl -n openbao get pod openbao-0 -o wide
kubectl -n openbao get pod openbao-0 -o jsonpath='{.spec.containers[*].image}{"\n"}'
kubectl -n openbao logs openbao-0 --previous --tail=100 --timestamps
kubectl -n openbao get sts openbao -o jsonpath='{.spec.updateStrategy.type}{"\n"}'

This error means the configured seal provider is unavailable in the server:

Error configuring seal "gcpckms": unknown wrapper: gcpckms

Check for a declarative plugin "kms" "gcpckms" definition and an installed plugin binary. For OCI distribution, also check plugin_auto_download and a writable plugin_directory. Follow the upstream seal plugin configuration and declarative plugin guide. KMS plugins require server configuration because they must work before unseal.

Verify the active Flux Git source before editing manifests. The existing k3s production cluster reconciles igresc/gitops, while this repository owns the Talos staging configuration. Update the production release in its owning repository when repairing the k3s instance.

Before changing the server, establish a recovery path for its existing Raft data. Install and declare a pinned GCP KMS plugin compatible with the server. Preserve the existing KMS key, seal parameters, credentials, and Raft PVC. Pin the chart and server image so chart defaults cannot silently cross a seal compatibility boundary. Test startup and auto-unseal in staging before rollout. Do not reinitialize OpenBao to address a missing plugin.

For an OnDelete StatefulSet, replace the pod only after the corrected configuration and plugin are available. A successful Helm upgrade does not prove that the replacement pod can start. After repair, use the verification steps above and the unavailability runbook.

References

docs/runbooks/openbao-auto-unseal.md