ADR 010
Store Terraform state in Google Cloud Storage
The main OpenTofu root keeps its state in a versioned, locked GCS bucket that stays up when everything OpenTofu manages is down.
- Status
- Accepted
- Related
Context
The main OpenTofu root used to keep its state in Garage on TrueNAS, through an S3-compatible backend. Garage is still useful storage, but Terraform state is a recovery-critical asset. That backend had no proven locking and no proven way to recover an old version. Plans also stalled when TrueNAS disks were busy.
The state backend must stay available when Kubernetes, TrueNAS, the Git forge
and the infrastructure that OpenTofu manages are down. The native gcs
backend supports state locking and stores state in a bucket that exists before
the root. Google Cloud Storage also has Object Versioning and soft delete, so
an overwritten or deleted state object can be recovered.
Decision
Use Google Cloud Storage as the authoritative backend for the main root's state.
The state bucket homelab-igresc-terraform-state is in europe-west1 and has
these properties:
- Standard storage class.
- Public access prevention enforced.
- Uniform bucket-level access.
- Object Versioning enabled.
- Soft delete enabled, with an explicit retention of 30 days.
- Google-managed encryption at rest.
force_destroy = falseandprevent_destroy = true.
Do not add a customer-managed encryption key. The backend is part of the recovery path. Another key and its IAM would make recovery harder and would not address the main threat to a homelab.
A separate small OpenTofu root, terraform/bootstrap/gcs-state, owns the
bucket. It uses a local backend. Never move its state into the bucket that it
creates.
The main root uses the gcs backend with prefix homelab. Keep credentials
out of the backend block and out of Git. Operators use their own Google
Application Default Credentials. CI will use GitHub OIDC through Workload
Identity Federation, with a read-only plan identity for any branch and an
apply identity for main only. No service-account JSON keys.
Offsite copy
GCS versions protect against bad writes and deletes, but not against loss of
the Google account or project. A daily scheduled workflow copies the state of
each root to the B2 bucket homelab-igresc-state-backups, under the prefix
terraform/, encrypted with age to the recovery recipients in
scripts/recovery/backup.recipients.txt. It uploads only when the serial
changed. It keeps the latest copy of each root for ever and older copies for
90 days. The workflow arrives in Phase 3 of the simplification plan. Until
then, the operator copies the state to B2 by hand after each apply. The
bootstrap root's state is always copied by hand, after its rare applies.
If GCP is lost, at most one day of state is lost. Resources created in that day must be imported or recreated, and generated credentials such as Talos machine secrets can be lost with them.
No in-cluster backup job, wrapper around OpenTofu commands, or extra workload identity is built for state backups.
Recovery rules
Once newer state is written to GCS, never restore the old Garage copy over it. Recovery starts from the newest known-good GCS generation or an explicitly selected B2 copy.
Never combine a backend change with a CLI change, provider upgrades, CI automation, a forge migration, OpenBao changes or unrelated infrastructure changes.
Implementation status
Done. The bootstrap root created the bucket on 3 October 2026. Locking,
previous-generation recovery, soft-deleted object recovery and soft-deleted
bucket recovery were tested before the cutover. The main root moved from
Garage to GCS in October 2026 with terraform init -migrate-state, and its
lineage, serial and resources were checked after the move. It then moved to
OpenTofu 1.13.1 as a separate change. The
staging state-backup CronJob and its identities were removed. The daily B2
workflow is still to do.
The final Garage copy stays untouched until the rollback window closes.
Consequences
Normal state operations no longer depend on Garage, TrueNAS, Kubernetes or the Git forge.
Native locking reduces the risk of two concurrent writers. Object Versioning keeps every old generation readable, and soft delete protects against deletion of objects and of the bucket.
GCS and the operator's Google account are now recovery prerequisites. The B2 copy is necessary so that loss of GCP access does not also remove the only copy. Until the daily workflow exists, the B2 copy is only as fresh as the last manual copy.
The bootstrap root's local state is small but must be protected separately and must never be committed to Git.
Old generations and soft-deleted data use storage until lifecycle and retention rules remove them. Do not expire old generations aggressively.