Skip to content

Restore an unavailable Terraform backend

Impact

OpenTofu cannot read or lock the state in GCS, or the state is inconsistent. The state contains Talos secrets and infrastructure identity. Stop lifecycle changes until the backend is trustworthy.

Triage

From a management host, check these items:

  • The Google Cloud Storage status and network access to storage.googleapis.com.
  • The active account and Application Default Credentials, as in Authenticate the operator.
  • The bucket homelab-igresc-terraform-state and the object homelab/default.tfstate: their existence, current generation and serial.
  • A stale lock object homelab/default.tflock, and any OpenTofu writer that is still running.

Decide what to do

  • If GCS is temporarily unavailable, wait. Do not use local state or -backend=false.
  • If credentials expired or the account changed, sign in again with Application Default Credentials and reinitialize the backend.
  • If a lock stays after an interrupted operation, follow Handle a lock.
  • If the state object is wrong, deleted or corrupt, follow Recover a previous state generation or Recover a deleted state object or bucket.
  • If GCS cannot supply good state, follow Restore from the B2 copy.
  • If an OpenTofu operation may still be running, find its owner before you remove a lock or write state.

Check the resolution

Resolve the alert only when tofu init -reconfigure, tofu state pull, and a normal read-only plan succeed against the expected state serial. Make sure that the newest B2 copy has the current serial.

docs/runbooks/alerts/terraform-backend-unavailable.md