Operate the GCS Terraform state

Use this runbook to run OpenTofu against the main root's state in GCS, to copy the state to B2, and to recover the state. The design is in ADR 010. Run the commands from the repository root unless a step says otherwise.

Authenticate the operator

1. Sign in to the Google Cloud CLI and to Application Default Credentials with your own account. OpenTofu uses the Application Default Credentials. Both logins must use the same account.

gcloud auth login
gcloud auth application-default login
gcloud auth application-default set-quota-project homelab-igresc

1. Make sure that GOOGLE_APPLICATION_CREDENTIALS and GOOGLE_CREDENTIALS are not set. These variables take precedence over Application Default Credentials.
2. Do not copy the credential file into Git, command output or notes.

Normal state operations need roles/storage.objectAdmin on the bucket. Recovery of a soft-deleted bucket needs roles/storage.admin on the project.

Plan and apply

Run these commands from the repository root. They do not change the working directory.

1. Initialize the checkout. Use -reconfigure, not -migrate-state. GCS already holds the authoritative state. The OpenBao login and token setup are in Terraform platform documentation. If your token comes from the OpenBao Agent sink, put direnv exec terraform before each tofu command in this runbook.

umask 077
tofu -chdir=terraform init -reconfigure -input=false -lockfile=readonly

1. Save a plan in a private temporary directory and read it:

plan_dir="$(mktemp -d)"
tofu -chdir=terraform plan -input=false -out="$plan_dir/homelab.tfplan"
tofu -chdir=terraform show "$plan_dir/homelab.tfplan"

1. Apply only the saved plan that you read. Then delete the plan:

tofu -chdir=terraform apply "$plan_dir/homelab.tfplan"
rm -rf "$plan_dir"

1. Copy the state to B2. Refer to Copy the state to B2.

A plan file can contain secret values. Keep it outside Git and delete it after the apply.

Copy the state to B2

Do this after each apply that changed the state, until the Phase 3 daily workflow exists. Also do it after each apply of the bootstrap root.

1. Install the age YubiKey plugin if it is not installed. Refer to Set up age encryption.
2. Pull the state into a private temporary directory and record its serial:

umask 077
state_dir="$(mktemp -d)"
tofu -chdir=terraform state pull > "$state_dir/homelab.tfstate"
jq -r '.lineage, .serial' "$state_dir/homelab.tfstate"

For the bootstrap root, use terraform/bootstrap/gcs-state/terraform.tfstate as the source file. 3. Encrypt the state to all recovery recipients:

age --encrypt -R scripts/recovery/backup.recipients.txt \
  -o "$state_dir/homelab.tfstate.age" "$state_dir/homelab.tfstate"
sha256sum "$state_dir/homelab.tfstate.age"

1. Upload homelab.tfstate.age to the bucket homelab-igresc-state-backups under the prefix terraform/ with your B2 copy step. Put the root name, the UTC time and the serial in the object name.
2. Make sure that B2 reports the same SHA-256 checksum or SHA-1 content hash that you have locally.
3. Delete the temporary directory:

rm -rf "$state_dir"

Never upload plaintext state. Never encrypt to a key that is not in scripts/recovery/backup.recipients.txt.

Set up age encryption

Encryption needs only the public recipients. It does not need a YubiKey or a private key. To encrypt to the YubiKey recipients, the age YubiKey plugin must be on PATH.

sudo apt-get install libpcsclite-dev pkg-config
cargo install age-plugin-yubikey --version 0.5.1 --locked
export PATH="$HOME/.cargo/bin:$PATH"
age-plugin-yubikey --version

Without administrator access, you can build the plugin without sudo if the PC/SC runtime library is installed. The locked pcsc-sys build reads PCSC_LIB_DIR. These commands are for x86-64 Ubuntu with the Rust 1.90.0 toolchain installed. They put a temporary linker symlink to the existing library in a build directory:

(
    set -euo pipefail
    age_plugin_link_directory="$(mktemp -d /tmp/homelab-age-plugin-build.XXXXXX)"
    trap 'rm -f "$age_plugin_link_directory/libpcsclite.so"; rmdir "$age_plugin_link_directory"' EXIT
    test -f /usr/lib/x86_64-linux-gnu/libpcsclite.so.1
    ln -s /usr/lib/x86_64-linux-gnu/libpcsclite.so.1 "$age_plugin_link_directory/libpcsclite.so"
    PCSC_LIB_DIR="$age_plugin_link_directory" cargo +1.90.0 install age-plugin-yubikey --version 0.5.1 --locked --jobs 2
)
export PATH="$HOME/.cargo/bin:$PATH"
age-plugin-yubikey --version

The installed binary uses the system PC/SC runtime library. Run ldd "$HOME/.cargo/bin/age-plugin-yubikey" and make sure that no runtime library is missing. To remove this installation, run cargo uninstall age-plugin-yubikey.

Each of the three recipients can decrypt alone: two YubiKeys and the offline identity. Decryption with a YubiKey needs a PC/SC daemon and the YubiKey on the same machine.

Handle a lock

OpenTofu writes homelab/default.tflock at the start of each operation that writes state, and deletes it at the end. A lock that stays after an operation usually comes from an interrupted process.

1. Read the lock ID, the owner and the creation time from the OpenTofu error.
2. Find out if that operation is still running. Ask the owner, or check the process on the host that is named in the lock.
3. If the operation is still running, wait. Do not continue.
4. Make sure that the lock owner stopped and that no other writer is active.
5. Get the owner's approval. Remove the lock, then plan with locking on:

tofu -chdir=terraform force-unlock <lock-id>
tofu -chdir=terraform plan -input=false -detailed-exitcode

1. If the plan shows unexpected changes, stop. The interrupted operation can have written part of the state. Refer to Recover a previous state generation.

Never use -lock=false for an apply.

Select a recovery procedure

Each recovery procedure starts with a writer freeze and needs owner approval for the selected recovery generation. gcloud storage restore and gcloud storage cp write to the bucket directly. They do not use the OpenTofu lock. Only the writer freeze and the generation preconditions prevent a conflicting write.

Recover a previous state generation

Use this procedure when the live object has incorrect state. Object Versioning keeps each old generation.

1. Stop all OpenTofu writers and keep them stopped until state verification and the main-root plan pass.
2. List the generations of the state object. Record the generation of the live object:

gcloud storage ls --all-versions -l \
  gs://homelab-igresc-terraform-state/homelab/default.tfstate

1. Download the generation that you want and the current object. Compare their lineage, serial and resource list:

umask 077
recover_dir="$(mktemp -d)"
gcloud storage cp \
  "gs://homelab-igresc-terraform-state/homelab/default.tfstate#<generation>" \
  "$recover_dir/selected.tfstate"
gcloud storage cp gs://homelab-igresc-terraform-state/homelab/default.tfstate \
  "$recover_dir/current.tfstate"
jq -r '.lineage, .serial' "$recover_dir/selected.tfstate" "$recover_dir/current.tfstate"

The lineage must be the same. If the current object is not valid JSON, compare the selected generation with the last known lineage and serial. Keep current.tfstate as the copy of the corrupt generation. If infrastructure changed after the selected generation, stop. Fix the newest state instead of restoring an older one. 4. Get owner approval for the selected recovery generation. Copy the selected generation over the live object. The precondition stops the copy if another write occurred:

gcloud storage cp \
  "gs://homelab-igresc-terraform-state/homelab/default.tfstate#<generation>" \
  gs://homelab-igresc-terraform-state/homelab/default.tfstate \
  --if-generation-match=<current-generation>

The copy makes a new live generation. The current generation stays as a noncurrent version. 5. Initialize and plan with locking on:

tofu -chdir=terraform init -reconfigure -input=false -lockfile=readonly
tofu -chdir=terraform plan -input=false -detailed-exitcode

Do not release the writers until the plan has no unexpected changes. 6. Delete $recover_dir.

Recover a deleted state object or bucket

A delete of the live object in a versioned bucket makes the object noncurrent. It does not soft-delete it. Soft delete keeps a deleted noncurrent generation or a deleted bucket for 30 days. Try the procedures in this order.

Recover a noncurrent generation

1. Stop all OpenTofu writers and keep them stopped until state verification and the main-root plan pass.
2. List all generations of the state object:

gcloud storage ls --all-versions -l gs://homelab-igresc-terraform-state/homelab/default.tfstate

If the list has no generations, go to Recover a soft-deleted object. 3. Select a generation. Download it and examine its lineage, serial and resource addresses. The lineage must agree with the last known lineage.

umask 077
recover_dir="$(mktemp -d)"
gcloud storage cp \
  "gs://homelab-igresc-terraform-state/homelab/default.tfstate#<generation>" \
  "$recover_dir/selected.tfstate"
jq -r '.lineage, .serial' "$recover_dir/selected.tfstate"
tofu -chdir="$recover_dir" state list -state="$recover_dir/selected.tfstate"

state list reads the file offline. The recovery directory has no .tf files, so OpenTofu uses no backend. The list shows each resource instance by its full address, with its for_each or count key. If OpenTofu cannot read the file, do not use this generation.

Compare the list with the reviewed inventory. Use the output of tofu -chdir=terraform state list from before the incident, or the list from the last B2 copy. If a difference has no known cause, stop.

1. Get owner approval for the selected recovery generation. Copy it to the live name. The precondition stops the copy if a live object exists:

gcloud storage cp "gs://homelab-igresc-terraform-state/homelab/default.tfstate#<generation>" gs://homelab-igresc-terraform-state/homelab/default.tfstate --if-generation-match=0

If the precondition fails, stop. A live object exists. Use Recover a previous state generation. 5. Initialize and plan with locking on:

tofu -chdir=terraform init -reconfigure -input=false -lockfile=readonly
tofu -chdir=terraform plan -input=false -detailed-exitcode

Do not release the writers until the plan has no unexpected changes. 6. Delete $recover_dir.

Recover a soft-deleted object

Use gcloud storage restore only for a generation that the soft-deleted list shows.

1. Stop all OpenTofu writers and keep them stopped until state verification and the main-root plan pass.
2. List the soft-deleted generations of the state object:

gcloud storage ls --soft-deleted -l \
  gs://homelab-igresc-terraform-state/homelab/default.tfstate

1. Get owner approval for the selected recovery generation. Restore it. The precondition stops the restore if a live object exists:

gcloud storage restore "gs://homelab-igresc-terraform-state/homelab/default.tfstate#<generation>" --if-generation-match=0

If the precondition fails, stop. A live object exists. Use Recover a previous state generation. 4. Examine the lineage and serial of the restored object. The lineage must agree with the last known lineage.

gcloud storage cat gs://homelab-igresc-terraform-state/homelab/default.tfstate \
  | jq -r '.lineage, .serial'

1. Initialize and plan with locking on:

tofu -chdir=terraform init -reconfigure -input=false -lockfile=readonly
tofu -chdir=terraform plan -input=false -detailed-exitcode

Do not release the writers until the plan has no unexpected changes.

Recover a deleted bucket

Restore the bucket first. Then restore its objects.

1. Stop all OpenTofu writers and keep them stopped until state verification and the main-root plan pass.
2. List the soft-deleted buckets in the project:

gcloud storage ls --buckets --soft-deleted --full --project=homelab-igresc

1. Select the entry for homelab-igresc-terraform-state by its deletion time. Use its generation value, not its metageneration value.
2. Get owner approval for the selected recovery generation. Restore the bucket:

gcloud storage restore "gs://homelab-igresc-terraform-state#<bucket-generation>"

1. Restore the state object. Do Recover a noncurrent generation or Recover a soft-deleted object.
2. Run a bootstrap plan to make sure that the bucket settings are correct:

tofu -chdir=terraform/bootstrap/gcs-state plan -input=false -detailed-exitcode

Restore from the B2 copy

Use this procedure when GCS cannot supply good state. A restore from B2 loses all state changes after the copy.

1. Stop all OpenTofu writers and keep them stopped until state verification and the main-root plan pass.
2. Get the B2 read credential from Bitwarden cloud or the offsite bundle.
3. Download the newest copy for the root from homelab-igresc-state-backups, prefix terraform/. Make sure that its checksum agrees with B2.
4. Make a private recovery directory:

umask 077
recover_dir="$(mktemp -d)"

1. Decrypt the copy with one recovery identity.

For a YubiKey, make an identity file for its age key. Plugin slot 1 is PIV slot 82. age-plugin-yubikey --list-all shows the serial and slot of each key. Do not make new keys.

age-plugin-yubikey --identity --serial <device-serial> --slot 1 > "$recover_dir/yubikey.identity"
age --decrypt --identity "$recover_dir/yubikey.identity" -o "$recover_dir/restored.tfstate" <copy>.tfstate.age

For the offline identity, use its file on the removable drive as --identity. 6. Examine the restored state. The lineage must agree with the last known lineage. Review the resource addresses:

jq -r '.lineage, .serial' "$recover_dir/restored.tfstate"
tofu -chdir="$recover_dir" state list -state="$recover_dir/restored.tfstate"

state list reads the file offline. The recovery directory has no .tf files, so OpenTofu uses no backend. The list shows each resource instance by its full address, with its for_each or count key. If OpenTofu cannot read the file, do not use this copy.

Compare the list with the reviewed inventory. Use the output of tofu -chdir=terraform state list from before the incident, or the list from the previous B2 copy. If a difference has no known cause, stop.

1. If the bucket is gone and soft delete cannot restore it, recreate it with the bootstrap root. If the bootstrap state is also gone, restore it from its B2 copy to terraform/bootstrap/gcs-state/terraform.tfstate first. Then follow Bootstrap root. Apply only an approved bootstrap plan.
2. If a live state object exists, record its generation and keep a copy:

gcloud storage ls --all-versions -l gs://homelab-igresc-terraform-state/homelab/default.tfstate
gcloud storage cp gs://homelab-igresc-terraform-state/homelab/default.tfstate \
  "$recover_dir/current.tfstate"
tofu -chdir="$recover_dir" state list -state="$recover_dir/current.tfstate"

If OpenTofu cannot read the copy, find the cause before you continue. Make sure that the error does not come from an unsupported state version, the wrong OpenTofu version or a mistake in the command. If the object is corrupt, go to step 10. 9. Get owner approval for the restored state. Initialize, push the state and plan with locking on:

tofu -chdir=terraform init -reconfigure -input=false -lockfile=readonly
tofu -chdir=terraform state push "$recover_dir/restored.tfstate"
tofu -chdir=terraform plan -input=false -detailed-exitcode

A plain push refuses a different lineage, a lower serial, or different contents at the same serial. If the push fails, find the cause of the mismatch. Use -force only with the owner's approval after that investigation. Do not use -force to retry. Go to step 11. 10. If the live object is corrupt, state push cannot read it. -force does not help, because OpenTofu must decode the current state before it writes. Continue only if all of these conditions are true:

- `tofu state list` reads the restored state in step 6.
- You compared its addresses with the reviewed inventory.
- The owner approved the restored state.

Replace the object with the precondition set to the generation that you
recorded in step 8:

```bash
gcloud storage cp "$recover_dir/restored.tfstate" gs://homelab-igresc-terraform-state/homelab/default.tfstate --if-generation-match=<recorded-current-generation>
```

Then initialize and plan with locking on:

```bash
tofu -chdir=terraform init -reconfigure -input=false -lockfile=readonly
tofu -chdir=terraform plan -input=false -detailed-exitcode
```

1. Import or recreate resources made after the copy. Do not release the writers until the plan has no unexpected changes.
2. Delete $recover_dir. This also deletes the YubiKey identity file.

Failures

Bootstrap root

The bootstrap root terraform/bootstrap/gcs-state owns the bucket. Its state is local and is not in Git.

1. Plan and read the plan:

umask 077
tofu -chdir=terraform/bootstrap/gcs-state init -input=false -lockfile=readonly
tofu -chdir=terraform/bootstrap/gcs-state plan -input=false -out=bootstrap.tfplan
tofu -chdir=terraform/bootstrap/gcs-state show -no-color bootstrap.tfplan

1. Stop if the plan destroys or replaces the bucket. The bucket has prevent_destroy = true. Do not remove that rule to make a plan pass.
2. Apply the saved plan, then delete it.
3. Copy the bootstrap state to B2. Refer to Copy the state to B2.

Offline checks for the bootstrap root, with no cloud credentials:

bash terraform/bootstrap/gcs-state/validate.sh

Roll back to Terraform

To go back from OpenTofu to Terraform after a state write, follow the rollback in Terraform platform documentation.

References

• OpenTofu GCS backend
• OpenTofu state locking
• OpenTofu force-unlock
• OpenTofu state push
• Configure local Application Default Credentials
• Cloud Storage Object Versioning
• Use soft-deleted objects
• Restore soft-deleted buckets
• Copy preconditions
• Cloud Storage IAM roles
• age
• age YubiKey plugin

docs/runbooks/terraform-gcs-state.md