Talos disaster recovery

Use this runbook when the Terraform-managed Talos Kubernetes cluster is lost. ADR 006: Backups and rebuild is the design. Application-specific restore procedures still apply.

Git restores desired state. Terraform restores infrastructure. Volume, database and OpenBao backups restore data. etcd snapshots restore Kubernetes objects, and you use them only for control-plane failures.

Safety rules

• Do not run talosctl bootstrap against a healthy cluster.
• Do not create new Talos machine secrets when you restore an existing etcd snapshot. The snapshot, nodes, and client configurations must use the original cluster secrets from Terraform state.
• Do not initialize an empty Terraform backend or apply it over surviving infrastructure. Recover state first.
• Do not restore etcd only because the Kubernetes API is unavailable. First find out whether the fault is in the network, API server, or Cilium.
• Before you attempt recovery, copy every candidate snapshot. Keep the original unchanged.
• Do destructive recovery from the Proxmox console or another management path that does not depend on the failed cluster.

Recovery expectations

The backup table in ADR 006 sets the backup method, max age, retention, restore method and status for each type of data. This runbook does not repeat it.

Each backup alerts when its max age is exceeded, not after two missed intervals. No alert rule exists yet. The backup stale runbook covers the response.

An upload is not proof that a backup works. A restore test is. No fixed recovery time is promised.

The OpenBao schedule does not replace the snapshot required after important key changes. Follow key protection before applications depend on new keys.

Required recovery assets

Proxmox VM backups can shorten recovery. They do not replace Terraform state, etcd snapshots, or application backups.

Verify the offsite recovery bundle

Before you move irreplaceable data to Talos, get the recovery credentials from outside the home. Use only Bitwarden cloud and the offsite bundle, with no onsite system. Confirm that you can get the decryption material and cloud-account access, including MFA recovery, through that route.

Decrypt one backup artifact. Test the recovery-only application-key copies described in key protection. Keep credentials, keys, and sensitive storage locations out of Git.

Choose the procedure

After cluster loss, rebuild from Git. This is the default. There is no second full-recovery path and no recovery mode.

Take an etcd snapshot

Use this manual procedure until the monitored talos-backup CronJob writes age-encrypted snapshots to B2. The job will use only the os:etcd:backup Talos role, a B2 write credential, and the public age recipients. The private age identities and the B2 read credential stay outside Kubernetes. Encrypt directly to scripts/recovery/backup.recipients.txt with age and the YubiKey plugin. Do not create an intermediate backup key on the VM. Follow the encryption setup before you run the job.

umask 077
export SNAPSHOT="/tmp/talos-etcd-$(date -u +%Y%m%dT%H%M%SZ).snapshot"
talosctl etcd status --nodes <healthy-control-plane-ip>
talosctl etcd snapshot "$SNAPSHOT" --nodes <healthy-control-plane-ip>
sha256sum "$SNAPSHOT" > "${SNAPSHOT}.sha256"

Copy the snapshot and checksum to the encrypted off-cluster backup target. Verify the copied checksum. Remove the local copy only after verification.

If quorum is already lost and a normal snapshot fails, take a last-resort database copy from a surviving control-plane node:

talosctl cp --nodes <control-plane-ip> /var/lib/etcd/member/snap/db /tmp/disaster.snapshot

This copy may be inconsistent. Keep it. Use --recover-skip-hash-check only when no validated regular snapshot exists.

Restore Terraform state

The state is in GCS, as ADR 010 describes. Restore it from a GCS object generation, or from the B2 copy if GCS cannot supply good state. The detailed commands are in Operate the GCS Terraform state.

Interim until Phase 3: the Terraform root still reads provider credentials from OpenBao. terraform/providers.tf and terraform/cilium.tf read them through the vault provider, and terraform/.envrc gets its token from OpenBao. Terraform steps work only while an OpenBao outside the lost cluster is reachable. Today that is the production k3s OpenBao. Phase 3 moves the foundation credentials to Bitwarden. This runbook has no workaround for a lost OpenBao before then.

1. Stop all Terraform writers. Record the incident start time.
2. From an independent management host, get the backend, B2, and Proxmox recovery credentials from Bitwarden or Bitwarden cloud. Until Phase 3, also confirm that the OpenBao outside the lost cluster is reachable.
3. Restore the latest known-good state object in the backend named in terraform/versions.tf. If the backend is unavailable, decrypt the latest validated B2 state copy and restore that version to the backend. Until Phase 3, B2 copies exist only if someone encrypted and uploaded one by hand after each apply. Keep the damaged or current object under another name.
4. Initialize without applying. Inspect the recovered state:

cd terraform
umask 077
direnv allow
tofu init -reconfigure
tofu state pull > /tmp/recovered-talos.tfstate
tofu state list
tofu plan -refresh=false -out=/tmp/recovered-talos.tfplan

1. Confirm that the state contains module.talos_vms.talos_machine_secrets.cluster, all expected Talos nodes, and no unexpected destroy or replacement actions in the saved offline plan. After the Kubernetes API is available, run a normal refresh and plan. -refresh=false does not check live infrastructure.
2. Remove the pulled state and saved plan after review:

rm -f /tmp/recovered-talos.tfstate /tmp/recovered-talos.tfplan

If no state copy with the original Talos secrets exists, you cannot safely attach an old etcd snapshot to new cluster secrets. Rebuild from Git as a new cluster and restore applications from their own backups.

Recover Git

Gitea is the current forge and the current source for recovery. Check out the recovery commit and confirm its hash before Terraform or Flux uses it.

Gitea retires in Phase 2 of the migration. After that, clone from GitHub. If GitHub is unavailable, use the Docker host's NFS-backed bare repo, or a dated bundle from TrueNAS or B2. Verify a bundle with git bundle verify, clone it, and run git fsck --full. The Docker host Git backup job is prepared but is not installed or running yet. Its SSH endpoint and first live restore still need owner setup.

The Flux source URL will become a bootstrap input, so a rebuild during a GitHub outage can point Flux at the Docker host's read-only copy. That endpoint needs both the Docker host and TrueNAS. If either is unavailable, restore a bundle on a recovery host first. Today Terraform hard-codes the source and the staging path. See Talos Cilium and Flux bootstrap.

Keep applications suspended

ADR 006 holds every application writer behind one suspension boundary. A bootstrap input starts the cluster with that boundary suspended, and Git reconciliation does not undo it. Restore tooling and its prerequisites, such as storage controllers and OpenBao, stay outside the boundary.

The boundary does not exist yet. Phase 3 of the migration builds it. Today several Flux Kustomizations reconcile applications on their own: apps, fdm-monster-staging, and grafana-staging on staging. A fresh bootstrap starts them as soon as their dependencies are ready.

Until the boundary exists, use this interim method. It has no staging test. It fits the staging layout only. Do not use it on the production tree. There apps includes openbao/ (kubernetes/apps/production/kustomization.yaml:28), so suspending apps suspends OpenBao too.

Before you change anything, write down its current value: the suspend value of each Kustomization, the suspend value of each CronJob, and the replica count of each Deployment and StatefulSet. Resuming undoes every change in this record.

1. Keep worker VMs powered off until Flux runs. This works only while workers are separate VMs. On hyperconverged nodes, application pods can start as soon as Flux reconciles them.
2. When Flux runs, list the Kustomizations and suspend every one that deploys an application writer:

kubectl -n flux-system get kustomizations
kubectl -n flux-system patch kustomization <name> --type=merge \
  -p '{"spec":{"suspend":true}}'

1. List the Deployments and StatefulSets those Kustomizations manage. Record each replica count. Then scale stateful or data-dependent controllers to zero.

kubectl get deployments,statefulsets -A \
  -l kustomize.toolkit.fluxcd.io/name=<name>
kubectl -n <namespace> scale <deployment-or-statefulset>/<name> --replicas=0

1. Check for CronJobs, Jobs and application operators that can restart a writer. Stop them too. Record each CronJob suspend value before you suspend it.
2. Check that pruning cannot delete storage objects or resources you need for the restore.
3. Check that the suspension survives when the parent Kustomization reconciles. Git reconciliation must not undo it. The staging Kustomizations come from kubernetes/clusters/staging, which flux-system applies. Reconcile flux-system, then confirm that each suspended Kustomization still has suspend: true:

flux reconcile kustomization flux-system --with-source
flux get kustomizations

Rebuild from Git

This is the default after cluster loss.

1. Stabilize Proxmox, the network, DNS, NTP, the Terraform backend, TrueNAS, Bitwarden, B2 access, and the Git source. Until Phase 3, the Terraform steps also need an OpenBao outside the lost cluster, which today is the production k3s OpenBao. See the interim note in Restore Terraform state.
2. Restore Terraform state and recover Git at the recovery commit.
3. Recreate the Kubernetes API endpoint and the control-plane VMs from Terraform.
4. Bootstrap Cilium and Flux with the Talos Cilium and Flux bootstrap runbook. Keep the applications suspended. Until the bootstrap input exists, use the interim method.
5. Restore in dependency order. A service that supplies platform prerequisites restores first, whatever its recovery tier in the glossary. Flux brings up the infrastructure Kustomizations in dependsOn order, as kubernetes/clusters/staging/infrastructure.yaml declares. No step reconciles a subset of them first. Wait until they are all Ready:

flux get kustomizations

cert-manager and External Secrets are in infra-controllers. Which Kustomizations must stay held until the restores finish is undecided until Phase 3. 6. Restore OpenBao from its Raft snapshot before Longhorn and CloudNativePG. Their B2 credentials arrive through External Secrets, so they need OpenBao first.

bao operator raft snapshot restore <decrypted-snapshot-file>

For unseal, follow OpenBao auto-unseal. Check that the rebuilt OpenBao can reach its GCP KMS key and unseals. Today it reaches the key through the method that ADR 009 describes. The plan moves to a static KMS-scoped key in Bitwarden. 7. Confirm that External Secrets can read OpenBao. 8. Restore Longhorn volumes and CloudNativePG databases while the applications stay suspended. 9. Check the restored data against the matching secret and key versions. 10. Resume the applications. With the interim method, undo every change you recorded. Set each Kustomization and CronJob suspend value back to its recorded value, and scale each Deployment and StatefulSet back to its recorded replica count:

```bash
kubectl -n flux-system patch kustomization <name> --type=merge \
  -p '{"spec":{"suspend":false}}'
kubectl -n <namespace> patch cronjob <name> --type=merge \
  -p '{"spec":{"suspend":false}}'
kubectl -n <namespace> scale <deployment-or-statefulset>/<name> \
  --replicas=<recorded-count>
```

1. Validate DNS, certificates, ingress, authentication, storage, monitoring, and external probes. Test one write and read in each restored stateful application.
2. Rotate credentials if backup media or state may have been exposed.

Recover etcd from a snapshot

Use this procedure only for a control-plane failure, after you confirm that etcd quorum cannot be repaired. It replaces the current etcd database. It is not the default for full cluster loss. If the data volumes are lost too, rebuild from Git.

1. Recover Terraform state and the original Talos client configuration.
2. Select the newest snapshot whose checksum and off-cluster copy are valid.
3. Confirm that Git still contains the desired manifests. Restored Flux controllers start to reconcile again when the API returns.
4. An etcd restore brings back every application as it was at snapshot time. Flux and the restored workloads start as soon as the API returns. The interim method races them and cannot prevent this. Use an etcd restore only for control-plane failures where the data volumes survive.
5. Recreate only failed control-plane VMs from the recovered Terraform state and original machine configuration. The Terraform root also contains Kubernetes and Helm resources, so an unrestricted apply can fail while the API is down. Use a saved, reviewed -target plan for only the required Proxmox VM and its dependencies. Test this break-glass path in staging first. Do not try it for the first time during an incident:

tofu plan \
  -target='module.talos_vms.proxmox_virtual_environment_vm.node["<control-plane-name>"]' \
  -out=/tmp/talos-control-plane-recovery.tfplan
tofu apply /tmp/talos-control-plane-recovery.tfplan

If the failed VM still exists in Proxmox but its disk or OS is corrupt, run the plan again with an explicit replacement:

tofu plan \
  -target='module.talos_vms.proxmox_virtual_environment_vm.node["<control-plane-name>"]' \
  -replace='module.talos_vms.proxmox_virtual_environment_vm.node["<control-plane-name>"]' \
  -out=/tmp/talos-control-plane-recovery.tfplan

The saved plan must show only the intended VM creation or replacement and its expected machine-config dependencies.

Do not let Terraform create new cluster secrets. Run a normal plan only after the API is back. Splitting the bootstrap infrastructure from the cluster resources is Phase 3 work. If the targeted plan cannot run with the API offline, recover the VM from a Proxmox backup. 6. Verify that every control-plane node uses machine type controlplane:

talosctl get machinetype --nodes <control-plane-ip-1>,<control-plane-ip-2>,<control-plane-ip-3>

1. After you keep copies of all available snapshots, wipe only EPHEMERAL on each control-plane node so that each etcd service enters Preparing:

talosctl reset --nodes <control-plane-ip> --graceful=false --reboot \
  --system-labels-to-wipe=EPHEMERAL

1. Check that etcd is Preparing on every control-plane node:

talosctl service etcd --nodes <control-plane-ip-1>,<control-plane-ip-2>,<control-plane-ip-3>

1. Run recovery exactly once against one control-plane node:

talosctl bootstrap --nodes <control-plane-ip-1> \
  --recover-from=/path/to/validated.snapshot

1. Wait for the other control-plane nodes to join. Then verify one leader, no learners, no errors, and matching applied indexes:

   talosctl etcd status --nodes <control-plane-ip-1>,<control-plane-ip-2>,<control-plane-ip-3>
   talosctl health
   kubectl get nodes -o wide
   

For a disaster database copy made with talosctl cp, add --recover-skip-hash-check to the one bootstrap command. Write down why you skipped the integrity check.

Check the recovery

Recovery is complete when all of these checks pass:

talosctl health
talosctl etcd status --nodes <all-control-plane-ips>
kubectl get nodes
kubectl get pods -A
kubectl get kustomizations,helmreleases -A

Also test one write and read in every restored stateful application.

Test schedule

ADR 006 lists the first tests. Each runs once on staging. After that, run one drill a year. It rotates through a volume restore, a database restore, an etcd restore and the Docker host's Git source. Run the full rebuild rehearsal again after any material change to recovery, such as control-plane topology, state storage, secrets, CNI bootstrap, or backup tooling.

References

• Talos disaster recovery
• ADR 001: Keep the homelab in a monorepo
• ADR 006: Backups and rebuild
• ADR 007: Run Talos as three hyperconverged nodes on three hosts
• ADR 008: Use Longhorn for persistent storage
• ADR 009: Use OpenBao as the secret authority
• ADR 010: Store Terraform state in Google Cloud Storage
• Flux suspension
• Flux pruning

docs/runbooks/disaster-recovery.md