Skip to content

Talos node replacement

Use this runbook to replace one failed, corrupt, or retired Talos VM while the cluster can still recover. If etcd quorum is lost, stop. Use disaster recovery.

Preconditions

  • Terraform state is available and contains the existing talos_machine_secrets.cluster resource.
  • The replacement retains the declared inventory identity, role, IP, VM ID, and machine configuration unless a reviewed migration explicitly changes them.
  • Application data on local or node-bound volumes has an independent recovery path. Recreating a Talos VM does not restore workload data.
  • The cluster is healthy except for the target node.
  • A control-plane replacement retains a healthy majority of etcd members.

Prepare access as described in lifecycle management. Then record the target node, IP, role, Proxmox host, VM ID, datastore, and reason for the replacement.

Replace a worker

  1. If the worker is reachable, let Talos drain and erase it:
talosctl reset --nodes <worker-ip>
  1. After its workloads are safe elsewhere, delete the old Kubernetes Node object:
kubectl delete node <worker-name>
  1. From terraform/, review a targeted replacement plan:
tofu plan \
  -replace='module.talos_vms.proxmox_virtual_environment_vm.node["<worker-name>"]' \
  -out=/tmp/talos-node-replace.tfplan
  1. If the plan replaces cluster secrets, another VM, or shared infrastructure, stop. Otherwise, apply the saved plan:
tofu apply /tmp/talos-node-replace.tfplan
  1. Terraform recreates the VM and reapplies a configuration derived from the existing cluster secrets. Do not run talosctl bootstrap.
  2. Verify that the node becomes Ready, Cilium is healthy on the node, workloads reschedule, and Flux remains ready.

Replace a healthy control-plane node

Replace one control-plane node at a time.

  1. Verify all etcd members, the leader, and errors:
talosctl etcd status --nodes <all-control-plane-ips>
talosctl etcd members --nodes <healthy-control-plane-ip>
  1. Take and copy an off-cluster snapshot. Use the DR procedure.
  2. Gracefully reset the target. Talos drains the target and removes it from etcd:
talosctl reset --nodes <target-control-plane-ip>
kubectl delete node <target-control-plane-name>
  1. Before changing Proxmox, confirm that the remaining etcd members are healthy.
  2. Run the worker Terraform -replace plan and apply, using the control-plane inventory key.
  3. Do not bootstrap the replacement. It joins the existing control plane and etcd cluster from its Terraform-generated machine configuration.
  4. Wait for the replacement to become a voting member. Before you replace anything else, verify one leader, no learner, no errors, and a Ready Node.

Replace an unreachable control-plane node

If the node cannot perform a graceful reset but the surviving etcd members still have quorum, do the following:

  1. Take a fresh snapshot from a healthy member.
  2. List members. Identify the failed member by name and peer address:
talosctl etcd members --nodes <healthy-control-plane-ip>
  1. Remove only the confirmed failed member:
talosctl etcd remove-member <member-id> --nodes <healthy-control-plane-ip>
  1. Delete the stale Kubernetes Node object. Replace the VM through the reviewed Terraform plan.
  2. After the replacement joins, verify etcd membership and cluster health.

Do not remove an etcd member based only on an alert label or a guessed IP. If the surviving members lack quorum, removing a member does not repair the cluster. Use the etcd restore procedure.

Stop conditions

Stop and move to incident recovery if any of these conditions occur:

  • Terraform proposes replacing talos_machine_secrets.cluster.
  • A second control-plane or storage node fails.
  • etcd loses its leader or majority.
  • The replacement tries to join with a duplicate identity or unexpected IP.
  • Local persistent data has no verified restore source.
  • Cilium does not become healthy on the replacement.

Completion evidence

talosctl health
talosctl etcd status --nodes <all-control-plane-ips>
kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl get kustomizations,helmreleases -A

Record the Terraform plan, replacement VM ID, etcd membership before and after the replacement, snapshot identifier, disrupted workloads, and validation result.

Reference

docs/runbooks/node-replacement.md