Skip to content

Talos node unavailable

Impact

One Talos API endpoint or Kubernetes Node is unavailable. The impact depends on its role, etcd quorum, workload replicas, and node-local storage.

Triage

kubectl get node <node-name> -o wide
kubectl describe node <node-name>
talosctl version --nodes <node-ip>
talosctl service kubelet --nodes <node-ip>
talosctl dmesg --nodes <node-ip>

From Proxmox, check VM power, host health, datastore availability, NIC/VLAN, and recent migrations or backups. If the target is a control plane, run this immediately:

talosctl etcd status --nodes <all-reachable-control-plane-ips>

Decision

  • If Talos is reachable but kubelet has failed, inspect kubelet logs, disk, memory, time, and certificates before rebooting.
  • If the VM is stopped but the host and disk are healthy, start it and observe Talos services.
  • If a worker is unrecoverable, follow node replacement.
  • If one control plane is unrecoverable and etcd retains quorum, replace one node.
  • If etcd quorum is lost, follow etcd unhealthy.
  • If a Proxmox host or datastore failure affects multiple nodes, stop per-node actions and invoke disaster recovery.

Do not repeatedly reboot a node with an unknown disk or etcd fault. Protect node-local workload data before replacement.

Resolution checks

Resolve the alert when the Talos API is reachable, required services are healthy, the Kubernetes Node is Ready, Cilium is ready on the node, and evicted workloads have recovered. For a control-plane node, confirm that etcd membership is healthy.

docs/runbooks/alerts/talos-node-unavailable.md