Skip to content

etcd unhealthy

Impact

etcd stores Kubernetes control-plane state. Loss of the majority stops control-plane writes. Loss of the database requires snapshot recovery. This alert is critical when there is no leader, a majority is unavailable, or a NOSPACE alarm exists.

Triage

talosctl etcd status --nodes <all-control-plane-ips>
talosctl etcd members --nodes <healthy-control-plane-ip>
talosctl etcd alarm list --nodes <healthy-control-plane-ip>
talosctl service etcd --nodes <all-control-plane-ips>
talosctl logs etcd --nodes <affected-control-plane-ip>

Check whether members agree on membership and the leader. Check whether any member is a learner, whether applied indexes converge, and whether the fault correlates with a network, disk, time, or Proxmox failure.

Decision

  • If one member has failed and the majority is healthy, repair connectivity or use node replacement.
  • If membership contains a confirmed dead node but the majority is healthy, use the unreachable control-plane procedure in the node replacement runbook.
  • If NOSPACE is active, preserve a snapshot. Inspect DB size, then follow the Talos etcd maintenance procedure. Defragment one member at a time. It blocks reads and writes on that member.
  • If there is no leader or no majority, stop routine maintenance. Determine whether quorum can be restored without changing membership.
  • If quorum is irrecoverable, use the tested etcd restore procedure.

Do not run bootstrap, wipe EPHEMERAL, or remove members during initial triage.

Resolution checks

Resolve the alert after all expected members report one leader, no learner, no errors, no active alarms, and converged applied indexes. Then verify talosctl health, Kubernetes API writes, Flux reconciliation, and the age of the latest successful snapshot.

docs/runbooks/alerts/etcd-unhealthy.md