Skip to content

Kubernetes API unavailable

Impact

The Kubernetes API endpoint cannot serve requests. Workloads may continue, but scheduling, reconciliation, and operator-driven recovery cannot work normally. Treat a production-wide failure as critical.

Triage

  1. From an independent management host, confirm the alert:
curl --fail --show-error --connect-timeout 5 \
  --cacert <cluster-ca.pem> https://<stable-api-endpoint>:6443/livez
talosctl version --nodes <control-plane-ip>
  1. Check whether the stable endpoint, DNS, or TCP load balancer has failed while direct control-plane addresses still work.
  2. Without Kubernetes, check the Talos control-plane services:
talosctl service kube-apiserver --nodes <all-control-plane-ips>
talosctl service etcd --nodes <all-control-plane-ips>
talosctl etcd status --nodes <all-control-plane-ips>
talosctl logs kube-apiserver --nodes <affected-control-plane-ip>
  1. From the management plane, check the Proxmox VM state, VLAN reachability, DNS, NTP, and storage.

Decision

  • If direct APIs are healthy but the stable endpoint has failed, repair the TCP load balancer, VIP, or DNS. Do not change etcd.
  • If the API server has failed but etcd is healthy, inspect certificates, static-pod logs, disk, memory, and recent machine changes.
  • If one control plane has failed and etcd has quorum, follow node replacement.
  • If etcd has no leader or quorum, follow etcd unhealthy.
  • If the control plane is completely lost and a valid snapshot exists, follow disaster recovery.

Resolution checks

curl --fail --show-error --cacert <cluster-ca.pem> \
  https://<stable-api-endpoint>:6443/livez
talosctl health
kubectl get nodes
kubectl get kustomizations,helmreleases -A

Confirm that the stable endpoint works from both an operator host and a cluster node.

docs/runbooks/alerts/kubernetes-api-unavailable.md