Skip to content

Cilium unhealthy

Impact

Cilium provides pod networking and replaces kube-proxy. An agent failure can leave a node NotReady. A broad failure can break services, DNS, ingress, and the Flux controllers that repair the cluster.

Triage

kubectl -n kube-system get pods -l k8s-app=cilium -o wide
kubectl -n kube-system get pods -l io.cilium/app=operator -o wide
kubectl -n kube-system get daemonset cilium
kubectl -n kube-system logs -l k8s-app=cilium --all-containers \
  --prefix --since=15m --max-log-requests=20
kubectl -n kube-system logs -l io.cilium/app=operator --all-containers \
  --prefix --since=15m --max-log-requests=10
kubectl -n kube-system get helmrelease cilium
kubectl -n kube-system describe helmrelease cilium
kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status --verbose
kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-health status --verbose

Compare failed agents with unavailable Talos nodes. Check the cilium Flux Kustomization. Check whether a chart, CRD, kernel, capability, or bootstrap value changed recently.

Decision

  • If one agent has failed because its node is unhealthy, use Talos node unavailable.
  • If a Flux release has failed after a change, suspend promotion. Inspect the HelmRelease, and revert the Git change if it is the confirmed cause.
  • If a fresh cluster has no working CNI, use the Cilium and Flux bootstrap runbook.
  • If API access through the Service IP is broken before Cilium starts, use the KubePrism bootstrap path documented there. Do not add ad hoc kube-proxy.
  • If a cluster-wide dataplane failure persists with healthy nodes, preserve the Cilium status and logs before restarting agents. Escalate as a platform incident.

Resolution checks

Resolve the alert when every scheduled Cilium agent is ready, the operator is available, all Nodes are Ready, CoreDNS resolves service names, a cross-node pod connection succeeds, and probes pass for ingress and one ClusterIP service.

docs/runbooks/alerts/cilium-unhealthy.md