Skip to content

ADR 003

Use Cilium for Talos networking

Talos runs without its default CNI or kube-proxy. Cilium handles pod networking, service routing and NetworkPolicy, and installs at bootstrap before Flux.

Status
Accepted
Related
007

Context

The new Kubernetes cluster will run on Talos and use GitOps. It needs NetworkPolicy, bare-metal LoadBalancer services, and observability.

The CNI should require less maintenance than the current k3s setup. Extra features are worthwhile only when they do not create constant operating work.

Talos can use its default CNI. That option is simpler but provides fewer platform features.

Decision

Use Cilium as the cluster CNI from day one.

Run Talos without the default CNI and with kube-proxy disabled. Cilium provides pod networking, kube-proxy replacement, NetworkPolicy, and the base for service advertisement and observability.

Install Cilium during cluster bootstrap, before normal GitOps workloads. Do not depend on Flux being fully healthy to install the CNI. Nodes need a CNI before they can become ready.

The day-one Cilium scope is:

  • Pod networking.
  • Kube-proxy replacement.
  • NetworkPolicy enforcement.
  • LoadBalancer IPAM.
  • L2 service advertisement.

Traefik stays the ingress controller. Using Cilium's Gateway API in place of Traefik belongs to the later ingress and remote access plan. Keep Cilium's Gateway API support and the Gateway API CRDs enabled while the cilium Flux Kustomization depends on the gateway-api Kustomization (kubernetes/clusters/staging/infrastructure.yaml).

Defer these capabilities:

  • Gateway API as the ingress path.
  • BGP service advertisement.
  • Hubble observability.

Enable BGP and Hubble only after staging has proven the upgrade and backup procedures.

Consequences

The cluster gets NetworkPolicy and service advertisement from one networking layer.

Cilium is more complex than the default Talos CNI. Bootstrap must install it early because nodes need the CNI to become ready.

Keep the initial Cilium configuration small. Test advanced networking in staging before production depends on it.

ADR 007 sets how L2 announcements and Traefik's externalTrafficPolicy work together.

Gateway-level SSO and advanced edge authentication are not requirements. If needed later, evaluate Envoy Gateway on top of Cilium.

docs/decisions/003-talos-cni.md