Skip to content

Talos Cilium and Flux bootstrap

cilium.tf uses Terraform to bootstrap Cilium and Flux Operator for the Talos cluster. Terraform installs Cilium before Flux becomes ready because this cluster disables kube-proxy and uses Cilium for service routing.

Current limitation

The current Terraform code reads the GitHub deploy key from OpenBao at kv/infra/github/flux-staging and always selects the staging Flux path. Bootstrap needs the production OpenBao, which runs outside staging. A staging rebuild works while that OpenBao is up. Bootstrap can't recover from an OpenBao outage or a whole-site loss on its own.

Terraform also keeps the old Gitea deploy key, from kv/infra/gitea/flux-staging, as Secret flux-system, so Flux can roll back to Gitea. To retire it, remove flux_git_gitea and its Secret from terraform/cilium.tf and apply. The bootstrap module then deletes the Secret. Do this only after Flux has run from GitHub without problems and Gitea is no longer a rollback target.

Phase 3 of the migration adds bootstrap inputs for the Flux Git source, revision, and path. They are set where Git reconciliation can't revert them. None of this exists yet. When it does:

  • Staging bootstraps from kubernetes/clusters/staging.
  • Production bootstraps from kubernetes/clusters/prod.
  • A rebuild during a GitHub outage points the Git source at the read-only Docker host Git backup on NFS (ADR 001).
  • A rebuild starts with the applications suspended. That is a bootstrap input, not a separate recovery path (ADR 006).

The foundation root will read its credentials from Bitwarden, not OpenBao (ADR 009).

Pre-CNI API access

The Flux Operator bootstrap Job uses host_network = true because pod networking does not exist before Cilium is installed. Host networking alone does not give in-cluster Kubernetes clients access to the API.

Kubernetes injects the kubernetes Service address into every Pod. It normally sets these variables:

KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_SERVICE_PORT=443

That ClusterIP is unavailable until Cilium establishes the service dataplane. The bootstrap Job overrides these variables with Talos kubePrism. kubePrism listens on every Talos node before Cilium starts:

job = {
  host_network = true
  env = {
    KUBERNETES_SERVICE_HOST = "localhost"
    KUBERNETES_SERVICE_PORT = "7445"
  }
}

cilium-bootstrap-values.yaml uses the same localhost:7445 endpoint. A CNI bootstrap without working kube-proxy service routing needs an API endpoint that is reachable outside the Service CIDR. Talos uses kubePrism. Another platform can use a control-plane or load balancer address.

The upstream flux-operator-bootstrap module documents that CNI installation requires job.host_network, and it exposes job.env. The module does not document the host-network Service-IP limit or the kubePrism override.

Failure signature

Without the override, the Helm release fails because its post-install Job cannot reach the API server through 10.96.0.1:443.

error validating data: failed to download openapi:
Get "https://10.96.0.1:443/openapi/v2?timeout=32s": dial tcp 10.96.0.1:443: i/o timeout

The bootstrap Pod may run on a worker node and report Error. Networking causes the failure, not RBAC or the Flux Operator chart.

Diagnose and recover

Use the kubeconfig that Terraform generates. The default local kubeconfig can point to a different cluster.

tofu output -raw talos_kubeconfig >/tmp/talos-test.kubeconfig
KUBECONFIG=/tmp/talos-test.kubeconfig helm status flux-operator-bootstrap -n flux-operator-bootstrap
KUBECONFIG=/tmp/talos-test.kubeconfig kubectl logs -n flux-operator-bootstrap job/flux-operator-bootstrap
KUBECONFIG=/tmp/talos-test.kubeconfig kubectl get pods -n flux-operator-bootstrap -o wide

After you add or change the job.env values, run tofu apply. The changed Helm values trigger an upgrade and create a new bootstrap Job. Do not manually uninstall Helm.

Bootstrap resource lifecycle

After the bootstrap Job completes successfully, it deletes its ConfigMap, ServiceAccount, and ClusterRoleBinding. Terraform retains the bootstrap namespace, Helm release metadata, and bootstrap Secret. Terraform can then rerun the Job and reconcile the Flux Git deploy key when inputs change.

Do not delete these resources from a provisioner after a successful apply. Terraform detects drift and recreates them on the next apply. If the bootstrap must run only once, remove the flux_operator_bootstrap module after Flux is healthy. Then run tofu apply. Terraform uninstalls the bootstrap release and deletes its namespace. Terraform can no longer rerun the bootstrap or reconcile its managed Flux secret.

docs/runbooks/talos-flux-bootstrap.md