Skip to content

Flux reconciliation stalled

Impact

Git changes are not reaching the cluster, or a platform dependency is blocked. Running workloads may remain available, but drift and failed releases accumulate.

Triage

kubectl -n flux-system get fluxinstance,gitrepositories
kubectl get kustomizations,helmreleases -A
kubectl -n flux-system get pods
kubectl -n flux-system logs deploy/source-controller --since=15m
kubectl -n flux-system logs deploy/kustomize-controller --since=15m
kubectl -n flux-system logs deploy/helm-controller --since=15m

Inspect the first failed dependency, not only the last blocked Kustomization. The Talos staging dependency order is gateway-api, cilium, infra-controllers, infra-configs, then applications.

Decision

  • If a Git fetch fails, check Gitea reachability, the deploy key, known_hosts, branch, and repository URL. Terraform owns the flux-system pull secret.
  • If rendering or validation fails, reproduce the failure with ./scripts/validate.sh. Fix the manifest in Git.
  • If a HelmRelease fails, inspect its source, values, hooks, events, and controller logs. Do not make an unrecorded live fix.
  • If controllers or the FluxInstance are absent on a recovered cluster, use the bootstrap runbook.
  • If Flux is blocked by Cilium, use Cilium unhealthy.

After correcting the source of truth, request reconciliation or wait for the declared interval:

flux reconcile kustomization <name> -n <namespace> --with-source
flux reconcile helmrelease <name> -n <namespace> --with-source

Do not use --force or --reset as the default repair. Avoid deleting Flux resources owned by Terraform.

Resolution checks

Resolve the alert after the GitRepository fetches the intended revision, every Kustomization and HelmRelease is Ready, controller logs have no repeating errors, and the applied revision matches the expected Git commit.

docs/runbooks/alerts/flux-reconciliation-stalled.md