Skip to content

ADR 002

Manage the Talos cluster lifecycle with Terraform

Terraform and the Talos provider create the VMs, generate the machine configuration and bootstrap the cluster, so the cluster can be rebuilt from Git.

Status
Accepted
Rollout
0 of 5 gates

Accepted, not yet implemented.

Context

The homelab needs cloud-native patterns that one person can operate.

The current k3s cluster needs too much manual maintenance. The Talos cluster should make OS updates, Kubernetes upgrades, node replacement, backups, restore tests, and disaster recovery easier.

The platform uses PVE, Talos, Kubernetes, Terraform, Ansible, and GitOps. Cluster lifecycle and ownership need clear boundaries. Low maintenance matters more than adding tools for a portfolio.

The options were:

  • Manage Talos directly with Terraform.
  • Use Cluster API (CAPI).
  • Use Omni with Proxmox infrastructure providers.
  • Use Omni while Terraform retains ownership of Proxmox infrastructure.

Decision

Use Terraform and the Talos provider for the Talos and Kubernetes lifecycle.

Terraform is the source of truth for PVE infrastructure and Talos bootstrap. It creates VMs, configures networking, storage, and placement, generates Talos machine configuration, creates the cluster, and generates kubeconfig.

Keep the cluster lifecycle configuration in Git and apply it with Terraform. Declare Talos patches, machine roles, endpoints, and bootstrap settings in this repository.

Use talosctl for troubleshooting and recovery. Record every lasting change in Terraform.

Do not use Cluster API (CAPI).

Do not use Omni for the initial Talos platform. This avoids a subscription and another control plane, and keeps the cluster rebuildable from Git.

Ownership boundaries are:

Terraform -> Proxmox infrastructure, Talos configuration, bootstrap, kubeconfig
GitOps    -> Platform components and applications
Ansible   -> Non-Kubernetes systems and automation

The lifecycle flow is:

  • Terraform creates stable Proxmox VMs with declared names, VM IDs, CPU, memory, storage, network placement, and IP addressing.
  • The VMs boot a Talos image suitable for direct Talos configuration.
  • Terraform generates Talos machine secrets and machine configuration.
  • Terraform applies Talos configuration to the nodes.
  • Terraform bootstraps the first control plane node and retrieves kubeconfig.
  • After the Kubernetes API is available, Flux owns cluster add-ons, infrastructure controllers, and applications.

Keep node names, IPs, DNS records, VLANs, VM sizing, machine patches, and bootstrap settings in Terraform.

Versions and upgrades

Terraform variables declare the Talos and Kubernetes versions (talos_version and talos_kubernetes_version in terraform/variables_talos.tf). Renovate will bump them. talosctl runs the upgrades, one node at a time, as the Talos lifecycle runbook describes. tuppr may replace the talosctl runs later, after a trial proves it.

The Talos lifecycle runbook also covers version skew, upgrade order, and rollback.

Terraform roots

Split Terraform into three roots that use one CLI:

  • The bootstrap root owns the GCS state bucket. It uses a local backend.
  • The foundation root owns the Proxmox VMs, Talos, the Flux bootstrap, DNS, and the KMS and IAM resources that OpenBao needs to unseal. It reads its credentials from Bitwarden as sensitive variables, never from OpenBao. The operator applies it locally, because it can reboot nodes and must work while the cluster is down.
  • The app root owns what needs OpenBao: OpenBao policies and auth, and Grafana. CI plans it on every pull request and applies it on main.

The foundation root does not depend on anything that runs in the cluster. A rebuild can start with the cluster and OpenBao unavailable.

Staging

Run one permanent three-node staging cluster next to production. Staging has one small control-plane VM on each of three Proxmox hosts. It tests what a single-node cluster cannot: etcd quorum, a host failure, and Longhorn and etcd upgrades. Every platform change runs on staging first. Do not build temporary rehearsal clusters.

The existing Talos module instance (terraform/main.tf:112) is staging. Production is a second instance of the same module.

Implementation status

The staging module instance exists with one node. The bootstrap root exists at terraform/bootstrap/gcs-state. The foundation and app roots are still one root in terraform/, and the foundation credentials still come from OpenBao. Renovate does not manage talos_version yet. The foundation and app root split, the Bitwarden credentials, Renovate bumps for the Talos and Kubernetes versions, the three-node staging cluster, and production are future work.

Consequences

Benefits

  • Keeps the Talos lifecycle in Git and Terraform.
  • Avoids an Omni subscription.
  • Avoids another lifecycle control plane.
  • Enables reproducible cluster configuration through Git.
  • Avoids a CAPI management cluster.
  • Makes upgrades and recovery core workflows.
  • Staging and production use the same module, so staging tests the production code path.

Costs

  • Talos and Kubernetes upgrades are less automated than with Omni.
  • Node lifecycle needs good runbooks and scripts.
  • Terraform state is sensitive because it may contain Talos secrets and kubeconfig data.
  • Manual talosctl changes can create drift unless they move back to Terraform.
  • The permanent staging cluster uses capacity on three hosts that production also uses.

Rationale

Terraform-based Talos management fits this homelab. Cost, reproducibility from Git, and long-term simplicity matter more than a managed lifecycle UI.

Terraform already manages homelab infrastructure and PVE. Extending it to Talos bootstrap keeps the cluster lifecycle close to the VM definitions and avoids another source of truth.

The foundation root reads Bitwarden and not OpenBao because OpenBao runs in the cluster that the foundation root builds. Applying it locally keeps the steps that can reboot nodes out of CI.

A cluster that can be upgraded, restored, and rebuilt is more useful than one with many components and an unclear recovery path. Add a component only when it reduces risk, improves recovery, or provides a clear platform capability.

Omni remains a future option if direct Talos lifecycle becomes too much work. If that happens, move Talos configuration, node enrollment, and upgrades from Terraform and runbooks into Omni.

Rollout gates

  • The foundation root plans and applies with OpenBao unreachable. open
  • CI plans and applies the app root. open
  • Staging runs three control-plane nodes on three hosts. open
  • On staging, a Talos upgrade, a Kubernetes upgrade, and a host failure have passed. open
  • On staging, one etcd snapshot restore has passed. open

docs/decisions/002-talos.md