Skip to content

ADR 007

Run Talos as three hyperconverged nodes on three hosts

Production runs three Talos control-plane nodes on three Proxmox hosts. Workloads, Longhorn and etcd share them, so one host can fail without manual work.

Status
Accepted
Rollout
0 of 5 gates
Related
002

Accepted, not yet implemented.

Context

The production Talos cluster must keep etcd quorum and workload capacity after one Proxmox host fails, without manual work. More control-plane members do not improve this target. Four members still need three votes, while three members need two.

Separate worker and control-plane VMs double the number of nodes to upgrade, patch, and size, and they put two VMs on some hosts. For one operator, fewer nodes cost less to maintain.

The five Proxmox hosts share the Layer 2 network used by the Kubernetes API.

Decision

Production nodes

Run three production Talos VMs on three different Proxmox hosts. Each VM is a control-plane node with allowSchedulingOnControlPlanes: true and a dedicated Longhorn data disk. Workloads, Longhorn, and etcd share the three nodes. Add dedicated workers only when CPU or memory runs short.

Label every node with its physical Proxmox host so that Kubernetes and Longhorn can spread replicas across hosts.

Capacity

Keep at least a third of each node's CPU and memory free, so two nodes can carry the load while one host is down. Measure the real load, then add margin for Longhorn rebuild traffic.

Controllers

Run two replicas of each of these controllers, spread across nodes, each with a PodDisruptionBudget:

  • The Cilium operator.
  • The Longhorn CSI controllers.
  • CoreDNS.
  • Traefik.
  • The cert-manager webhooks.
  • External Secrets.

Ingress addresses

The Cilium L2 announcement policy selects all three nodes, not one. Traefik uses externalTrafficPolicy: Cluster. Cilium documents Local as incompatible with L2 announcements.

Control-plane access

Use a Talos Layer 2 virtual IP as the Kubernetes API endpoint. Reserve the address outside DHCP and configure it only on control-plane nodes.

Configure talosctl with all three control-plane node addresses as direct endpoints. Do not use the virtual IP as a Talos management endpoint, because the virtual IP depends on etcd quorum.

Change one control-plane member at a time. Wait for etcd health and matching membership before the next change.

Staging

Run a permanent three-node staging cluster with the same shape: one small control-plane VM on each of three Proxmox hosts, about 2 vCPU and 6 to 8 GB of memory each. With five hosts, staging and production share some hosts. ADR 002 records what staging tests.

Implementation status

Staging has one control-plane node and one worker. The L2 announcement policy selects only talos-test-worker-01 (kubernetes/infrastructure/staging/network/ingress-address.yaml). Staging Traefik uses Cluster, but the shared values still say Local (kubernetes/infrastructure/controllers/traefik/release.yaml:78). Terraform still allows Longhorn only on workers (terraform/variables_talos.tf:56 and terraform/modules/talos_vms/variables.tf:134).

Consequences

The cluster survives one Proxmox host failure. etcd loses quorum after two of the three hosts fail.

Workloads share nodes with etcd and the API server. A workload that exhausts a node's CPU or memory can slow the control plane, so requests, limits, and the capacity margin matter more than with dedicated workers.

A shared switch, power source, or datastore remains a shared failure point even when VMs use separate hosts.

The API virtual IP avoids another load-balancer service. A sudden control-plane loss can interrupt API access while the virtual IP moves.

With externalTrafficPolicy: Cluster, traffic that arrives on a node without a Traefik pod can reach Traefik with a node address instead of the client address.

References

Rollout gates

  • Staging runs three control-plane nodes on three hosts. open
  • The controllers above run two replicas with PodDisruptionBudgets on staging. open
  • The L2 policy selects all three nodes, and Traefik uses Cluster in every environment. open
  • On staging, a host failure leaves the API, ingress, and workloads available without manual work. open
  • Production runs on three hosts with the capacity margin measured. open

docs/decisions/007-talos-production-topology.md