Talos lifecycle management

Use this runbook for planned changes to the Terraform-managed Talos cluster. If etcd quorum or the whole control plane is lost, use disaster recovery. For one failed or retired node, use node replacement.

Ownership

Use manual talosctl changes only for upgrades, diagnosis, and recovery. Before the maintenance window closes, represent every persistent configuration change in Terraform.

Current staging topology

The checked-in inventory is not ready for production. It has one control plane and one worker. Both run on PVE5 and ssd-zfs. The Kubernetes endpoint is the control-plane address 10.0.20.204:6443. The inventory also bootstraps the staging Flux tree and uses the staging deploy key.

Before you move workloads to production, all of these gates must pass:

• Run three hyperconverged Talos nodes on three different Proxmox hosts, as ADR 007 describes. Each node is a control-plane node with allowSchedulingOnControlPlanes: true. Add dedicated workers only when CPU or memory runs short. Staging gets the same shape.
• Use a Talos Layer 2 virtual IP as the Kubernetes endpoint. Configure talosctl with all three control-plane addresses because the virtual IP depends on etcd quorum.
• Label every node with its physical Proxmox host. Use those labels for workload and Longhorn replica placement.
• Add a dedicated Longhorn data disk and Talos user volume to every node. Build the Talos image with siderolabs/iscsi-tools and siderolabs/util-linux-tools. Mount the user volume at /var/mnt/longhorn, bind it into kubelet as /var/mnt/longhorn, and give the Longhorn namespace its required privileged Pod Security label.
• Create a production inventory, Flux instance, deploy key, and cluster path.
• Add bootstrap inputs for the Flux Git source, revision, and path, and one that starts the cluster with the applications suspended. See Talos Cilium and Flux bootstrap.
• Keep Bitwarden, the Terraform state backend, B2, TrueNAS, and the credentials needed to reach those systems available when Kubernetes is down.
• Schedule every backup in the ADR 006 backup table. A Proxmox disk with backup = true is only eligible for backup. It does not create a backup job.
• Complete a staging Talos upgrade, Kubernetes upgrade, node replacement, etcd restore, OpenBao restore, Longhorn restore, CloudNativePG point-in-time recovery, and representative workload restore.
• Expand staging to three control-plane nodes before you test normal control-plane replacement. Replacing its current single control plane is an etcd disaster-recovery exercise, not a healthy rolling replacement.
• Deploy Talos-compatible monitoring and the alerts linked from the alert runbook index.

Operator access

Use the client configurations that Terraform generates from state. They contain sensitive credentials. Do not commit them or leave them in a shared directory.

cd terraform
umask 077
tofu output -raw talosconfig > /tmp/talosconfig
tofu output -raw talos_kubeconfig > /tmp/talos-kubeconfig
export TALOSCONFIG=/tmp/talosconfig
export KUBECONFIG=/tmp/talos-kubeconfig

After the operation, remove the files:

rm -f /tmp/talosconfig /tmp/talos-kubeconfig
unset TALOSCONFIG KUBECONFIG

Standard health gate

Run this before and after every lifecycle operation. Stop if etcd has no leader, a control-plane service is unhealthy, any node is already unavailable, Flux is stalled, or a critical workload cannot tolerate a drain.

talosctl version
talosctl health
talosctl service etcd --nodes <control-plane-ip>
talosctl etcd status --nodes <control-plane-ip-1>,<control-plane-ip-2>,<control-plane-ip-3>
kubectl get nodes -o wide
kubectl get pods -A
kubectl get kustomizations,helmreleases -A

For the one-control-plane staging cluster, pass only its address to etcd commands. Production commands must list every control-plane address.

Routine changes

Change machine configuration

1. Change only declared Terraform inputs or Talos patches.
2. Run tofu fmt -check -recursive and tofu validate in terraform/.
3. Run tofu plan. Inspect every Talos and Proxmox change.
4. If talos_machine_secrets.cluster is replaced, stop. Replacing cluster secrets is a cluster migration, not routine maintenance.
5. Apply in staging and run the standard health gate.
6. Promote the same reviewed change to production during a maintenance window.

Never edit the machine-configuration snippets on Proxmox. They are sensitive Terraform outputs, not another source of truth.

Add a node

Add a unique entry to the production talos_nodes inventory. Apply the reviewed plan. Terraform creates the VM and applies a machine configuration derived from the existing cluster secrets. A new worker or control-plane node joins automatically. Do not run talosctl bootstrap for an additional node.

For a new control-plane node, wait for it to become a voting etcd member. Before you add another node, wait for all members to report the same leader.

Remove a node

Do not remove an inventory entry first. While Terraform still has the node address and credentials, drain and reset the node:

talosctl reset --nodes <node-ip>
kubectl delete node <node-name>

The reset drains the node and leaves etcd when applicable. After cluster health is restored, remove the inventory entry and apply Terraform. Never remove a control-plane node if the remaining members cannot retain quorum.

Upgrade Talos

Upgrade one adjacent Talos minor release at a time. Use the latest patch in each minor. Use a talosctl client that matches the currently running Talos version. Talos upgrades do not upgrade Kubernetes.

1. Review the Talos release notes, support matrix, Cilium compatibility, and system-extension compatibility.
2. Test the exact target in staging.
3. Take and copy an etcd snapshot off-cluster. Use disaster recovery.
4. Keep Terraform on the running version during the rolling operation. Derive the target installer from the declared Image Factory schematic. Do not apply target-version machine configuration to every node:

cd terraform
export TARGET_TALOS_VERSION="<target-talos-version>"
export TALOS_SCHEMATIC_ID="$(tofu output -raw talos_image_factory_schematic_id)"
export TALOS_INSTALLER_IMAGE="factory.talos.dev/nocloud-installer/${TALOS_SCHEMATIC_ID}:${TARGET_TALOS_VERSION}"

1. Preserve the installer platform as well as its version and schematic. These Proxmox VMs use nocloud-installer, not generic installer. A generic image can rename eth0 to ens18, invalidating the existing LinkConfig and removing IPv4 connectivity. For a same-version extension change, use the exact Terraform installer output for that node class. Longhorn workers use talos_longhorn_installer_image rather than the default schematic.

Upgrade workers one at a time. Run the health gate after each node:

talosctl upgrade --nodes <worker-ip> --image "$TALOS_INSTALLER_IMAGE" --wait

1. Upgrade control-plane nodes one at a time. Confirm etcd quorum before and after each node:

talosctl upgrade --nodes <control-plane-ip> --image "$TALOS_INSTALLER_IMAGE" --wait

1. Confirm that all nodes run the target version, Cilium is healthy, Flux is ready, and application probes pass.
2. Update talos_version to the running target version. Download the matching ISO from Image Factory. Verify its checksum. Upload it to the static Proxmox file ID declared by every node:

https://factory.talos.dev/image/<schematic-id>/<target-version>/nocloud-amd64.iso

Terraform currently derives this URL but does not upload or verify the ISO. Before you apply Terraform, confirm that the artifact at isos:iso/talos-nocloud-amd64.iso matches. 9. Run Terraform plan. If it replaces cluster secrets or VMs, do not continue. Apply only after every node already runs the declared Talos version.

If an upgraded node rejoins but workloads fail, use talosctl rollback --nodes <node-ip>. Investigate the failure. Do not update the Terraform version declaration until the incident is resolved. If you already changed it, revert that change before the next apply. Talos also attempts an automatic A/B rollback when the new image cannot boot. Do not roll back multiple control-plane nodes at once.

Upgrade Kubernetes

Use talosctl upgrade-k8s to upgrade Kubernetes separately. Upgrade only one minor version at a time. Verify that the running Talos version supports the target Kubernetes version.

1. Complete the standard health gate and take an etcd snapshot.
2. Review removed/deprecated APIs and test the target in staging.
3. Set the CLI target without the leading v. Terraform uses the same version with a leading v. Run the dry run against one healthy control-plane node:

talosctl upgrade-k8s --nodes <control-plane-ip> --to <target-kubernetes-version> --dry-run

1. Run the upgrade. The command updates the whole cluster. It is safe to restart after a failure:

talosctl upgrade-k8s --nodes <control-plane-ip> --to <target-kubernetes-version>

1. Update talos_kubernetes_version to the same version with a leading v. Review the Terraform plan for accidental downgrades. Then apply it to remove drift.
2. Run the health gate and verify every node reports the target version:

kubectl get nodes -o wide

Kubernetes downgrade is not a normal rollback. Prefer fixing forward. If the control plane is unusable, follow the tested etcd recovery path.

Maintenance cadence

Record the commit, Terraform plan, snapshot identifier, start/end time, and validation result for every production lifecycle operation.

References

• Talos production clusters
• Upgrading Talos
• Upgrading Kubernetes
• Talos Cilium and Flux bootstrap
• ADR 007: Run Talos as three hyperconverged nodes on three hosts
• ADR 008: Use Longhorn for persistent storage

docs/runbooks/talos-lifecycle.md