Use Longhorn for persistent storage

Context

The Talos cluster needs persistent block storage that survives one Proxmox host failure. The current k3s platform already uses Longhorn. Rook with Ceph would add a second storage system to operate, NFS would move availability to one server, and the Proxmox CSI driver lacks a tested cluster-loss recovery path.

Longhorn replicas protect service availability. They do not replace an off-cluster backup.

Decision

Use Longhorn on the three control-plane nodes from ADR 007. Give every node a dedicated data disk and a Talos user volume for the Longhorn data path. Build the Talos image with the siderolabs/iscsi-tools and siderolabs/util-linux-tools extensions before installing Longhorn.

Mount the user volume at /var/mnt/longhorn. Add a shared, read-write kubelet bind mount from /var/mnt/longhorn to /var/mnt/longhorn, and configure Longhorn to use /var/mnt/longhorn. Label the Longhorn namespace as privileged under Pod Security Admission. These permissions apply only to the storage system namespace.

Use the same path in the host, kubelet, and Longhorn configuration. A kubelet-only path alias is insufficient: Longhorn host-side disk checks can resolve that alias on the OS filesystem instead of the dedicated disk. Before importing data, verify Longhorn disk status reports the dedicated filesystem and capacity, not merely that its path exists.

Map Longhorn nodes to physical Proxmox hosts. Keep replica node soft anti-affinity disabled so that Longhorn does not place two replicas of one volume in the same failure domain.

Provide two storage classes:

• longhorn is the default. It keeps two replicas.
• strict-local keeps one replica on the node that runs the pod. Use it only for an application that replicates its own data and has a tested drain procedure.

Before a node drain, an application on strict-local must move or rebuild its member on another node, because its volume cannot follow the pod. Write that procedure and test it on staging before the application uses strict-local in production.

Set Longhorn's node-down-pod-deletion-policy, so that pods with Longhorn volumes on a failed node restart on a healthy node without manual work.

Run OpenBao as three Raft members on strict-local volumes. Pod anti-affinity must place the three members on different nodes. OpenBao Raft owns replication for this data. Three application replicas on replicated volumes would multiply the copies without increasing Raft quorum.

Send recurring Longhorn backups directly to a dedicated B2 store. Longhorn owns backup retention. Do not apply B2 lifecycle deletion or Object Lock to the active Longhorn backup store.

Send CloudNativePG base backups and WAL archives directly to a separate B2 store through the Barman Cloud plugin. The plugin owns retention and the recovery window.

Use TrueNAS for local virtual-machine backups, ZFS snapshots, and complete artifacts. Do not use TrueNAS Cloud Sync to copy a live Longhorn or Barman backup store to B2.

Implementation status

Staging runs Longhorn on its worker with the longhorn-staging-r1-retain class. Terraform still allows Longhorn only on workers. The two classes, the node-down policy, and the B2 backups are future work.

Consequences

The platform keeps a familiar storage system and a documented B2 restore path. Longhorn adds synchronous writes, replica rebuild traffic, and capacity cost on top of the Proxmox ZFS layer. On hyperconverged nodes, rebuild traffic competes with workloads and etcd for the same disks and network.

Two replicas survive one host failure. A second failure before the rebuild finishes loses the volume. Recovery then comes from the B2 backup, and writes since the last backup are lost.

The strict-local OpenBao volumes cannot move with a failed member. If one member and its volume are lost, remove the failed Raft peer and create a new member. Restore the whole OpenBao cluster from a Raft snapshot only after quorum is lost.

The existing longhorn-1-replica storage class is not part of the target design. Its name says one replica while its configuration requests two. Migrate each workload to a correctly named class instead of changing existing volume semantics in place.

References

• Longhorn support for Talos
• Longhorn backup targets
• Longhorn node-down pod deletion policy
• CloudNativePG backup
• ADR 006: Backups and rebuild

Rollout gates

• Longhorn disk status on every node reports the dedicated filesystem and capacity. open
• The longhorn and strict-local classes exist, and longhorn is the default. open
• On staging, the node-down policy has passed a power-off test and a network-cut test. open
• On staging, the drain procedure has passed for every strict-local application. open
• Scheduled Longhorn backups reach B2, and one volume restore has passed. open

docs/decisions/008-longhorn-storage.md