Skip to content

ADR 006

Backups and rebuild

Git and Terraform rebuild configuration. Every other kind of data gets one backup method, a maximum age and a tested restore.

Status
Accepted
Rollout
0 of 5 gates
Related
004 008 009 010 012

Accepted, not yet implemented.

Context

Git and Terraform recreate configuration and virtual machines. Neither restores etcd, databases, volumes or secrets. Each of those has its own consistency and restore rules, so one backup tool can't protect them all safely.

The disaster this ADR plans for is cluster loss: every Talos VM and every Longhorn volume is unusable, while Proxmox, the network, TrueNAS, Bitwarden and the cloud services still work. Site loss has no recovery time target, but an encrypted offsite copy of all data must survive it. Losing the site and the GitHub account together is accepted. That case gets a slow, manual rebuild.

One operator runs the platform. A recovery path nobody runs goes stale, so recovery uses the same code path as a normal bootstrap, and every path gets a first test and a place in a yearly drill.

Decision

What is backed up

Each data type has one backup method and one freshness limit:

Data Backup Max age Retention Restore Status
Git repos Docker-host job writes a bare repo and dated bundles to TrueNAS over NFS; the existing TrueNAS job copies them to B2 24 hours 30 daily, 12 monthly Clone, or point Flux at the Docker host Job and runbook prepared, not running yet
Terraform state GCS versioning and soft delete, plus a daily age-encrypted copy to B2 when the serial changed 24 hours Latest copy kept indefinitely, older copies 90 days Pick a generation, or decrypt the copy Not automated
etcd Talos snapshot CronJob, age-encrypted, to B2 6 hours 30 days Only for control-plane failures. Rebuild from Git is the default. Manual only
OpenBao Raft snapshot, age-encrypted, to B2. Also after important key changes. 24 hours 30 days bao operator raft snapshot restore Manual only
Longhorn volumes Longhorn backup to B2 24 hours 7 daily, 4 weekly Restore while apps are suspended Not implemented
Postgres CNPG Barman to B2 Continuous WAL, daily base 14 days CNPG recovery bootstrap Not implemented

Container images are not backed up. CI rebuilds them from source.

ADR 010 covers the Terraform state backend and its daily B2 copy.

Longhorn and CNPG write directly to their own B2 stores, and their controllers own retention. Don't copy those stores through TrueNAS, and don't apply B2 lifecycle rules to them.

The etcd snapshot job runs in Kubernetes with the os:etcd:backup Talos role. It gets only a B2 write credential and the public age recipients. Restores use a separate B2 read credential and an age identity kept outside Kubernetes.

Velero is not installed. Talos, Git, Longhorn, CNPG and OpenBao already cover every data type. Reconsider it only when a workload has Kubernetes resources or mounted files that none of them cover.

Git backups

A daily systemd timer on the Docker host updates a bare repo and keeps dated bundles (30 daily, 12 monthly) for every repo. It writes to a dedicated TrueNAS dataset over NFS. The Talos installer image for the pinned version sits next to them. ZFS snapshots and the existing TrueNAS to B2 job cover the dataset. Credentials stay on the Docker host outside the backup tree. TrueNAS remains a NAS and runs no custom backup or Git-serving code.

The Docker host serves the homelab mirror over read-only SSH. The Flux deploy keys are authorized there, and its host key is in Flux's known hosts. The Flux source URL is a bootstrap input, set where Git reconciliation can't revert it, so a rebuild during a GitHub outage can point Flux at the Docker host. This endpoint needs both the Docker host and NAS. Bundles on B2 remain the fallback if either is lost.

Offsite freshness comes from the success of the TrueNAS to B2 job, not from the Docker host's fetch.

Alerts

Each row in the table has its own last-success signal and its own alert. The alert fires when the max age is exceeded, not after two missed intervals. Two missed intervals would let a daily backup go 48 hours unnoticed. Jobs that run outside the cluster also ping Uptime Kuma (ADR 004).

Default recovery

After cluster loss, rebuild from Git:

  1. Bootstrap the cluster from Git with the applications suspended. The suspension is a bootstrap input.
  2. Restore volumes, databases and OpenBao.
  3. Check the restored data and the matching secrets.
  4. Resume the applications.

There is no separate recovery mode and no write-inhibition control. Normal bootstrap and recovery bootstrap are the same path with one input changed.

This needs one suspension boundary that holds every application writer and none of the components needed to restore them. The suspension must hold from the first reconciliation, and Git reconciliation must not undo it.

etcd restores are only for control-plane failures, such as lost etcd quorum with a valid snapshot. They are not the default for full cluster loss.

Restore order

Restore order follows dependencies, not only the value of the data. A service that supplies platform prerequisites restores first, even when its own data is replaceable. The recovery tiers in the glossary don't override that order.

Encryption and recovery credentials

Encrypt every backup artifact directly to the same three age recipients as today: two YubiKeys and the offline key. Each one can decrypt on its own. Backup jobs get only the public recipients, listed in scripts/recovery/backup.recipients.txt, and the age YubiKey plugin. No intermediate backup identity exists, and no private key lives in Kubernetes.

Keep an old identity while retained archives still need it. Changing recipients only affects new artifacts.

Self-hosted Bitwarden holds the normal credentials. The recovery credentials also live in Bitwarden cloud and in one offsite bundle, so recovery works with the lab down. Retire the redundant local recovery copy once offsite recovery is verified on its own.

The offline age key stays disconnected, in secure physical custody, apart from the YubiKeys and every onsite system.

Tests

Run each of these once on staging:

  • A rehearsal: rebuild from Git with the old cluster unreachable and OpenBao restored from a snapshot, one Longhorn volume restore, one CNPG point-in-time restore, and an application write and read.
  • One etcd snapshot restore.
  • One drill with Garage stopped, including a ruler restart, that confirms a notification arrives.
  • One Flux reconcile from the Docker host's NFS-backed Git copy with GitHub blocked.

After that, run one drill a year. It rotates through a volume restore, a database restore, an etcd restore and the Docker host's Git source. Run the rehearsal again after any material change to recovery.

Consequences

  • Several small backup jobs instead of one tool. Each one fits the consistency rules of its data.
  • B2, GCS and Bitwarden cloud are production dependencies.
  • An upload is not proof. A restore test is.
  • No fixed recovery time is promised.
  • The Flux layout has to change before the default recovery works (see below).

Implementation status

The table's Status column shows each backup. The suspension boundary does not exist yet. fdm-monster-staging reconciles on its own (kubernetes/clusters/staging/fdm-monster.yaml), and OpenBao sits inside the production apps tree (kubernetes/apps/production/kustomization.yaml:28). Phase 3 of the migration builds the boundary and its bootstrap input. The BackupStale alert runbook exists, but no alert rule does yet.

References

Rollout gates

  • Every row of the backup table runs on a schedule and alerts on its max age. open
  • Each test in the Tests list has passed, plus one restore of a TrueNAS Git bundle and one restore of a B2 Terraform state copy. open
  • The suspension boundary exists, and a staging rebuild with OpenBao unreachable keeps the applications suspended from the first reconciliation. open
  • The staging rehearsal and the other first tests above have passed. open
  • Recovery credentials can be retrieved and an artifact decrypted using only Bitwarden cloud and the offsite bundle, with no onsite system. open

docs/decisions/006-layered-disaster-recovery.md