Backup stale or failed
Impact
The newest backup for etcd, Terraform state, a virtual machine, a persistent volume, or an application is older than the max age for its row in the ADR 006 backup table. Each row alerts as soon as its max age is exceeded. No alert rule exists yet. Services may remain available, but the possible data-loss window grows until a verified backup succeeds.
Triage
- Use the OpenTelemetry alert labels and job logs to identify
backup_type, the target, the last successful timestamp, the schedule, the storage destination, and the error. - Confirm whether monitoring alone is stale or the backup object is missing.
- From a management path outside the cluster, check destination reachability, capacity, credentials, encryption, the clock, and the retention policy.
- Verify the newest object checksum and metadata without overwriting the object.
Decide what to do
- If the etcd backup failed but etcd is healthy, take a manual snapshot with the DR procedure. Copy it off-cluster, then repair the schedule.
- If the Terraform state backup failed, stop non-emergency Terraform applies until the versioned backend and independent copy are healthy.
- If the Proxmox backup failed, check PBS and the job. A VM backup does not replace etcd or application backups.
- If the PersistentVolume or application backup failed, freeze risky application changes and follow its service-specific restore documentation.
- If backup objects exist but no restore test is current, treat the backup as unproven and schedule a staging restore.
Before you delete old recovery points to make a failing backup pass, review the retention policy and confirm that at least one verified copy remains elsewhere.
Check the resolution
Resolve the alert only after a new backup completes, reaches the required failure domain, passes checksum or tool-native verification, and falls within its max age. Record the object identifier. If the backup format, credentials, or destination changed, schedule a restore test.