Alert runbooks
These runbooks define the first response for future Talos production alerts.
Alert rules must set runbook_url. The managed Alertmanager notification template
renders that annotation.
Use this repository URL prefix until the documentation site has a stable URL.
Example:
annotations:
summary: Kubernetes API is unavailable
runbook_url: https://git.local.isabelsoler.es/igresc/homelab/src/branch/main/docs/runbooks/alerts/kubernetes-api-unavailable.md
Runbook map
| Failure signal | Runbook |
|---|---|
| API endpoint or API server unavailable | Kubernetes API unavailable |
| etcd member, leader, quorum, or space alarm | etcd unhealthy |
| Talos machine or Kubernetes Node unavailable | Talos node unavailable |
| Cilium agent, operator, or dataplane unhealthy | Cilium unhealthy |
| Flux source, Kustomization, or HelmRelease stalled | Flux reconciliation stalled |
| etcd, state, VM, PV, or application backup stale | Backup stale |
| Alerting, metrics, logs, or dead-man signal silent | Monitoring pipeline silent |
| Proxmox host, datastore, or multiple VMs unavailable | Proxmox infrastructure unavailable |
| GCS state backend unavailable or inconsistent | Terraform backend unavailable |
| OpenBao sealed or unavailable | OpenBao unavailable |
| PersistentVolume or storage system degraded | Persistent storage degraded |
Alert contract
Every platform alert provides severity, cluster, component, and
runbook_url. Add node, namespace, name, and backup_type when relevant.
The alert expression retains the labels that its runbook uses.
Runbooks start with read-only diagnostics. For destructive action, use node replacement or disaster recovery. Those runbooks contain the safety gates.