Restore a silent monitoring pipeline
Impact
Metrics, logs, alerts, or the dead-man signal are missing. Other incidents may go undetected. Treat monitoring silence as a loss of safety controls, not a cosmetic failure.
Triage
From outside the cluster, confirm that Alertmanager, Mimir, Loki, Grafana, and the external probe path are reachable. Compare the last ingested sample and log timestamps with the alert start. In Kubernetes, inspect monitoring collectors, scrape targets, remote-write errors, queues, certificates, DNS, and network policies.
Decide what to do
- If the cluster is healthy and only ingestion failed, repair credentials, the destination, capacity, or the network. Do not restart every collector at once.
- If the Kubernetes API is also unavailable, follow Kubernetes API unavailable.
- If Cilium or DNS prevents egress, follow Cilium unhealthy.
- If alert routing alone failed, verify the Alertmanager configuration and each receiver with a controlled test alert.
Check the resolution
Resolve the alert only when fresh metrics and logs arrive with current timestamps, the dead-man signal is active, a controlled test alert reaches the expected receiver, and you have checked the outage window for missed alerts.