Restore degraded persistent storage
Impact
A PersistentVolume, storage node, replica, or backup target is degraded. Data may remain available with reduced redundancy. Do not replace or drain nodes if doing so could remove the last healthy copy.
Triage
kubectl get storageclasses,persistentvolumes,persistentvolumeclaims -A
kubectl get volumeattachments -A
kubectl get events -A --field-selector type=Warning
Inspect the CSI or storage controller in use, node placement, replica health, capacity, attachment state, backup target, and application logs. Identify every workload that uses the affected volume before you act on its node.
Decide what to do
- If an attachment or mount has an issue but replicas are healthy, repair the node, CNI, or CSI path without deleting the volume.
- If redundancy is reduced, stop voluntary disruption in the affected failure domain until replicas rebuild.
- If volume data is corrupt or lost, stop the application and use its tested restore procedure. Git and etcd do not restore PersistentVolume contents.
- If the backup is also stale, use Backup stale and escalate the data-loss risk.
Check the resolution
Resolve the alert only when all intended replicas and attachments are healthy, the application passes a write and read check, a current backup exists outside the failure domain, and storage warning events have stopped.