How can I resolve a back-off error in the kotsadm pod?

Issue Copied

When the kotsadm console fails to start, the kotsadm pod may enter CrashLoopBackOff. You may see errors like the following.

Cluster status warnings:

error: kotsadm-minio PVC using more than 80% of storage
error: kotsadm-rqlite PVC using more than 80% of storage
error: Pod itrs-analytics/kotsadm-7876f4db54-jhknq status. Status: CrashLoopBackOff

In the kotsadm-rqlite logs:

[raft] 2025/04/11 19:12:06 [ERROR] failed to take snapshot: error="failed to start snapshot: open transaction at end of WAL file"
[store] 2025/04/11 19:12:06 failed to generate user-requested snapshot: failed to start snapshot: open transaction at end of WAL file
[store] 2025/04/11 19:12:06 failed to snapshot due to WAL threshold: failed to start snapshot: open transaction at end of WAL file

These errors indicate that the kotsadm-minio or kotsadm-rqlite PVCs are nearly full or more than 80% has been used. When rqlite cannot write to disk, snapshot and WAL operations fail, which prevents the kotsadm pod from starting.

Resolution Copied

Resize the affected PVC (kotsadm-minio, kotsadm-rqlite, or both). See Resizing Persistent Volumes using Kubernetes for guidance.

After the PVC is resized, the kotsadm pod should recover automatically once rqlite can write to disk again.