Service crash-loop caused by full loopback-mounted log volume and root-owned debug log
Symptom
A containerised API service entered a crash-loop and was absent from running services. The root disk was healthy at 65% utilisation, but a dedicated loopback-mounted log volume (172 MB) was at 100% capacity, with 0 bytes available. A single debug log file (168 MB, owned by root) consumed the entire volume, preventing the service from writing startup logs on every restart attempt.
Context
The log volume was a loopback-mounted filesystem (172 MB) mounted at a service-specific log directory. The service ran as an unprivileged user, but the debug log file was owned by root. Standard automated cleanup using a find command with a -mtime +7 filter freed zero space because all log files were less than 7 days old. A truncation attempt without elevated privileges failed with a Permission Denied error.
Root cause
The loopback-mounted log volume (172 MB) was completely filled by a single root-owned debug.log file (168 MB). Because the file was owned by root rather than the service user, standard cleanup commands failed without sudo. With no free space remaining, the service could not write startup logs and entered a crash-loop on every restart attempt.
Resolution
1. Truncated the debug.log file to 0 bytes using `sudo truncate -s 0`, freeing 168 MB and bringing the loopback-mounted volume to 1% used. 2. Restarted the service using `sudo systemctl restart`. The service was confirmed running after restart. 3. A daily cron job (`0 2 * * * find /opt/<service>/logs -type f -mtime +7 -delete`) had been added during the investigation to prevent gradual log accumulation.
What we learned
1. A -mtime +7 retention policy is insufficient for burst logging scenarios — consider tightening to -mtime +1 or adding a size-based cap (e.g. truncate if file exceeds 100 MB). 2. Log files written by privileged processes (root-owned) require sudo for cleanup — ensure the connector or operator account has appropriate sudoers entries for log management commands. 3. Idempotency guards on change execution can block re-execution even when a prior run had no real-world effect — always verify actual system state after any 'already_applied' response. 4. Propose changes with explicit asset bindings from the start to avoid policy enforcement rejections at execution time.
Ops Intel investigates incidents like this automatically and remembers every fix. See plans →