API Service Crash Due to Dedicated Log Volume Reaching 100% Capacity on a Loop-Device Filesystem
Symptom
A uvicorn-based API service was found completely down and not accepting connections. The service process was not running. CPU load was 0.08, memory usage was 21% of available RAM, and the root filesystem was at 68% — none of which indicated a problem. The dedicated log volume, mounted as a fixed-size loop-device filesystem, was at 100% capacity: 168 MB used of a 172 MB cap, with 0 bytes free.
Context
The service ran on an Ubuntu 24.04 instance. Its log output was directed to a dedicated filesystem mounted at a subdirectory under the application path. This filesystem was implemented as a fixed-size loop-device volume with a hard cap of 172 MB, entirely separate from the root filesystem.
Root cause
The dedicated loop-device filesystem used exclusively for application log storage filled to 100% capacity (168 MB used of a 172 MB fixed cap, 0 bytes free). With no free space available, the application could not write log output and crashed. Because this volume is independent of the root filesystem, standard disk-usage checks targeting the root mount point would not detect saturation of this mount.
Resolution
1. Truncated the primary debug log file to zero bytes using a privileged command (the file was root-owned, requiring sudo). 2. Deleted all other log files present in the log directory. 3. Restarted the application's systemd service. After remediation, the log volume was confirmed at 1% usage (24K used) and the service process was confirmed running on port 8090.
What we learned
A fixed-size loop-device filesystem fills independently of the root disk, making it invisible to monitoring that only checks the root mount point. Log rotation or periodic truncation must be configured for any such volume to prevent recurrence. Disk-usage alerting should include an explicit threshold alert (for example at 80% capacity) targeted at this specific mount point.
Ops Intel investigates incidents like this automatically and remembers every fix. See plans →