AI agent storage incident

Codex SQLite WAL Disk Full On Linux

If `~/.codex/logs_2.sqlite-wal` grows to GBs or hundreds of GBs, deleting the visible WAL may not free space. Stale or suspended Codex TUI sessions can keep deleted WAL file descriptors open, so `du ~/.codex` looks small while `df -h` still says the filesystem is full.

What is happening

SQLite WAL files are normal, but they need checkpointing. Long-lived readers can prevent truncation. If a stale Codex process still holds a deleted `logs_2.sqlite-wal`, the directory size may look fixed while the filesystem remains full because the deleted inode is still allocated.

Safe recovery order

1. Stop starting new Codex sessions while the filesystem is near full.
2. Use `lsof +L1` to prove whether deleted WAL or SHM files are still open.
3. Identify stale or suspended Codex processes with `ps`, `tmux ls`, and `fuser`.
4. Exit stale sessions cleanly if possible; otherwise terminate only the stale PIDs that hold the deleted WAL.
5. Run a SQLite checkpoint only after readers release the database.
6. Re-check `df -h`, `du -xsh ~/.codex`, and `lsof +L1` before deleting anything else.

du -xsh "$HOME/.codex" 2>/dev/null
df -h "$HOME"
lsof -nP +L1 2>/dev/null | awk 'NR>1 && $7 ~ /^[0-9]+$/ && $7 > 1000000000 {print $7, $2, $1, $4, $9}' | sort -nr | head -40 | numfmt --field=1 --to=iec --suffix=B
fuser "$HOME/.codex/logs_2.sqlite" 2>/dev/null
sqlite3 "$HOME/.codex/logs_2.sqlite" "PRAGMA wal_checkpoint(TRUNCATE);"

Do Not Delete First

• Do not keep deleting visible WAL files if `lsof +L1` shows deleted inodes are still open.
• Do not remove all of `~/.codex` before exporting or backing up session state you care about.
• Do not run broad cache cleaners while the issue is actually a live file descriptor problem.
• Do not kill every shell or tmux process; target the stale Codex PIDs that hold the deleted WAL.