When Memory Runs Out — the OOM Killer
A program asks for memory the server doesn't have. Linux then picks a process — usually the biggest — and kills it: the OOM killer (out of memory). The program's own log says nothing; it's simply gone.
How much is there?
free -h
total used free shared buff/cache available
Mem: 2.0Gi 305Mi 1.4Gi 18Mi 307Mi 1.5Gi
Swap: 0B 0B 0B
available is what programs can still get. buff/cache is the kernel's cache of files — it gives that back when programs need it.
Who uses it?
ps aux --sort=-%mem | head -5
RSS is the memory a process really holds, in KiB; %MEM its share.
The kernel's log: dmesg
The OOM killer writes to the kernel's log. On Debian only root may read it:
sudo dmesg | grep -i "killed process"
[311204.551522] Out of memory: Killed process 2871 (python3) total-vm:2421460kB, anon-rss:1694320kB, file-rss:256kB, shmem-rss:0kB, UID:1000 pgtables:3644kB oom_score_adj:0
anon-rss is what it held: about 1.6 GiB, on a 2 GB server. The number in brackets is seconds since boot — -T shows the time instead:
sudo dmesg -T | grep -i "out of memory"
The journal keeps the same messages: journalctl -k (k for kernel), and journalctl -k -p err only the errors.
Watching it
watch -n 5 free -h
watch runs a command again and again (here every 5 seconds) on a clear screen, until Ctrl+C.
What helps
| Fix | How |
|---|---|
| Use less | fewer workers, smaller caches — in the program's config |
| A safety net | swap (the Swap lesson) |
| A limit for one service | MemoryMax=1G in its unit: then that service alone is killed, nothing else |
| More memory | a bigger server |
💡 A service the OOM killer ended says so:
systemctl statusshowsResult: oom-kill.