memcg: bypass the reclaim and oom killer for dying tasks once oom_reaper is done
Linux
Affected versions
| Product | Affected | Fixed in |
|---|---|---|
| Linux Linux |
| No fix yet |
| Linux Linux |
|
|
Linux
| Product | Affected | Fixed in |
|---|---|---|
| Linux Linux |
| No fix yet |
| Linux Linux |
|
|
6.18.517.2.57.3In the Linux kernel, the following vulnerability has been resolved:
memcg: bypass the reclaim and oom killer for dying tasks once oom_reaper is done
At Meta, we are seeing instances where an OOM killed job is stuck in the exit path for several hours. In one particular case, the job was stuck for more than 8 hours and I had to manually remove the memory.max limits to allow the process to exit.
The job was a single process job and had ~55 GiB memory.max and zswap enabled. It had almost 0 anon in memory and ~111 GiB in zswap compressed to ~51 GiB zswap pool (i.e. almost all of memory.current was zswap). Nothing was left on the LRUs to reclaim.
On further inspection, I observed ~20k threads of that process stuck with the following stack:
[<0>] mem_cgroup_out_of_memory+0x4e/0xa0 [<0>] charge_memcg+0x8bf/0x990 [<0>] mem_cgroup_swapin_charge_folio+0x4e/0x80 [<0>] __read_swap_cache_async+0x10c/0x260 [<0>] swapin_readahead+0x116/0x3f0 [<0>] do_swap_page+0x13c/0x1ce0 [<0>] handle_mm_fault+0x61d/0x11f0 [<0>] do_user_addr_fault+0x3e7/0x6d0 [<0>] exc_page_fault+0x8f/0x110 [<0>] asm_exc_page_fault+0x22/0x30 [<0>] __get_user_8+0x14/0x20 [<0>] futex_cleanup+0x27/0x1c0 [<0>] futex_exit_release+0x47/0x60 [<0>] do_exit+0x107/0x940 [<0>] do_group_exit+0x81/0xa0 [<0>] get_signal+0x2b1/0x6e0 [<0>] arch_do_signal_or_restart+0x1a/0x1c0 [<0>] exit_to_user_mode_loop+0xa8/0x1c0 [<0>] do_syscall_64+0x152/0x250 [<0>] entry_SYSCALL_64_after_hwframe+0x4b/0x53
In addition the dmesg was filled with "Out of memory and no killable processes..." messages.
I have no idea why oom reaper was not able to reap/unmap the process. My guess is that since oom reaper tries to acquire mmap_lock in read mode limited number of times and then gives up, there might be a thread of that process which had mmap_lock in write mode at that time.
From the CVE record · Linux
The Linux kernel's memory-cgroup OOM handling can leave an OOM-killed process stuck while exiting. A local workload with many threads and memory-cgroup limits can encounter prolonged delays if the OOM reaper cannot unmap the process and exiting threads wait on the OOM lock. The input does not describe a remote attack or authentication requirements.
Engineers running Linux kernels with memory-cgroup limits should check their deployments; affected versions are not specified.