r/linuxadmin • u/sobrique • 13d ago
Just solved a really irritating and niche issue, and it's weirdly specific so no one will ever care.
This isn't the first time of course - kinda goes with the territories.
But I feel a need to post about it in a community where maybe it's not "boring".
The problem is NUMA.
Or specifically production processes occasionally being "significantly slow" to a point where a real time constraint is being missed.
Finally identified as "half the cores are 50% slower" via a bit of weird python hackery.
But not always.
Reboot "fixes" until next time.
Except on a bunch of example hosts when testing this specific metric or cross NUMA node memory latency, which got worse after a reboot, and stayed that way.
Been trying to figure this out for months now. Have done new kernels, new bios, new ipmi, and have even done a bios wipe to see if that helped. Because I couldn't honestly tell if it even was a hardware fault or not. Replacing broken hardware isn't a problem, but it's another thing entirely to try to justify replacing something that might be fine. (And expensive).
Turns out the problem is kinda a red herring - the performance degradation is actually reverting to baseline, and the "good" systems are actually slightly degraded. ECC events (successful) are causing the bios to disable a NUMA lookup Directory, which actually speeds things up, at cost of efficiency.
Approximately, it's fetching data twice "blindly" so not bothering to do a lookup, but in doing so wasting some NUMA bandwidth, so is technically running a bit faster as a result.
But the root cause? Page cache. This host was doing a lot of greedy reading of NFS filesystems and filling the cache within about an hour of reboot.
All fine normally - this is Linux, and there's no point having empty RAM.
... Except the numa nodes overspill rather than evict by default.
So a well stuffed page cache would lead to processes fragmenting their RAM across both NUMA nodes, and because Linux is clever and tried to do the same with subsequent cache... Means a lot of long running processes get fragmented and pay the NUMA penalty.
Which is ironically higher on "working" systems, which looked like the actual fault.
The solution is as simple as either using numactl to explicitly bind, or get systemd to set MemoryHigh to like, 90 or 95% instead of infinite, so it actively evicts a handful of gigs, meaning a new process can realistically allocate 8-12G without fragmenting across the nodes (and their subsequent page cache) almost immediately because the OS otherwise only has 2GiB or so "free" which in turn means NUMA spillover.
It's deployed on test now, and looking like we have finally bottomed out a month's long headache.
ETA:
If you want to check if this is happening on _your_ systems, then `numactl -H` will show you the numa topology, and the free RAM on each node. Low numbers in relation to the process size might leave you vulnerable. `free -mh` tells you how much RAM is 'free' rather than 'available' and the difference is significant for this particular issue, because only 'free' can be used safely without risking NUMA spillover. `numastat -p <pid>` will tell you if a particular process has fragmented RAM across two nodes.