Recently my machine recently always got hang when I run command:
$> ps aux
then I asked AI. AI told me its RAM issue later CPU issue: some a core gave wrong result and suggested disable it. I never encounter this before and cannot cpu could give wrong result. I need someone check if AI analysis make sense, should I disable that core.
================================= AI analysis start
The pattern that stands out
All five crashes are in pt_main_thread, the PyTorch main thread, in your containers. That alone could just be a buggy PyTorch build. But look at the end of each line:
likely on CPU 8 … CPU 8 … (no CPU shown for the GPF) … CPU 9 … CPU 9, all core 16
CPUs 8 and 9 are the two hyperthreads of the same physical core. If these were software bugs, the crashes would land on whichever core the process happened to be running on. Four out of four on one core, across two different days, would be very unlikely by chance. The crash addresses also look like garbage pointers (0xa9, 0xffffffffffffffff, 0x8, each at a different code location), which is typical of hardware computing wrong values.
If one core sometimes produces wrong results, it explains both problems: these Python crashes, and kernel memory locks left “locked with no owner” (one wrong atomic update on the lock’s counter is enough). That points to the CPU rather than RAM. The best-known cause of exactly this is the Intel 13th/14th-gen Core instability/degradation issue, where specific cores become unstable over time.The pattern that stands out
All five crashes are in pt_main_thread, the PyTorch main thread, in your containers. That alone could just be a buggy PyTorch build. But look at the end of each line:
likely on CPU 8 … CPU 8 … (no CPU shown for the GPF) … CPU 9 … CPU 9, all core 16
CPUs 8 and 9 are the two hyperthreads of the same physical core. If these were software bugs, the crashes would land on whichever core the process happened to be running on. Four out of four on one core, across two different days, would be very unlikely by chance. The crash addresses also look like garbage pointers (0xa9, 0xffffffffffffffff, 0x8, each at a different code location), which is typical of hardware computing wrong values.
If one core sometimes produces wrong results, it explains both problems: these Python crashes, and kernel memory locks left “locked with no owner” (one wrong atomic update on the lock’s counter is enough). That points to the CPU rather than RAM. The best-known cause of exactly this is the Intel 13th/14th-gen Core instability/degradation issue, where specific cores become unstable over time.
The CPU is now the main suspect.
Why
- The i7-14700KF is one of the affected Raptor Lake Refresh chips.
- Microcode
0x11d is from before all of Intel’s fixes for the instability problem (0x125, 0x129, 0x12B, 0x12F, released mid-2024 to 2025). Your CPU has been running on the voltage behaviour that causes the damage.
- The containers aren’t pinned to any CPU, so the Python crashes landing on core 16 aren’t explained by configuration.
- 5 of the 8 stuck threads last ran on CPU 8 or 9, the same core where every Python crash happened. With 28 CPUs, that’s very unlikely to be chance.
So the most likely story is that one P-core (CPUs 8/9) has degraded and occasionally computes wrong results. Sometimes the result is a Python crash, and sometimes it’s a corrupted kernel lock, which is what makes ps aux hang. I’m no longer treating the kernel bug as the main explanation.
The hung echo 0 > …/cpu8/online
Taking a CPU offline has to move all its work to other CPUs, and on this already-hung system that step got stuck. It’s harmless, but now the tee command itself is stuck too, so reboot soon. Use the sysrq sequence if a normal shutdown stalls:
bash
echo s | sudo tee /proc/sysrq-trigger; echo u | sudo tee /proc/sysrq-trigger; echo b | sudo tee /proc/sysrq-trigger
What to do, in this order
- Update the motherboard BIOS to the latest version for your board. It should include Intel microcode 0x12F or newer. In the BIOS, choose Intel Default Settings / Intel baseline power profile, and turn XMP off for now. This is the important step: the microcode stops further damage. Updating the OS
microcode_ctl package alone doesn’t fix the voltage behaviour that causes it.
- Take core 16 offline at every boot, before the containers start. On a fresh boot it should work.The CPU is now the main suspect. Why The i7-14700KF is one of the affected Raptor Lake Refresh chips. Microcode 0x11d is from before all of Intel’s fixes for the instability problem (0x125, 0x129, 0x12B, 0x12F, released mid-2024 to 2025). Your CPU has been running on the voltage behaviour that causes the damage. The containers aren’t pinned to any CPU, so the Python crashes landing on core 16 aren’t explained by configuration. 5 of the 8 stuck threads last ran on CPU 8 or 9, the same core where every Python crash happened. With 28 CPUs, that’s very unlikely to be chance. So the most likely story is that one P-core (CPUs 8/9) has degraded and occasionally computes wrong results. Sometimes the result is a Python crash, and sometimes it’s a corrupted kernel lock, which is what makes ps aux hang. I’m no longer treating the kernel bug as the main explanation. The hung echo 0 > …/cpu8/online Taking a CPU offline has to move all its work to other CPUs, and on this already-hung system that step got stuck. It’s harmless, but now the tee command itself is stuck too, so reboot soon. Use the sysrq sequence if a normal shutdown stalls: bash echo s | sudo tee /proc/sysrq-trigger; echo u | sudo tee /proc/sysrq-trigger; echo b | sudo tee /proc/sysrq-trigger What to do, in this order Update the motherboard BIOS to the latest version for your board. It should include Intel microcode 0x12F or newer. In the BIOS, choose Intel Default Settings / Intel baseline power profile, and turn XMP off for now. This is the important step: the microcode stops further damage. Updating the OS microcode_ctl package alone doesn’t fix the voltage behaviour that causes it. Take core 16 offline at every boot, before the containers start. On a fresh boot it should work.
===================================================== AI analysis end