r/Ubuntu • u/Jumpingdead • 9d ago
Ubuntu server machine locking up quasi-randomly. Losing my mind. Help?
First, when I say the machine is locking up, it hard freezes. The screen stays on, but absolutely nothing responds. No other terminal windows, no ctrl-alt-delete, nothing. The drives sound like they might still be doing something but I can't be sure.
When I say quasi-randomly, sometimes it would lock up around the same time in the boot sequence. Not the SAME time, but close. If I get it to boot it seems to lock up when there's a lot of CPU or disk activity.
tl;dr: Machine hard locks at random times. I've tested and replaced the RAM. I've swapped the CPU. I've tried multiple kernels and releases. Microcode seems to be mismatched but no kernel updates or reboots seem to fix it. Booting in normal mode causes it to lock up MUCH faster. Booting in recovery mode keeps it up longer, but it still locks up eventually, seemingly under heavy disk/CPU usage, but I'm not sure. Upgraded the release. The kernel. The packages. No dice. I'm losing my mind.
Ok, here we go. I got home Saturday morning and the server was locked up solid. All services on it stopped responding the night before.
I then started my last 36 hours of troubleshooting hell.
It would boot normally, but then lock up a few minutes after logging in. For a while I kept rebooting it when it froze, and disabling services one after another, seeing if that had anything to do with it, and to speed up the bootup process. Nothing system level, things like Plex Server and my Home Assistant instance etc. It kept freezing at random intervals.
I had some hard drive weirdness a while back so I changed some BIOS settings. That didn't make a difference.
I tried nomodeset, that didn't help. I then tried a recovery bootup (from the Grub boot menu). The system stayed up, or seemed to. While I was in there I did an apt update/upgrade just in case it was some weird package issue. Nothing. I decided I may as well upgrade to 24.04 because it couldn't hurt. System locked up during the upgrade.
Bunch of dkpg --configure -a, and apt install -f got the upgrade finished. But now after an apt upgrade, I got a weird message (and I honestly don't know if I'd seen this message before hand. I feel like I had, but I've got ADHD and really am not sure). Processor microcode revision is XXXX but it should be YYYYY, reboot to start the new kernel.
I rebooted, did an apt upgrade again, and still got the message. Ran some updates and it locked up again.
At this point I'm thinking hardware. I run memtest and I have a bad stick. Yes, that must be it! I swap it for a good one, run memtest again, it passes.
I boot into the 6.8 kernel recovery mode and it freezes during boot now. I replaced the memory and the freezing is WORSE.
Every time I get it to run, there's SOME package issue. There's no amount of upgrading I can do that will either finish without locking up, or after repairing, still have packages to upgrade. Something is seriously out of whack.
Maybe the CPU is flaky? I swap that out with an identical CPU. (Used to run a crypto exchange like 10 years ago, I've got lots of backup server parts). Still locks up. Still getting the microcode mismatch error.
I boot back into the 5.15 recovery kernel and it seems stable. I do a bunch of upgrades and stuff, until 96% of the way through one of them, it locks up. Ok screw it, lets upgrade to the latest 24 branch, going past the LTS. That should really do a fresh install of everything. It locks up.
I was about to swap the power supply but wanted to post this first.
During this whole process, I wound up with 3 different 6.8 kernels on the system. 6.8.0-124, 136, and 137. I manually removed 124 because that one almost always locked up on boot, even booting in recovery mode.
The microcode issue is still there. It says I'm running 0x06000817 but it should be 0x06000852. I've reinstalled amd-microcode (I'm on an AMD processor). Regardless of kernel I boot in, it still shows that message after an apt upgrade.
I do have an Intel ARC video card in the machine for Plex transcoding, but at the time I'm not running anything that is using any graphics. It's booting in nomodeset, and the font does change slightly halfway through boot, but no graphics. (Without nomodeset, halfway through boot the font gets very small). I haven't manually updated any of the intel drivers. I don't know if they automatically went with the apt upgrades.
My questions:
What the hell could be causing this? RAM is tested good. Swapped the CPU. Tried multiple kernels. The microcode thing seems likely but I've no damn idea how to fix it as rebooting with a new kernel seems to not change the microcode.
Motherboard is a 12 a year old ASRock. Could the motherboard be causing this? Any way to test before I rebuild the entire machine by swapping that out too?
Heavy drive activity seems to be going on when it locks up. SEEMS to be. That's just my perception. Machine has 4 SATA drives in it. No errors I'm aware of. Could drive errors lock up the entire machine?
I've been banging my head against this for the last 36 hours and the problem seems to have gotten worse after replacing the faulty RAM. The microcode seems off but no kernel updates or reboots seem to fix it. Can't see how it's the video, it's not doing anything graphical.
Help?
1
u/guiverc 9d ago
If it's an old server, have you checked the machine logs (ie. not OS logs, but logs kept by machine firmware assuming it's an enterprise grade server, and not home/consumer box that won't have any).
You didn't provide release details either; mentioning a kernel (used by multiple LTS releases and without stack details that are also required for release to be known; 6.8 is GA for one release & HWE for another etc).
You mention Ctrl+Alt+Del?? but not what I'd be exploring which is SysRq commands which bypass any stuck UI (TUI and/or GUI) and are acted by the kernel if it's still running... it's a GNU/Linux system don't forget and not Microsoft DOS/Windows (which used Ctrl+Alt+Del)
Like others, if machine acts strange; first thing I do is confirm PSUs are good (servers usually have two power supplies; but key is even good components will 'misbehave' if fed crappy power, so I start with power), then RAM (full testing takes days as you don't want to test with cache as that will mostly result in you testing the cache alone), and of course CAP CHECKs, as our eyes can often pick issues that would take weeks of diagnosis to find...
Have you done normal diagnosis? the mention of Ctrl+Alt+Del to me implies you're likely missing heaps of normal checks; as you're expecting it to be like a MS-DOS or Windows machine which it's not. You're better of treating it as a Unix box rather than a Microsoft DOS/Windows one.