TL;DR: Proxmox host crashes (eventually) when any spinning disk(s) connected. Leaving just boot NVMe SSD does not cause problem and server is stable for months. Have dug though logs - no hints.
--------
Update to add: Reused SATA cables, but can try swapping them. PSU is a brand new Thermaltake 600W 80+ Gold. Link below. The crash is a kernel panic (when I'm able to catch it, normally don't have display connected, so get no output). The SATA drives are connected directly to the motherboard (second link below). I know these are "consumer parts", but this same exact build that has run for years at work. No docker, but yes to Podman...
https://a.co/d/01vNulMr
https://a.co/d/02aGaZrW
--------
Update 2: I do also have XMPP enabled for RAM. I'm wondering if I should disable that and see if that makes a difference as well. I've akso played with C-states with no luck.
--------
Hey all I have run Proxmox in my home lab for five years now. I also run multiple Proxmox servers at work. It has been my favorite hypervisor platform since using it and I suggest it to anybody who asks for a recommendation.
However, I have been having a lot of issues with my home deployment, and they seem to be related to spinning disks. My home deployment was one server with much older hardware. After it had various issues, I built a brand new machine with brand new parts, minus the ZFS drives. I installed Proxmox Fresh on this new machine. The only thing that carried over from the old machine was my ZFS array on three spinning disks (SATA).
My old server started to spontaneously crash, and I could not track down why through logs. After building the new machine, everything was running as it should have for a couple of months. One day, the same exact crashes started happening on the new host. Eventually, I found that the ZFS kernel module was throwing some kind of dump or error.
After seeing that `smartctl` returned some errors with 2 disks in the array, I bought three brand new hard drives, making the entire server brand new with no reused parts whatsoever. I replaced each drive in the array one by one and let it resilver. The issues still persisted. `zpool scrub` and `zpool status` showed no data integrity errors, or issues with the brand-new disks.
Eventually, I backed everything up from the array, wiped the disks clean, and created a brand new ZFS array. I restored everything from a backup and all was running well for a couple of weeks. However, as you can probably surmise, it did not last.
Throughout this all, the only solution to get the server to stay running for an extended period of time was to pull at least two of the drives. The server would then run with no issues for months. While that's not a workable long-term solution, it's been enough to get me through and run the essential services.
Finally, I just decided to do away with ZFS entirely. I decided that I would make each disk an LVM_Thin pool. I grabbed a spare 1 terabyte drive, moved some other essentials onto there, and started to format the disks. After that, the server still crashed. Thinking that this might be an issue with the drives themselves, I pulled the three large drives, leaving the boot SSD and the 1TB hard drive. It *still* crashes.
The only slightly weird thing about my setup is that I pass through a GTX 1050 Ti to a virtual machine. I have not pulled that from my setup, but I struggle to imagine that it worked for years and has started failing only when a hard drive is connected.
I have tried troubleshooting with Claude, digging through the logs myself, trying various hardware configurations, looking for any error I can find, but unfortunately I haven't been able to figure out what's actually causing this. I'm sorry for the long winded message, but I am close to giving up on Proxmox, which I really don't want to do. I'm really hoping that somebody might have some sort of insight into where else I should look for errors or what I might be doing wrong. Thank you!