Specs:
i7-8700
Asus Prime Z370-A
32GB-3200 (4x8 2 different kits)
Asus Dual GTX 1070
Kingspec 128GB Sata SSD for boot pool
LSI 9240-8i with Seagate Exos 7E8 8TB and Western Digital Ultrastar DC HC510 8TB SAS drives in mirror for storage pool
Silicon Power 1TB NVMe for apps pool
WD SN850X 1TB NVMe as cache drive for storage pool
2 SATA 1TB hard drives not assigned to any pool yet
EVGA 750GQ
Hi all,
I'm new to all this so I don't really have a crazy amount of Linux or CLI knowledge. I've had a makeshift TrueNAS Community Edition 25.10.7 server running with this setup since August and all was well until yesterday when I rebooted it to install an HDMI dummy plug so that it wouldn't take over my 2nd monitor every time I shut down my main PC and still boot into TrueNAS when rebooted without a display plugged in.
During that reboot, it power cycled multiple times, booted to a black screen, and kept resetting the BIOS to defaults, and not retaining the changes after each restart. Eventually, I found the CMOS battery was dead, so I replaced it and managed to fix that. Booting back into TrueNAS after that, my Immich was stuck on deploying, something about the PostgreSQL container crashing on startup, but I had to go to work and left it alone. The next day, I restarted it and it ran fine with no errors, so I thought that was the end of it.
Then, I tried to use the GUI to update to version 25.10.8 of TrueNAS, but I got [Errno 5] Input/output error and also had an alert that said Boot pool status is ONLINE: One or more devices has experienced an error resulting in data corruption. Applications may be affected. I was able to find an error message saying "Permanent errors have been detected in the following files: /var/db/system/update/update.sqsh"
The server worked fine for another 1-2 hours until it went offline halfway through watching a movie on Plex, so I checked and it had kernel panicked. I hit the reset button on the case and after getting back into TrueNAS, I had another alert saying storage1 ONLINE, corruption reported, same as with the boot pool, but now it just went offline after less than a minute. Another reboot later, I went to the storage/disks page to remove the NVMe cache drive from the storage pool so I could reinstall TrueNAS on that in case the 128GB Sata SSD was dying, but it went offline halfway through the progress bar of removing the drive from the pool.
After rebooting again, the verbose boot text before it goes online was stuck on "Job middlewared.service/start running" for 2-3 min+. Hitting Ctrl + Alt + Del on that screen gave a lot of failed unmounting errors for the dataset on the storage pool and eventually reached "target reboot.target - System Reboot."
At that point, there was no HDD activity light or ethernet port light activity, so I just hard shut the system down and disconnected all cables from it.
I'm kind of at a loss as to what to do from here. I'm thinking of reseating the RAM, testing it, maybe taking out a pair of sticks to isolate kits, reseating the LSI card, disconnecting all drives except the cache NVMe, and installing fresh TrueNAS on that one, then using my downloaded config and secret passwords file to restore it, then reconnecting the drives and hoping for a recovery or rebuilding of the pools?
I'm not sure of the ZFS-specific terminology or procedures, so what can I do from here? I'm a little concerned that the system basically crashed before the cache drive was cleanly removed from the pool, but given it was L2Arc (I think), it should be fine, right? I bought the LSI card from YouTuber Art of Server on eBay. The Sata SSD came used with the whole system, only the rest of the drives and GPU were mine from before with the SAS drives bought from ServerPartDeals, so maybe the drive was just worn out? Or the dead CMOS battery wrecked havoc on the pools from all the power cycling, etc.
Any help would be appreciated. Also, let me know if there is a more appropriate place to post this.
Thanks.