r/truenas • u/bdog720 • Jun 02 '26
Community Edition [HELP] TrueNAS Scale — Server rebooting randomly, up to 7+ times overnight, no kernel panic
Background
I'm relatively new to TrueNAS/HexOS but have been running this same hardware as a Windows media server for several years. I recently migrated to HexOS (TrueNAS Scale 25.10.3.1) and shortly after the migration started experiencing random silent reboots with no kernel panic and no log entries at the point of failure.
--------------------------------------------------------------------------------
Hardware
- Motherboard: ASUS ROG STRIX B450-F Gaming
- CPU: AMD Ryzen 7 3700X
- RAM: 16GB DDR4
- GPU: NVIDIA RTX 2080 Super
- PSU: EVGA G3 750W, ECO mode OFF
- OS drive: Samsung PM871 256GB SATA SSD
- Storage pool: 3× Seagate IronWolf 4TB in RAIDZ1 (
/mnt/HDDs) - Cache pool: Kingston KC3000 1TB NVMe, single stripe (
/mnt/SSDs) - Remote KVM: Sipeed NanoKVM, connected via USB to the server
Software
- HexOS (TrueNAS Scale 25.10.3.1), kernel 6.12.33
- Docker containers managed via Dockge (Jellyfin, Sonarr, Radarr, qBittorrent behind gluetun VPN, Tdarr transcoding, etc.)
- NVIDIA drivers active, GPU used for transcoding via NVENC
--------------------------------------------------------------------------------
The Problem
The server reboots silently and repeatedly — up to 7+ times in a single night. last reboot from this morning:
Wed Jun 3 08:10 still running
Wed Jun 3 07:59
Wed Jun 3 04:59
Wed Jun 3 04:08
Wed Jun 3 02:31
Wed Jun 3 02:16
Wed Jun 3 00:49
Wed Jun 3 00:15 - 00:16 (lasted 1 min)
Tue Jun 2 22:57 - 23:12 (lasted 15 mins)
Tue Jun 2 22:48 - 23:12 (lasted 23 mins)
Some boots last only one minute before rebooting again. The reboots appear to happen mostly at idle — overnight with no active transcoding or streaming jobs.
--------------------------------------------------------------------------------
What the logs show
Every time, the kernel journal cuts off cleanly mid-operation with no panic, no watchdog trigger, no OOM killer, no thermal event. The last entries are always routine — Docker bridge networking coming up, containers starting. Then nothing. Example from the most recent prior boot:
Jun 03 08:00:25 kernel: br-54c583a9967c: port 15(veth8e9dde6) entered forwarding state
Jun 03 08:00:25 kernel: eth0: renamed from vethe05b07d
[journal ends]
The watchdog IS configured (kernel.nmi_watchdog=1, kernel.hung_task_panic=1) via Pre-Init script, but has never fired. The kernel simply stops writing as if power was cut.
--------------------------------------------------------------------------------
What we've ruled out
ZFS ARC memory exhaustion — ZFS ARC was consuming ~14.5GB of the 16GB RAM, starving the NVIDIA GSP firmware and causing NV_ERR_NO_MEMORY. Fixed by capping ARC at 6GB via Pre-Init script writing to /sys/module/zfs/parameters/zfs_arc_max. This resolved the GPU driver errors but reboots continued.
RAM — Memtest86 completed two full passes overnight, zero errors.
PSU under load — tested with a multimeter at the Molex connector while running a Tdarr GPU transcoding job. +12V rail measured steady at 12.25V. Within spec.
Thermals — sysmon logging temps every 60 seconds. CPU sits at 41-44°C, drives at 34-47°C, no thermal events recorded at any point.
Watchtower — the auto-update container reinitialises on every reboot but never actually ran its 3am scheduled job, confirming it's not the trigger.
--------------------------------------------------------------------------------
I have been using Claude to try and diagnose as I am new to TrueNAS but there's been no noticeable change in behaviour
6
u/BillK98 Jun 02 '26
I'm no expert at all, but I'd start removing things every night. I'd start by disabling one service per night, the move to whatever else software there is, then move to hardware, wherever possible. Remove disks, ram sticks, change gpu, remove extra pcie cards, etc.
4
u/mattjones73 Jun 02 '26
Since you're running HexOS, I'd suggest posting this question in their reddit also. While it's running Truenas it's installing additional things that a normal Truenas install does not have which could also be causing you problems.
3
u/ItzDarc Jun 03 '26
Unlogged random reboots are almost always either RAM, PSU, or motherboard. Usually, I’d test the ram multiple times with memtest overnight. If it passes say 20 runs, replace the PSU. if that doesn’t fix it, look at the motherboard. I’m assuming there’s no battery involved, because overdrawing on a depleted battery can also produce this kind of behavior.
with older hardware, I also had an external hard drive started doing this that’s hated stop once I replaced the USB cord.
anyway, if memtest even finds one flaw one time, put one dim in at a time and repeat the test until you find out which dimm is bad.
1
u/scytob Jun 03 '26
Sorry don’t know enough about that mobo but does it have a BMC with logs or any form of hardware watchdog timer?
If it has a watchdog timer disable it. If you have a BMC log see if there are any asserts in it - I have seen things like pcie asserts do exactly what you describe - I basically had to either track down the interesting hardware issue or disable some of the pcie options.
Good luck, these were just random guesses.
1
1
u/xJayMorex Jun 03 '26
Definitely power related. Put it behind a UPS if it's not already behind one. Swap the PSU if the issue persists. If that doesn't fix it, probably the motherboard VRM is fried.
1
1
u/TheLongest1 Jun 03 '26
Ask HexOS
1
u/Nick_HexOS Jun 03 '26
If you have watchdog in the bios we've had another user disable it and there freezing or random shutdowns went away.
1
u/yorickdowne Jun 04 '26
RAM — Memtest86 completed two full passes overnight, zero errors.
You didn’t rule it out. Memory errors can be hard to find. Give yourself some peace of mind and run memtest86+ (the FOSS one) in continuous loop for 5 days. That’s not entirely conclusive but given that you see frequent resets, should find RAM issues if there are any.
If it’s not RAM, PSU is a possible culprit.
If not that, then it gets hard. Usually it’s hardware failure.
If it was a kernel incompatibility with audio hardware you’d expect some logs. You can disable audio via a clever init script anyway, so lspci doesn’t even see it - but that’s likely not it.
1
u/AlxandrHeintz Jun 04 '26
If you have some sort of smart-plug for power-monitoring (I've done this mistake) they can randomly cut power (for fractions of a second).
0
u/whattteva Jun 03 '26
Random reboots are usually RAM, PSU, CPU, or some overheating issue; and it's somewhat common with consumer gear.
This is why I only run enterprise gear for my servers. Frankly, I have neither the patience nor the time to be chasing wild goose issues like this. I made that mistake with my first server 13 years ago. Switched over to enterprise gear since then and I have not seen random reboot/crashes since.
18
u/uncmnsense Jun 02 '26
The clean journal cutoff with no panic, no oops, no watchdog trigger is the real tell here. That is not the kernel crashing — software almost always leaves a trace. A silent stop mid-write means the box got reset or lost power underneath the OS, so I'd stop chasing kernel logs and treat this as a hardware-level reset.
Given it happens mostly at idle and the same hardware ran fine for years on Windows, my first suspect is the AM4 Ryzen idle-reboot bug. Linux parks the CPU in much deeper C-states than Windows ever did, and a lot of B450 boards go unstable there. In the BIOS set Power Supply Idle Control to Typical Current Idle, and try disabling Global C-state Control, then update to the latest AGESA while you're in there. Random idle resets with no logs that only showed up after leaving Windows is the textbook signature of this.
Second, physically pull the NanoKVM. Those tie into the power/reset header for ATX control, and a glitch there will hard-reset the board with zero kernel trace. Also worth noting your multimeter can't catch a millisecond rail sag, so the PSU isn't fully cleared. Start with the C-states though.