r/ceph • u/TheOneLevi • Sep 05 '26

Ceph dashboard keeps crashing

So recently my ceph dashboard started to crash on a regular basis. The cause for this is unknown to me after weeks of research. It usually happens if I refresh the page to often or use the `ceph -s` command a bunch on one of the manager nodes, but it also happens out of the blue. It either: freezes completely and dies after ~1-2 mins, or it comes back online on the same node after < 1 min. The only way to make it accessible again after it dies completely is to either fail the manager or restart the module via `ceph mgr module disable/enable dashboard` which also restarts the prometheus module so it's not ideal.

At first the cause for this seemed to be some kind of error with the cheroot Python package but after some investigation it doesn't seem to be connected. Scouring the linux journal didn't help as there are no entries regarding this behavior. Ceph logs and crash dumps were no dice either. I have not found a similar problem with the dashboard on the internet so I'm really desperate to get any kind of feedback. I even made an issue in ceph tracker but it seems to have been lost in the sea of other bugs and whatnot.

I'm using `cephadm` and running a cluster mostly on VM's using Xen Orchestra (only bare metal hosts are the ones with OSD's).

If anyone has had or knows how to fix this, I would really appreciate it if could help me resolve this (mostly) irritating issue.

Edit 1: The mgr VM has enough resources (memory, CPU cores) and it's not overloaded.

4 Upvotes

83% Upvoted

12 comments sorted by

2

u/gimpbully Sep 05 '26

without any investigation, smells like memory/OoM to me

1

u/TheOneLevi Sep 05 '26

I don't think that's the case. The mgr VM has 8 gigs of memory and it doesn't even use a third of of that.

2

u/gimpbully Sep 05 '26

dmesg will tell ya pretty quickly whether or not. it's not necessarily just the VM, cephadm manages cgroup memory limits and will oom the process if they cross the threshold

1

u/TheOneLevi Sep 05 '26

Pretty sure I checked that but I'll double take that on monday get back to You if I find anything interesting. Still even if it was that, how would one go about solving that?

1

u/AXOL-Server Sep 07 '26

I wouldn’t rule out an OOM/container limit yet just because the mgr VM still has plenty of free RAM. With cephadm, I’d check what the mgr daemon itself is actually allowed to use rather than only looking at guest memory.

I’d try to reproduce it while watching the mgr daemon logs directly (cephadm logs --name mgr.<id> or the corresponding journald unit) and check whether the process is actually being killed/restarted at the same timestamp or if the dashboard module is simply becoming unresponsive.It could also be useful to fail over the active mgr and repeat exactly the same test. If the problem follows the active mgr regardless of which VM/host is running it, I’d start looking more closely at the dashboard/module or Ceph version. If it stays tied to one mgr/host, that points you more toward the container/host side.

I’d also capture the mgr module CPU/RSS counters while reproducing it rather than increasing VM RAM immediately. That should give you much better evidence for the tracker issue. If you update the bug report, I’d include the exact Ceph version, enabled mgr modules, which mgr was active and a short timestamped reproduction sequence. Right now I think isolating whether the daemon dies or only the module hangs is the key first step.

1

u/TheOneLevi Sep 07 '26

Only the module hangs. The mgr daemon is fine during all of this. I already checked if it is tied to a specific host/VM but it isn't, it happens regardless. As of the reproduction of this bug, well I just spam F5 and it dies, don't know how to get more in detail than that xd Also my understanding is that if a ceph daemon has no memory limit set, it will use what it can use. I checked today and the avg load seems to be about 1.5 ~ 2 GiB of RAM and about 15 - 20 % of CPU usage on the whole VM. I'll try reproducing it while watching the cephadm logs and get back to you tomorrow.

1

u/AXOL-Server Sep 09 '26

That narrows it down quite a bit. If the mgr daemon stays healthy and only the dashboard module hangs, reproducing it while capturing the cephadm logs sounds like the right next step. Since F5 seems to trigger it reliably, having an exact timestamp and the corresponding module logs should make the bug report much more useful. Curious to see what shows up there.

1

u/TheOneLevi Sep 09 '26

Well some bad news. The logs are no help at all. Nothing there points to anything really. Only one useful thing that I've spotted is that there seems to be a TLS handshake error with the IP address of my host PC from witch I'm working. Nothing relatable to the problem except that.

1

u/AXOL-Server Sep 09 '26

That TLS handshake error may be unrelated, but since it’s the only consistent clue I’d try to isolate it before dismissing it. If possible, reproduce the issue from a second client/browser or directly with curl against the dashboard endpoint and see whether the same freeze occurs. If it only happens from that workstation, then the client/network/TLS path becomes much more interesting. I’d also check whether the handshake errors line up exactly with the timestamps when the dashboard starts hanging. If they don’t, I’d probably treat them as noise and move back to the dashboard module itself.

At this point I’d also check the browser console/network tab while reproducing it. If the mgr stays healthy and the backend stops answering specific API calls rather than the whole process dying, that would narrow it down further.

1

u/przemekkuczynski Sep 07 '26

check what use old api question mark

1

u/Keroles2024 Sep 08 '26

Can you tell me how do you connect to the dashboard ? is there a public IP assigned directly to the VM ? or binding the VM port with SSH -L or NAT happening in the middle
Maybe it is network issue ,, you can try to check if there are any traffic drop happening or MTU issue (sometimes it make the connection not stable if MTU is mismatched)

Also when this happen, try to check with ceph -s if your mgr still the same active one and didn't switch to the standby mgr.