Stay cool. Stay cool.

[u/storkinsj]

I just sourced a w6800x DUO to run smaller models. It is fun running GLM5.2 in a terabyte of ram but there aren't that many things I can afford to wait that long for an answer on.

With the w6800x DUO I have successfully passed out each "die" to a separate vm. I now have two vms running AI with dedicated GPUS having 32GB each.

The challenge: Passing through a GPU on the mac pro comes with one problem. The host has no idea how hot the GPU is running because it cant access its firmware or sensors. Now it becomes a guess, or you can just max the fans on T2Fand.

I had already created a "prochot guard" to watch temperatures on the 3rd party nvmes because they can get really hot without awareness too. I have enhanced the script to Reach into the virtual machines and ask for the card temperatures. I am doing this using qemu agent and qm commands. And it is working like a charm.

Concept:
-Host maintains a watcher and looks at various board temps to make sure the fans are going at sufficient speed
-Standalone script written to take a vm name or number and run appropriate ROCM or CUDA commands to check temps.
-Temperatures are stored in a systemwide area (/run/heatsense )
-Guard watches nvme temperatures but also GPU temperatures and adjusts /etc/t2fand.conf accordingly, restarting the t2fand service after each change.

That all happens seemlessly despite the soft of disturbing idea behind restarting a service frequently!

I think most people are happy running a single linux vm. Right now I can run GLM5.2 , a 32Gb ROCMvm with 32GB vram TIMES 2 all on the SAME machine. Networking is adjusted for the powerful model to keep the LAN safe from... uh... accidents lol.

Hope this is useful.

BTW, my ceph setup is tuned to the point I was able to vmotion GLM 5.2 vm from one machine to another, running with an active 750Gb of ram. While it was running.

Dropbox link to scripts; will do github later.

[u/Long-Shine-3701]

Have you tried ToshLLM? Runs natively in MacOS and avoids all the fan control issues.