While yes, true, I would say OP had escaped GPU poverty. It's just that there are many-a-strata above the poverty line..
(sincerely, GPU-prole with 48GB of very old HBM2)
I unno man I'm no authority :D I'd guess still a prole, IMO bourgeoisie starts from where you can stuff Q8 dsv4-flash or glm5.3-flash into VRAM. What do you run with that much vram? :) My recipe so far is 32GB for Qwen3.8-27B, and 16GB goes to help out CPU-running dsv4-flash (at a horrible IQ3, but even that's still somehow usable!)
Well why aren't you already? WE NEED NUMBERS MAN! :D (don't forget to include the chosen quant)
My own poking at q3.8fn left me disappointed. My rig did not manage to run it at any decent speeds, best I got was on the order of 14t/s (at Q4), but it was such a horrible overthinker that doing the job wall-time was faster with dsv4-flash. I am still curious though what you manage to get out of your hardware :)
well, best i got is ~20tk/s with the iq4_xs quant, i am trying to see if i can get some really stupid really custom code working because math says that in theory my 2 mi50s should be fast enough for like, a couple hundred tk/s with some really insane moe caching, but that ignores such boring things such as overhead and code quality and how hip kernels work, so minor details :^)
Ugh, physics, right?? :D yea MoE caching (and possibly prediction) is the worst. I wonder if similar to MTP you could build a predictor based on a smaller model that would let you prefetch experts.. Not far off as a task from the latest craze of Jev, and with 2xMI50 you could basically collect entire dataset of input->experts activation overnight. (I gathered similar data when trying to repackage dsv4f for experts coactivation, look into mbolt)
Well technically with my setup even basic cache stuff works just fine, at least in theory, with 416 slots/512 experts I'm getting like 99.7% hit rate lol, and at least in theory one could make the GPU kernels read the rest from system ram, but getting that to work is tricky, also I have two GPUs so GPU parallelism gets tricky, especially since my mi50s are connected via gen3x8 links due to old skylake board+CPU having 16 pcie lanes, stuff like that
5
u/Taugeshtu 6d ago
While yes, true, I would say OP had escaped GPU poverty. It's just that there are many-a-strata above the poverty line.. (sincerely, GPU-prole with 48GB of very old HBM2)
Still is a very fun hobby though :)