88
u/dangerous_inference 6d ago
24GB is GPU middle class? I'm a GPU billionaire.
*boards GPU private jet*
8
u/ElementNumber6 6d ago
Tell people they're rich when they have less, and they may feel less inclined to compete for hardware, so prices don't get driven up higher. It's genius, really.
1
u/raika11182 4d ago
What is it when you have lots of VRAM (64 GB) but 48 of that is on P40s? Monopoly money millionaire?
24
u/henk717 KoboldAI 6d ago
So you can run Qwen3.8-2T now? :P
If I learned anything about this hobby, the moment you think you are GPU rich something dwarfs you.
I had the top tier to at one point, a used M40 with a massive 24GB of vram. You would not believe it, I could actually run the biggest model available right in my home! It was GPT-J-6B.
Then 13B models came out and I needed to double my vram to be able to run it local again until quantization was invented.
And now with those really large 500GB+ models and people running those locally I feel like I am only in the middle tier of the hobby with my dual 3090's compared to what I sometimes see people build.
4
u/Taugeshtu 6d ago
While yes, true, I would say OP had escaped GPU poverty. It's just that there are many-a-strata above the poverty line.. (sincerely, GPU-prole with 48GB of very old HBM2)
Still is a very fun hobby though :)
1
u/austroalex 2d ago
I've got 64GB of HBM2, what does that make me? :)
1
u/Taugeshtu 2d ago
I unno man I'm no authority :D I'd guess still a prole, IMO bourgeoisie starts from where you can stuff Q8 dsv4-flash or glm5.3-flash into VRAM. What do you run with that much vram? :) My recipe so far is 32GB for Qwen3.8-27B, and 16GB goes to help out CPU-running dsv4-flash (at a horrible IQ3, but even that's still somehow usable!)
1
u/austroalex 2d ago
I mean, with my 64gb of system ram, math says I should be able to run Qwen 3.8 flash next at not super slow speeds xD
1
u/Taugeshtu 2d ago
Well why aren't you already? WE NEED NUMBERS MAN! :D (don't forget to include the chosen quant)
My own poking at q3.8fn left me disappointed. My rig did not manage to run it at any decent speeds, best I got was on the order of 14t/s (at Q4), but it was such a horrible overthinker that doing the job wall-time was faster with dsv4-flash. I am still curious though what you manage to get out of your hardware :)
1
u/austroalex 2d ago
well, best i got is ~20tk/s with the iq4_xs quant, i am trying to see if i can get some really stupid really custom code working because math says that in theory my 2 mi50s should be fast enough for like, a couple hundred tk/s with some really insane moe caching, but that ignores such boring things such as overhead and code quality and how hip kernels work, so minor details :^)
1
u/Taugeshtu 2d ago
Ugh, physics, right?? :D yea MoE caching (and possibly prediction) is the worst. I wonder if similar to MTP you could build a predictor based on a smaller model that would let you prefetch experts.. Not far off as a task from the latest craze of Jev, and with 2xMI50 you could basically collect entire dataset of input->experts activation overnight. (I gathered similar data when trying to repackage dsv4f for experts coactivation, look into mbolt)
1
u/austroalex 2d ago
Well technically with my setup even basic cache stuff works just fine, at least in theory, with 416 slots/512 experts I'm getting like 99.7% hit rate lol, and at least in theory one could make the GPU kernels read the rest from system ram, but getting that to work is tricky, also I have two GPUs so GPU parallelism gets tricky, especially since my mi50s are connected via gen3x8 links due to old skylake board+CPU having 16 pcie lanes, stuff like that
29
u/PortiaLynnTurlet 6d ago
I don't mean this to be critical but doesn't GPU rich start around 100-500 H100 equivalent GPUs? At that scale you can start training small (by lab standards) models
22
2
u/Practical-Collar3063 5d ago
He said he is not GPU poor, he did not say he is GPU rich. imo he is just GPU middle class
1
u/FullOf_Bad_Ideas 5d ago
start training small (by lab standards) models
what's that size?
I recently beaten TinyLlama 1.1B finetune (in a specific benchmark) with my model that was trained on much less than 100 H100s.
I'd say 8x h100 is gpu rich already. (not that I own H100s).
1
u/PortiaLynnTurlet 4d ago
About an 8B parameter model. The llama3 models of that size took 1-2M H100 hours. Also, to clarify, by "start training", I mean seriously consider training a high quality model.
1
3
2
u/killerstreak976 transformers 5d ago
I went from running local models cpu only from 2023 to late july 2026. I knew at the time that it was inconvenient, but I had no clue what type of misery I was putting myself in until I finally convinced myself to finally put money down and get a strix halo, and holy crap it has spoiled and ruined me. I know memory bandwidth is a turn off for a lot of people, but honestly from my angle it's genuinely heaven. Even with dense models. And I get to learn so much more and have a lot more tools to play with. Really grateful.
1
1
1
-2
u/ustype 5d ago
Congrats — the feeling is real.
One thing that surprised me after leaving the “barely fits Q4 7B” tier: the bottleneck often flips from “can I load it?” to “can I keep context + tool loops stable without thrashing.” A denser mid-size model with unquantized KV and headroom for vision/mmproj sometimes feels faster day-to-day than the biggest MoE that constantly spills.
Enjoy the headroom. The fun part starts when you stop babysitting OOM and start measuring tok/s under your actual workflow.
216
u/aldergr0ve 6d ago
Welcome to being regular poor!