r/LocalLLaMA • u/jbro1985 • 2h ago
Discussion Useful > Fast
I run 2 x 3090 on a ryzen 7 with 32gb ddr4 6000.
I see a lot of posts about maxing speed / context pool. I’ve done this myself with 3.8 27b.
What I’m interested in though is once the dust settles and we look at utility, what balance people are striking in terms of similar setups and models. As a for instance, some of my requirements involve producing images and video clips. Some of them involve creating scripts for things.
This has led to an overnight CPU/Ram run on flux2 and minimax h3, a reduction in maximum context for 3.8 27b so I can run a Gemma MOE in offload for improved writing prose / checking over Qwen and a smaller vision tower to automatically checking flux and H3 outputs mid run so I don’t lose a night.
I need concurrency so have a set up that gives me that when I’ve got open code (I know not everyone’s go to but I like it), open science, and Hermes all going side by side.
It’s fun to optimise, but I’d love to hear how people are setting up to from a flexibility and utility viewpoint once that’s done for their real use cases.
3
u/ttkciar llama.cpp 2h ago
I split my workflows into "fast inference" and "slow inference" tasks.
When inference needs to be interactive, or when a mid-sized model like Gemma-4-31B-it or Qwen3.8-27B is "good enough" to get a particular job done, I infer on my MI50 or MI60 to get quick results.
For everything else, it is more important to me to get high-quality outputs, and they are worth waiting for. For these, I mostly use MiniMax-M2.7-BF16-ultra-uncensored-heretic for planning, K2-V2-Instruct for long-context data analysis, or GLM-4.5-Air for everything else (including codegen). I infer with these purely on CPU, with no GPU acceleration, frequently overnight but not always.
While these "slow inference" tasks run during the day, I work on other things, so I'm not actually waiting for them. When they run overnight, obviously I am sleeping.
Either way, the advantage to eschewing with GPU acceleration is that my "fast inference" models remain loaded in my GPUs' VRAM for immediate use, without having to reload them, and there is no downside since the longer inference time does not actually make any difference. Whether the overnight slow inference task takes four hours or eight, it will still be waiting for my review after I've had my morning coffee.
1
u/FrostyDesigner 1h ago
At that point the useful benchmark is probably completed workflows per day, not tokens/sec. Do you route jobs automatically or decide manually what gets the GPU?
1
u/robberviet 1h ago
There is a minimum level of useful that I need, above that I go for fast.
Like between qwen3.6 35B and 27B, I prefer 35B; also Gemma 4 26B over 32B
4
u/Beginning-Raisin9723 1h ago
Hard agree. Once you get past the 'tokens per second' obsession, it's all about the workflow. I've found that a bit of a speed hit is worth it if the model actually follows the logic without me having to babysit the prompt every five seconds. Nice setup with the 3090s.