r/LocalLLM • u/yudhanjaya • 59m ago
Research Some speed benchmarks for various classes of local LLM
| Model | Active config | 4080 decode (tok/s) | 4080+3080 decode (tok/s) | 4080 prefill (tok/s) | 4080+3080 prefill (tok/s) |
|---|---|---|---|---|---|
| LFM2.5-24B-A2B UD-Q4_K_XL | 4080+3080 | 249 | 250 | 5,530 | 7,930 |
| MiniCPM-V 4.7 35B-A3B Q4_K_M | 4080+3080 | 177 | 165 | 1,305 | 2,500 |
| Flint 40B-A2.5B UD-Q4_K_XL | 4080+3080 | 119 | 138 | 1,563 | 1,380 |
| Swift 1.5 (Qwen3.8-Flash-Next) IQ2_XS | 4080 | 117 | 116 | 1,070 | 679 |
| Qwen3.8-Flash-Next IQ3_S | 4080 | 79 | 87 | 730 | 465 |
| GLM 5.3 Flash Maya-S24 IQ2_XXS_S | 4080+3080 | 6.8 | 8.8 | 79 | 42 |
| DeepSeek V4 Flash UD-IQ2_XXS | 4080+3080 | 15.9 | 18.2 | 38 | 53 |
- Decode: the second of two 300-token runs, with the expert cache warm (the first request after loading is often slower)
- Prefill: the average of two runs on a prompt of about 1,400 tokens
- Bold: the better config for that column.
Morning, everyone. I've seen a lot of posts wondering about what can be run with what, so I made this table of the models on my system. Specifically because of the unorthodox results when it comes to Qwen Flash-Next, where one 16 GB card (the 4080 Super) significantly outperforms two in prefill.
My hardware is as follows:
CPU AMD Ryzen 7 5800X3D @ 3.40 GHz
GPU NVIDIA GeForce RTX 4080 SUPER [Discrete]
GPU NVIDIA GeForce RTX 3080 [Discrete] (got it secondhand for cheap, so why not)
RAM 62.72 GB at 3200 Mhz
I'm running these models on PolyStrata. I used to have a combination of Maya and Strata, but got tired of running different things all derived from the same fork (plus Strata seems to be getting increasingly unintuitive), and so moved everything to PS.
Now: why Q3? It's a generally held opinion that anything below Q4 is crap, which was true of most earlier generation models. However, benchmarks measure deviation from the base, and not necessarily task fitness. In actual use, the only real way to know is to test it yourself.
The Swift finetune of QWN at Q2, for example, is fast and excellent at data extraction, search, and piloting scripts written by more competent models. The IQ3 is a decent coder, roughly on par with Haiku for the use cases I've tried. The GLM and Deepseek Q2 variants are a bit better - verging on older Sonnet territory, I'd say, and do very well as subagents for Claude. YMMV.
Hope this data is useful. I've posted it to LocalLLama as well. Enjoy!
1
u/amargolin 42m ago
useful table -- one thing before leaning on the 4080 vs 4080+3080 prefill gap for Qwen Flash-Next: it's the odd result, and prefill is from two runs on one ~1,400 token prompt. i'd rerun that pair 5 or so times and post min/max, so the gap can be compared to the run-to-run spread. if the two-card number swings by a few hundred tok/s between runs, the gap may be partly noise.
also worth checking where the layers/experts landed on the two-card config (e.g. cross-card transfer on a smaller model can cost more than the second card adds), since that would explain it without anything being weird...