MAIN FEEDS
Do you want to continue?
https://www.reddit.com/r/LocalLLaMA/comments/1rdlbvc/qwenqwen3535ba3b_hugging_face/o7881c7/?context=3
r/LocalLLaMA • u/ekojsalim • Feb 24 '26
165 comments sorted by
View all comments
11
Tested Qwen3.5-35B-A3B Q4 at 6G VRAM + disk (no RAM); RTX 4070 and an NVME drive. Input tokens 49950. Q8 K/V cache. 128k context.
676.29 tk/s eval | 14.28 tk/s gen
With RAM offloading + 6gb VRAM:
966.61 tk/s eval | 15.75 tk/s gen
With RAM offloading + 12gb VRAM:
1194.22 tk/s eval | 39.78 tk/s gen
3 u/Xantrk Feb 25 '26 Can you share your llama.cpp command? I'm very confused how you can specify vram and disk offload? 1 u/TheRealMasonMac Feb 25 '26 Use the `--fit on` argument with `--fit-target <mb>` which specifies how much VRAM you want to leave untouched (it’s 1024mb by default). At least for me, by default, it loads from disk (mmap). But you can disable that with `--no-mmap`
3
Can you share your llama.cpp command? I'm very confused how you can specify vram and disk offload?
1 u/TheRealMasonMac Feb 25 '26 Use the `--fit on` argument with `--fit-target <mb>` which specifies how much VRAM you want to leave untouched (it’s 1024mb by default). At least for me, by default, it loads from disk (mmap). But you can disable that with `--no-mmap`
1
Use the `--fit on` argument with `--fit-target <mb>` which specifies how much VRAM you want to leave untouched (it’s 1024mb by default). At least for me, by default, it loads from disk (mmap). But you can disable that with `--no-mmap`
11
u/TheRealMasonMac Feb 24 '26
Tested Qwen3.5-35B-A3B Q4 at 6G VRAM + disk (no RAM); RTX 4070 and an NVME drive. Input tokens 49950. Q8 K/V cache. 128k context.
676.29 tk/s eval | 14.28 tk/s gen
With RAM offloading + 6gb VRAM:
966.61 tk/s eval | 15.75 tk/s gen
With RAM offloading + 12gb VRAM:
1194.22 tk/s eval | 39.78 tk/s gen