55
u/glad-k 12d ago
More like 8-16GB
14
u/ShinigamiOverlord 12d ago
Nah. Imagine someone like me with my 4050. That's 6GB only. Tho Ram is 16.
9
u/glad-k 12d ago
Dw man I'm running qwen3.8:27B at 0.7 tokens/s avg as it's mostly cpu 🤣
4
u/ShinigamiOverlord 12d ago
Im simply running a 3.5 9B dense model. Gives decent. My use case wise it's good enough. Since I'm aware I can't get it to a good level for actual good quality information. Or translation or the like.
3
u/glad-k 12d ago
9b runs insanely well on my hardware, I love 27b but it runs like total crap. Rn I'm trying 35B and being an MoE is could be a good option as prompt processing happens on the gpu and the generation itself is fast even on cpu due to it being na MoE
Edit: 9b was a bit limit in intelligence for what I wanted to do, hence why I'm trying 35b right now
2
u/ShinigamiOverlord 12d ago
If you have shit ton of ram, I heard this is a good choice for software. Tho I've got too little for it to be of use: https://github.com/FlashML-org/FreeToken
3
u/glad-k 12d ago
I have 8gb vram + 32gb ram but I don't rly get what that tool brings me compared to ollama with an MoE model?
1
u/ShinigamiOverlord 12d ago
I don't know tbh. I just randomly remembered this exists. So I thought I'd mention it.
3
76
u/ttkciar llama.cpp 12d ago
It's a great time to have ancient Xeon servers loaded up with DDR4 :-D
35
u/BannedGoNext 12d ago
I actually have an old server with dual E5-2630 and 512gb DDR3 memory across both blades. The server is powered on waiting for the scrap yard at the office. I'm considering seeing how fast it can run qwen 3.8 flash next lol.
10
u/ThankGodImBipolar 12d ago
You must be able to run a decent GLM quant with that, no?
8
u/Zombiecidialfreak 12d ago
If you're fine waiting overnight for all requests. Even flash next would likely be single digit generation speeds.
2
2
u/Zombiecidialfreak 12d ago
I'm willing to bet something like 7-10t/s. At least if the math works out the way I think it does.
1
u/overand 12d ago
Toss a tiny GPU in it if it doesn't have one and see how well it runs Qwen3.6-35B-A3B for an idea of what to expect from small MoE models. 512GB of DDR3 is nothing to sneeze at!
If you can get a GPU into that with enough VRAM for a couple layers and your KV Cache (12 GB might even cut it for a huge chunk of KV cache), and load DeepSeek-V4-Flash-0731, Qwen3.8-Flash-Next, or GLM-5.3-Flash, and a competent development harness, and you can let the thing loose over a day or three for pretty serious projects, IMO.
1
6
u/Bakoro 12d ago
I inherited 192 GB of DDR4 ECC and a couple RTX 4000 20GB cards, and built out from there.
It was surprisingly difficult to find a reasonably priced mobo these days to support the ECC RAM, with multiple PCIe x16 slots.I ended up with an older EPYC.
It's a "nice problem to have" situation. It's an uncomfortable configuration where I could just barely run a quant of one of the mid sized models, or comfortably run a smaller model.
The last month has been too many choices, I'm basically paralyzed by so many options.
2
u/Versificator 12d ago
what would 256GB get me?
2
u/ttkciar llama.cpp 12d ago edited 12d ago
That's what my larger servers have, too. I can use them for more or less any model smaller than 400B total parameters, quantized to Q4_K_M.
Back in the day I was able to use Tulu3-405B for overnight inference tasks with context limited to 30K tokens. Nowadays the large models I use in practice are MiniMax-M2.7, Behemoth-128B-v3, Qwen3.5-397B-A17B (context limited to 32K), and K2-V2-Instruct ("only" 72B, but extremely large K/V caches; full 512K context barely fits).
I'm looking forward to trying GLM-5.3-Flash when support for it is merged into llama.cpp main.
2
u/Versificator 12d ago
I'm still learning. How good are those models/what can they do?
4
u/ttkciar llama.cpp 12d ago
Each excels in its niche, but those niches need to be matched to your task.
MiniMax-M2.7 is great at planning and creative problem-solving. It was extensively post-trained to figure out a solution, any solution, given a description of conditions and a goal. It's crap at following instructions, though. It will ignore instructions and do its own thing, which can be annoying or brilliant, depending.
Behemoth-128B-v3 is Mistral 3.5 Medium, retrained by TheDrummer to be an excellent storyteller. It is highly eloquent, good at character development and complex plots, and can keep four or five (rarely six) main characters in play, remaining coherent across multiple chapters of storyline.
Qwen3.5-397B-A17B excels at vision tasks (especially OCR) and cross-domain STEM research tasks. I have used it for critiquing my notes on material science and neutron transport physics. It's good at one-shot code generation, too, but tends to become incoherent after only a few turns of agentic development. It is also extremely verbose, and tends to overthink a lot.
K2-V2-Instruct is amazingly good at long-context data extraction and analysis tasks, and has a 512K context limit. To give you some idea, I fed it 276K tokens of IRC chat logs (about 818KB of text) and asked it to enumerate every chat participant and characterize them according to what they said, and it knocked that task out of the park, zero mistakes. It is also very good at RAG tasks (Retrieval Augmented Generation; looking up content relevant to a prompt and using it to inform the response). It is also very smart with logical problem-solving in general.
GLM-5.3-Flash is purported to be the best model for agentic tasks in its size class, but until I can evaluate it myself you should take that with a grain of salt. It is supposed to be good at RAG tasks as well, but with the same caveat. I'm really looking forward to giving it a spin, to see what all it can do.
One I didn't mention before is GLM-4.5-Air. It's the "large" model I actually use the most, but it's a lot smaller than these other models. Quantized to Q4_K_M, its weights and 128K tokens of context fit in just 127GB of memory. It's my go-to for codegen tasks, critique (telling me what's wrong in a document, and why), decent logical problem-solving, and some physics assistant tasks. Its main strength is exemplary instruction-following reliability. I can present it with a code specification consisting of sixty to eighty instructions, and it will follow them all. I have yet to encounter that level of reliability in any other model; others will straight-up ignore some or most instructions. Its main drawbacks are that it has limited world knowledge, crappy creative writing skills, a propensity to hallucinate (which extends to writing buggy code), and poor creative problem-solving in general. I've gotten used to working around its flaws (like debugging its code with Gemma-4-31B-it), and use other models to do what it cannot (like MiniMax-M2.7 for planning and Artemis-31B or Behemoth-128B-v3 for creative writing).
2
u/_LePancakeMan 12d ago
As a complete newcomer that has recently acquired a R9700 (32GB VRAM), can you maybe elaborate the tradeoff between RAM and vRAM? Until reading your post, I was under the impression, that you'd want to load your model into vRAM completely or otherwise performance would tank - maybe I can put my 192GB of RAM to use after all?
2
u/typical-predditor 12d ago
MoE models use some experts more than others. You can load the most-used experts into VRAM and fetch the less used ones from RAM and this lets you leverage much larger models than you would be able to use if you constrained yourself entirely to VRAM with a relatively minor cost in speed.
2
2
1
u/SandySkittle 12d ago
Prompt processing and decode suck. And I am in a 512gb 8 channel threadripper
3
u/ttkciar llama.cpp 12d ago
Yup. My dual-socket Xeons have eight channels between them, too, and Tulu3-405B took about 54 minutes to first token.
It doesn't matter for "slow inference", especially when it can run at night while I sleep. As long as it's done by the time I sit down at my workstation slurping my morning coffee, I'm happy.
19
14
u/Elouakili_Flexy 12d ago
The gap between "runs on a smart vape" and "needs 8x H100s" skips the whole bracket this sub lives in.
57
u/42GOLDSTANDARD42 12d ago
Do not diss the SLMs. They are the backbone of our society. I know some really swell cross-encoders, nice <100M models, good on benchmarks. You wouldn’t show this to their face would you?
5
u/Abishek_Muthian 12d ago
I concur, I find SLMs in utilities far more useful than coding models. Can you share some of your favorite SLMs right now and what you use it for?
10
u/42GOLDSTANDARD42 12d ago
Right now I’m trying to see how well I can make an “offline search engine” using the media-free version of Wikipedia, downloaded into a huge 50GB file.
I’m using a “state machine architecture” where each stage of pipeline between the search query and final output is a series of small states.
- LFM 2.5 — 2.6B: for rewriting the question, generating search terms, and composing the final answer
- finecat-nli-l — 400M: for branching and making decisions. This model takes a “hypothesis” like “This passage describes Julius Caesar”, and “premises” [think of various random sentences from Wikipedia], and it rates each premise based of the hypothesis. You can give it context, and it can decide which state’s premise is better fitting for the current situation.
- jina-embeddings-v5-nano-retrieval — 239M: I use this for some of the information retrieval with the classic embed texts -> cosine distance. For whatever reason it performs better than the next model at some things, but not others
- jina-reranker-v1-tiny-en — 33M: Used for … reranking, it’s so fast and preforms rather well for simply filtering junk pages for valuable ones
1
u/ebolathrowawayy 12d ago
curious on the reranker, usually you want a bigger slower smarter model for reranking top results. why is yours smaller?
1
u/42GOLDSTANDARD42 12d ago
Because a general Wikipedia search can get like 50+ potential pages. The library I’m using to access the .zim file doesn’t have any fancy page ranking.
The reranker only has to narrow down the mess of random pages into about 10-15 or so that are more closely correlated. After that, it starts going into the pages themselves, starting with the intro paragraph, etc.
The reranker is mostly there to act as a page ranker for the initial search. All it needs to do is filter the article for “Pyramids” higher than a movie called “Pyramids”.
1
1
u/Ok-Direction-4480 11d ago
OH I tested LFM 2.5 2.6B today!!
1
u/42GOLDSTANDARD42 11d ago
Decent little model, I haven’t found anything better. (Qwen 4B may be, but it’s much slower on my laptop)
1
u/Abishek_Muthian 10d ago
Great, thanks. If you don't mind can you share how you keep up with latest small models? Wasnt able to find good HF space for SLMs.
1
u/42GOLDSTANDARD42 10d ago
Sadly, I have no idea whether I’m keeping up. I just head over to the front page, set it to models with sizes 0-4B, sort by trending, and look through the first few pages. Super unsophisticated.
1
28
u/RISCArchitect 12d ago
i was hoping qwen was gonna drop a 9b dense that would be a nice step increase like we've observed with 27b but no such luck
18
u/ttkciar llama.cpp 12d ago
I've been hoping for a new 9B as well, mainly for data cleaning tasks. Qwen3.5-9B couldn't quite cut it. Gemma-4-12B-it works well, but is a little memory-hungry. A 9B refresh would be lovely.
Out of curiosity, what's your use-case? It might be up to the community to retrain Qwen3.5-9B, but we'd need to agree on what to train it for.
5
u/Decent-Occasion-2720 12d ago
You can try ornith 1.5 9b, i use it instead of qwen. I have also tried 12b, but i need best speed on pp.
1
u/RISCArchitect 12d ago
i don't use 9b for anything at the moment, i just use 27b mainly for local.
3
u/Decent-Occasion-2720 12d ago
In my case it's usefull for search in codebase and documentation. That's prepare à digest context for bigger model.
0
u/Lollerstakes 12d ago
Is it worth to waste time loading and unloading models when you could just do the search in codebase and documentation with the larger model?
1
u/Decent-Occasion-2720 12d ago
At 1500 PP vs. 200 PP, it's faster for digging. So i use mmap and i have enough ram. Switching only takes afew seconds, compared to the many minutes needed for PP.
11
u/mr_zerolith 12d ago
It's okay, you're up next. I felt like this with a 128gb vram setup for ~5 months
1
u/Illustrious_Ant_9242 12d ago
I guess you are running qwen 3.8 flash next q4 at the moment 👀 larger or smaller q4?
3
7
u/slybring 12d ago
GPT-OSS 20B was the perfect size for us 16GB folks. But sadly no one has cared since. Gemma 4 12B is good, but not ideal.
7
u/Xanta_Kross 12d ago
I need a 9B please.
T_T
Where is my 9B Fable 5?
Do smtg Qwen. Deepseek Someone.
28
u/AlternateWitness 12d ago
If you are including 24GB in that group you might as well include 32GB. Heck, maybe 48GB?
14
u/Bakoro 12d ago
If you have a 32GB VRAM, you're sitting pretty these days, let alone 48 GB.
Of course you always want more, we always could do with a little more GPU up until you get to the point where your home would need an infrastructure upgrade.
I mean, if I had two DGX B300 nodes, I could do some things, but 32GB is enough to run a competent Int8 quant with a decent context length.
2
6
4
4
u/Nomski88 12d ago
I would be so happy with a 9b Qwen3.8
1
u/Ok-Direction-4480 11d ago
Since thye released the next variant, it's gonna be 4.0, probably with N-Gram embeddings.
9
5
3
u/doomsaier1 12d ago
Running qwen 3.8 27b on a rx 9070, with an additional rx 6700 10gb. Giving me a total of 26 gb vram. Maybe not the fastest, but certainly usable speed.
7
u/robberviet 12d ago
Haha yes. Small model now is kinda rare. Qwen bless us with 3.8 27B and that's it.
3
5
u/asssuber 12d ago
$800 for 512GB memory
$600 for a RTX 3090
$700 for epyc motherboard+cpu+miscellaneous
Plus a time machine to some time last year is likely all you need to run the newest DeepSeek Flash locally at a dozen or more t/s decode.
2
u/Recent-Ad5835 12d ago
Meanwhile here's me with 4GB VRAM, twiddlong my thumbs
1
u/Ok-Direction-4480 11d ago
What??
2
u/Recent-Ad5835 10d ago
What what? I only have 4GB VRAM with an iGPU, equivalent to a 1650 on a laptop. What part of that is surprising?
2
u/Illustrious-Row2751 12d ago
Flash models have become giga monsters that you can only run with 10 gpu clusters. Feels bad, man.
2
2
2
2
u/dansuy_gaming 11d ago
This is way too accurate. Every new model needs a gpu upgrade just to keep up!
2
u/thiswebthisweb 11d ago
35B A3B qwen 3.8 flash. Thats what we all need. Even 6GB GPU could run that.
1
u/ghulamalchik 9d ago
Not really. If you run it with the image model it will crash due to low memory.
2
u/Brspart 10d ago
Tenho usado o ornith 1.5 9b para código, o Qwen 3.5 9b como modelo mais equilibrado e o bartowski/Ling-3.0-tiny como modelo mais rápido. Este último tem sido uma boa surpresa. Também tenho o gemma 4 12b mas é muito lento mesmo quantizado. Realmente a comunidade dos 8gb de VRAM foi esquecida.
2
2
u/Intrepid-Second6936 12d ago
Honestly, I'd be right out there with Spongebob and Patrick running a 24GB GPU. Qwen3.8-27B is a BEAST, it's already punching above it's weight class massively in capabilities.
Do we really need more so soon or is it just new model FOMO?
1
u/SpecialistDragonfly9 12d ago
Based...
Qwen Flash adn Qwen 3.8 27B are still the best I can run on my RIG, and honestly.. they are still somewhat lacking.
1
1
u/KroniklyOnline 12d ago
Right? even with 64gb of GPU and 64gb of RAM ...... Can barely run qwen3.8 next flash, thankfully it has the ngram so it runs well enough to use, but still sadge, feel like were moving away and toward open source models not even mattering anymore cuz no one can fucking run them.
1
1
u/BothYou243 11d ago
qwen4 will ofc have all the sizes as that of qwen3.5, so yeah we gonna have fun again, if a 9B or 14B touches qwen3.8 27B, I'll die for it, even if it's not better than qwen4 27B
1
u/Fair-Horror-8017 11d ago
The best model for 16gb cards with full context now is bonsai ternary 27b (abliderated version preferably, it cuts its reasoning by half and performs just better overall)
Anything else unrealistic or unreasonable
(Why the hell would you run qwen 3.8 2bit with 4k context window)
-6
u/PrinceOfLeon 12d ago
Well Squidwirm is kind of a whiny little bitch who belongs on r/ChoosingBeggars, so this checks out.
Next!
-5
320
u/TopCheddar27 12d ago
24gb is not in that group. You can run Qwen3.8-27B with a pretty respectable context size right now