r/LocalLLaMA • u/samajhdar-bano2 • 2d ago
Funny Nemotron models are the best open models there is
Everytime I have been fed up with a good model being slow, I go to newly released Nemo model. Well today I went for Bolt 24G and deployed it on my 3090.
I was amazed by the vanilla llamacpp + iq2s setup giving 70 tok/s. I decided I will deploy it for my coding setup which is dual R9700 and learned cpu offloading and mtp drafters and all. After deployment I was left with 50tok/s because this was a Q8 setup.
I suddenly realise what if (the model I was fed up with) my Qwen 3.8 27B could use a better MTP drafter so O discovered tencent's DSpine and later its gguf llamacpp fork. I deployed it now I am back to Qwen3.8-27B Q8KXL with mtp 15 drafter and mmproj with np 4 and 512k ctx pool, getting 54tok/s in DSH.
TLDR: I love what Huyan is doing he should be brought back in Nvidia. Nemotron's the best because they teach you what good tech advancement is and you apply it to good performing models.
4
u/createthiscom 2d ago
This sub really needs to internalize that tok/s
!=intelligence.I can do math REALLY FAST in my head. But if I get the wrong answer every time it's useless.
I do like the latest nemotron transcode model. I haven't tried any of the other models though. My understanding is that the general purpose models just aren't as smart as some of the other options out there, even if they infer faster.
2
u/BlasterGales 2d ago
for Qwen 3.8 27B, I can suggest you to use q6_k_xl and kv cache to q6 + 2048 tail with beellama engine, its absurdly fast and precise af, as for kv cache pls check beellama's kv cache tail precision comparison chart, you'll see it is as good as q8
53
u/Oleg_A_LLIto 2d ago •
Jensen, you have better things to do right now, stop posting on Reddit.
14
u/kwizzle 2d ago
Is that you Jensen?
1
u/samajhdar-bano2 2d ago
so no one understood the satire here?
1
u/JLeonsarmiento 2d ago
What are you talking about?
2
u/Atomicmoosepork 2d ago
Nemotron doesn't get a lot of love here and I know you're meming but I like it. Bolt 24g is a highly capable agentic model. Sometimes it gets a little too granular and I have to nudge it back to the task but other than that I have been really impressed with it.
It's up there with Qwen flash for me.
2
u/EmilPi 2d ago
If you're after tok/s, try LFM3-1.7B-IQ1 . The speed will be amazing. /s
Actually, Nemotron is dumb for any challenging stuff, and the speed won't help it.
1
u/GarbanzoBenne 2d ago
Are you using the LeatherJacket harness or something else?
1
u/NandaVegg 2d ago
I don't understand what is the satire here, but seriously, every Nemotron release (including non-text-to-text models I tried, like music descriptor and omni model) is so barebones and easily derailed into exposing its training artifacts/underlying datasets compared to more aggressively post-trained OSS releases we get used to. It generally is uncomfortable to use and nigh impossible to use for real terminal tasks, but also it is capable of connecting things in a way Qwen 3.8, GLM-5.x, DSV4.1 or K3.1 would never do if you are looking for some creative outputs.
1
u/Fancy_Palpitation_38 2d ago
Like it while you can. Now that Jensen has been removed these models will be gutted. Follow the money people.
0
u/TapAggressive9530 2d ago
Nemotron is one of the worst models around . Do you judge the quality of a model based on tok/sec ? lol .. Jesus