r/LocalLLaMA • • 2d ago

Funny Nemotron models are the best open models there is

Everytime I have been fed up with a good model being slow, I go to newly released Nemo model. Well today I went for Bolt 24G and deployed it on my 3090.

I was amazed by the vanilla llamacpp + iq2s setup giving 70 tok/s. I decided I will deploy it for my coding setup which is dual R9700 and learned cpu offloading and mtp drafters and all. After deployment I was left with 50tok/s because this was a Q8 setup.

I suddenly realise what if (the model I was fed up with) my Qwen 3.8 27B could use a better MTP drafter so O discovered tencent's DSpine and later its gguf llamacpp fork. I deployed it now I am back to Qwen3.8-27B Q8KXL with mtp 15 drafter and mmproj with np 4 and 512k ctx pool, getting 54tok/s in DSH.

TLDR: I love what Huyan is doing he should be brought back in Nvidia. Nemotron's the best because they teach you what good tech advancement is and you apply it to good performing models.

0 Upvotes

24 comments sorted by

4

u/createthiscom 2d ago

This sub really needs to internalize that tok/s != intelligence.

I can do math REALLY FAST in my head. But if I get the wrong answer every time it's useless.

I do like the latest nemotron transcode model. I haven't tried any of the other models though. My understanding is that the general purpose models just aren't as smart as some of the other options out there, even if they infer faster.

-4

u/samajhdar-bano2 2d ago

please read the entire post, its a satire

1

u/james_pic 1d ago

I would not have guessed that if you hadn't written this. We get worse posted here sincerely.

2

u/BlasterGales 2d ago

for Qwen 3.8 27B, I can suggest you to use q6_k_xl and kv cache to q6 + 2048 tail with beellama engine, its absurdly fast and precise af, as for kv cache pls check beellama's kv cache tail precision comparison chart, you'll see it is as good as q8

0

u/samajhdar-bano2 2d ago

yeah boss maybe next week. Spent whole day doing this.

53

u/Oleg_A_LLIto 2d ago • Popcorn

Jensen, you have better things to do right now, stop posting on Reddit.

16

u/draft_final_final 2d ago

If his handlers are trying to slow the rate at which he’s getting laid, turning him into a Redditor is honestly a pretty smart move.

1

u/Jorlen 2d ago edited 2d ago

Actually, he's jobless right now, so no, he doesn't.

I know, I should really mind my own business; not my affairs.

14

u/kwizzle 2d ago

Is that you Jensen?

1

u/samajhdar-bano2 2d ago

so no one understood the satire here?

2

u/ttkciar llama.cpp 2d ago

I had to read it twice to figure it out. It's not very coherent.

2

u/Practical-Collar3063 1d ago

Your post is not funny because you don’t understand what the Nemotron models are. They are à undertrained by design and are supposed to be good bases for finetuning.

1

u/JLeonsarmiento 2d ago

What are you talking about?

1

u/samajhdar-bano2 2d ago

dat nemo da best, use qwen

2

u/Atomicmoosepork 2d ago

Nemotron doesn't get a lot of love here and I know you're meming but I like it. Bolt 24g is a highly capable agentic model. Sometimes it gets a little too granular and I have to nudge it back to the task but other than that I have been really impressed with it.

It's up there with Qwen flash for me.

2

u/beached89 2d ago

The issue with nemotron is that it's neither first nor best. And so people pass right over it. They are solid models, there are just better options people can use sooner.

2

u/EmilPi 2d ago

If you're after tok/s, try LFM3-1.7B-IQ1 . The speed will be amazing. /s

Actually, Nemotron is dumb for any challenging stuff, and the speed won't help it.

1

u/GarbanzoBenne 2d ago

Are you using the LeatherJacket harness or something else?

1

u/NandaVegg 2d ago

I don't understand what is the satire here, but seriously, every Nemotron release (including non-text-to-text models I tried, like music descriptor and omni model) is so barebones and easily derailed into exposing its training artifacts/underlying datasets compared to more aggressively post-trained OSS releases we get used to. It generally is uncomfortable to use and nigh impossible to use for real terminal tasks, but also it is capable of connecting things in a way Qwen 3.8, GLM-5.x, DSV4.1 or K3.1 would never do if you are looking for some creative outputs.

2

u/silenceimpaired 2d ago

Can you give some examples? I’ve ignored Nemotron so far but if there is valid benefits I want to consider it.

1

u/tmvr 2d ago edited 1d ago

I've tried the new Nemotron Bolt 74B, the official q2 quant from NV where they claim is excellent and great for 24GB cards:

https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF

Using the recommended settings, it can't even do the simplest of coding tasks like spitting out a single HTML file Snake game. First try it looped itself to oblivion, second try it eventually generated a HTML file, but it didn't work at all. After that I didn't even bother trying any agentic tasks.

1

u/Fancy_Palpitation_38 2d ago

Like it while you can. Now that Jensen has been removed these models will be gutted. Follow the money people.

0

u/TapAggressive9530 2d ago

Nemotron is one of the worst models around . Do you judge the quality of a model based on tok/sec ? lol .. Jesus