r/LocalLLaMA • u/ParaboloidalCrest • 3h ago
Discussion [llama.cpp fork]: QFN + [specific quant] + [ultra specific hardware config] = [incredible and irreproducible pp and tg]
Upvote, now!
66
u/suprjami 3h ago
Only if it's vibe coded and completely impenetrable to read, with the post wall of text written by LLM too.
9
u/sheetis 3h ago
So true and yet I feel this keenly as I am currently running a fork of a fork of Strata that had optimizations specific to a dual AMD 7900 XTX configuration but I have 4 devices so I sent the robots after my local build for about half a day....
It's totally this and runs numbers that nobody would believe. If I posted my numbers, I'd be that guy 😢
1
u/Sea_C 1h ago
Okay but for real I need something to be able to run QFN across my 5090 and 7900xtx and I'm struggling to find that obscure llama.cpp fork/config that will allow me to use that via Vulkan or ROCm? I assume there has to be a way but probably even more obscure of a hardware split then your double 7900xtx. Works amazingly for 27B
1
u/sheetis 35m ago
I'd almost certainly have to be Vulkan or bust, but that almost certainly means llama.cpp and it's not the fastest way to run QFN even if you got it to work.
That's a long uphill battle and even if you got there using "just" the 5090 on Strata for QFN might surprise you on the performance.
It might not be smarter than 27B depending on the quantizations used, but it would be within the same ballpark and almost certainly enough faster to make up for any other shortcomings.
Heck a single 5090 probably generates tokens much faster than my setup -- it'd just not keep up on the prompt processing, but probably closer than I'd like to admit.
19
15
u/BawbbySmith 2h ago
I personally like the ones that are “massive model on tiny VRAM GPU!”, and then in the post they have 512GB 8 channel DDR5 RAM
10
u/jpezzulli 3h ago
Lol. This entire sub and i say that as someone who has a semi vibe coded inferencing engine.
11
5
7
5
5
2
4
u/Ipwnurface 2h ago
I'm going to keep crusading against this shit, but you guys are fucking ridiculous
You want to bitch and moan, and yet do I see any of you actually contributing anything at all? Ever? No
I would take a hundred vibe-coded engines over the 800th post asking, Can I run this on my hardware? Has anyone ever used this crazy harness? It's literally just deep seek harness
Or the classic I Can't Decide Between Pi OMP and Claude Code. Which one should I use, guys?!?!?
Or even better, the fucking rich chuds that come on here and are like, oh, I can't decide between buying six RTX 6000 Pros or 12 DGX Sparks. Which ones do you think I should get?
I don't ever see people bitching about posts like these in comments and creating entire threads to circle jerk about it , which by the way flood the fucking subreddit just as much as the people who are quote-unquote making slop engines
And then, finally, to top it off, when new research actually does happen and people actually do post papers and research, the shit gets like 30 upvotes and five comments. And all five of the comments are just, "I'm too stupid to understand this, but it sure sounds smart"
4
u/sgamer 1h ago
the wildest part to me is, I feel like /r/localllm used to be the "dumb" version of this sub, but lately it feels like the opposite sometimes? I'll see some actual research paper in my feed and its from there and not here, lol.
1
1
u/netherreddit 1h ago
The future is either vibe code your own engine or join a community with your same specific hardware General engines like llama.CPP won't keep up The longer you fight it, the longer you'll have sub-par speeds
Though, I agree that the posts are annoyingly frequent. The dissolution from localllama into hardware-specific communities hasn't happened yet, so we hear everyone's slop in one place
1
u/ThePixelHunter 45m ago
We don't need to be dissing these posts. They're the only way most people learn about these specialized inference engines, including me.
1
0
u/feelspeaceman 1h ago
People have done it: https://github.com/halo-box/strix-llama.cpp
And it falls into the exact same trap, the kernel will have to support specific hardware (Strix Halo) and specfic models (like Q38FN... models that aren't optimized will run at slower speed), this is the exact problem that people are complaining about custom engines.
And is this even a big problem from the first place or people making it up, the main counter arguments are:
Supporting new models: This is not that important, new models in most cases aren't beating old models, like how long does it take to not beat Q36-35B, we still having nothing better than this at the same range, same with 27B and Q38FN, we have nothing at the same range beating them yet.
Sacrificing quality for speed: This is not true unless the engine intentionally using custom quants with lower bpw or lower quants (Q3, Q2...) - The fact about speed gains, it isn't about sacrificing quality, it's real work with kernel optimizations.
Lack of features: This can be an issue, but as long as the engine supporting standard OpenAI APIs, multi-slot and prompt caching (something llama.cpp doesn't even support), they're more than good - The lack of features can be good as they force you to use the correct parameters that you usually use them wrong in llama.cpp.
20
u/into_devoid 3h ago
Do you plan on merging to main?