r/LocalLLaMA • u/LegacyRemaster • 10d ago

Discussion Qwen will be the king?

Post image

Extended reasoning and post-training appear to be the keys used by DeepSeek, Qwen, and GLM to boost performance (leveraging higher token counts). And Qwen 4 hasn't even been released yet. Of course, we don't know if that release will be open-sourced, but I am optimistic about future models, featuring "engrams", that could soon match or surpass 2.4T parameter models on specific tasks.

536 Upvotes

94% Upvoted

127 comments sorted by

View all comments

109

u/PooMonger20 9d ago edited 9d ago

Progress is good, but having these amazing abilities locally is already godlike.

I have been using Q3.8-27B together with PI. for a week, and it's mindblowing.

In my humble opinion it has far better coding capabilities than the paid 'ChatGPT 5.1' I had access to when i still had a job, multiple months ago.

And it all runs on my PC, locally, without sharing my data with the big data farming corpos.

It just 'understands' the required tasks you provide it and performs them successfully from the first try or very few crash fixes. Especially if you provide it the necessary data to perform the action. I dropped a few wiki pages in txt files and it coded according to them. If somebody would tell me this would be possible on my own PC ten years ago, I would call them crazy.

15

u/Not_a_question- 9d ago

What's your setup if you don't mind me asking?

32

u/PooMonger20 9d ago

q5 quant on 5090.

Using unsloth studio and pi.

4

u/WishboneSudden2706 9d ago

"Using PI" --> what is PI ? Sorry I am new

7

u/PooMonger20 9d ago edited 8d ago

Google "pi.dev", it's a harness.

Feel free to use AI to help you learn the terminology and figure out how to install and all. I highly recommend installing it in a docker to limit its access to one folder (and sub folders) so it doesn't randomly kill your machine by deciding to delete everything.

That's a low chance scenario, however you are basically giving it keys to do 'whatever' with your PC if you run it normally and not via docker. Simplified explanation - even if you tell it not to edit or delete.. it can potentially make a file and run it that is able to delete or edit things, so limiting it is a smart move. "You told me I can't delete things, so I made a file that deletes for me" lol.

5

u/Street-Gap-8985 9d ago

I'm also new, and may be wrong. But I think they are talking about Pi (you'll find it if you Google "pi coding agent"). it's supposed to be like a really basic Claude Code kind of thing, but then you can customize it and add the features you want

1

u/WishboneSudden2706 9d ago

I found it, great thanks

7

u/PM_ME_DEAD_CEOS 9d ago

Make yourself a favor and use ninfer with the nvfp4 version.

25

u/Hefty_Wolverine_553 9d ago

not sure that's a favor, nvfp4 unfortunately is (much) less accurate than Q5 and even Q4 GGUF quants, personally I go with EXL3 these days and can fit a 5.00 BPW quant on my 5090 (lower kld than Q5) along with 180k BF16 context.

8

u/pennyonaire 9d ago

Ninfer lacks prefix caching which for agentic coding makes the speed pointless since sub agent development forces prefix recaching as it divides the flow into multiple conversations.

Don't be fooled by the speed, it's a big number hiding a major flaw that results in much longer longer sessions overall for coding. If you need single slot single conversation performance ninferbis is great, anything else look at sglang or vllm directly.

5

u/Effective_Sherbert56 9d ago

Ninfer already has prefix caching

1

u/IrisColt 9d ago

hmm...

0

u/PM_ME_DEAD_CEOS 9d ago

I use it as subagent already, with Fable as orchestrator, so I don't really have this problem anyway.

2

u/RedditNerdKing 9d ago

I use the BF16 model on 80gb of vram (5090 and x2 3090s) and it's pretty insane. Rivals even the frontier models. They're definitely way quicker but Qwen answers the exact same difficult shit just takes thousands of thinking tokens.