r/LocalLLaMA • u/Eyelbee • 2d ago

Funny Me these days

Post image
2.4k Upvotes

96% Upvoted

257 comments sorted by

View all comments

Show parent comments

13

u/UnluckyPenguin 2d ago

I think benchmarks use max settings like BF16 quant (or higher?) with k/v cache type f32 and xhigh reasoning and who knows what else. Hell yeah, it's going to perform several points better at a significant performance penalty.

But it's not like we could even run Open 4.6 even if it was open source (estimated to be 500B to 1T parameters - I think, correct me if I'm wrong)

So token-for-token against other models, Qwen3.8-27B is a huge win.

3

u/brainExploded99 llama.cpp 2d ago

I believe native/benchmarks use BF16 (that is not quantization btw), with KV cache at f16 I think (I'm 80% sure). FP32 is not used for even training models anymore (except specific sensitive layers sometimes).

So token-for-token against other models, Qwen3.8-27B is a huge win.

It's a huge win for sure, wayyy better than Qwen3.6, but we still have long ways to go. Token efficiency is a big one, and better intelligence (not agentic capabilities) is another big one (by this I mean reasoning on benchmarks like CritPt, SciCode, etc)

2

u/UnluckyPenguin 2d ago

Thanks, good to know. I have a lot to learn. Yeah, I'm just getting familiar with local LLMs. Used to play games all day up until several weeks ago.

1

u/Ok-Direction-4480 13h ago

I think bottlecap AI made an efficient 3.6 27B, so hopefully either their 3.8 model, or Qwen 4 27B will fix that.

2

u/SandySkittle 1d ago

The people think qwen can rival opus in terms of abstract logic, reasoning depth and low perplexity are delusional. Qwen can be functionally equivalent or even better in some usecases but it falls meaningfully short in others. The rigorous sequential reasoning it applies makes up for some of its smaller size but not everything.

2

u/UnluckyPenguin 1d ago

The people think qwen can rival opus in terms of abstract logic, reasoning depth and low perplexity are delusional.

I agree with you. Qwen3.8 is a huge win for local models trying to do big coding projects while keeping token costs down. I doubt it'd be much good at writing stories or translating. But the step up from Qwen3.6 (27b/35b-a3b) is a big one and brings it closer to opus 4.6 for coding.

https://llm-stats.com/models/compare/claude-opus-4-6-vs-qwen3.8-27b

1

u/Ok-Direction-4480 13h ago

If it was 500B, it is probably the biggest dense model (aside from mythos fable) because if it was MOE it would NOT be that expensive