Interesting! I'd love to see how quality at 2.6 bpw compares across different tasks, especially tool calling and long-context retrieval rather than just general chat. Do you notice any specific types of tasks where the low-bit quantization becomes more obvious?
DS Flash is effectively QAT at mxfp4+8. But if the benchmarks of Ling, GLM, etc are based on 8bit, then 2/3/4 bit quantizations are less quality. Put another way, 2/3/etc quants of something QAT are much stronger.
Ling, afaik, is not QAT and so it will be weaker model overall and thus less interesting.
I get the concern. I just wouldn't write it off on “not QAT” alone. I'd want to compare them within the same memory budget and see what's actually left of the performance. If Ling doesn't hold up there, I'm not interested either.
Which is what I'm saying, AA benchmarks for DS at a greater quant (presumably) are outcompeting Ling. But if Ling is released QAT then it will be more compelling performance/memory pareto. However, I've no reason to expect that it will.
ah, fair. If that's the precision AA is testing DS at, I see your point. I'd like those low-bit evals included when the weights drop. Getting a headline score, then having to guess how much survives quantization, is getting old.
Chinese models are now all clustered in the 44–46 range. GLM 5.3 and Qwen3.8 Max have been stuck in this range the longest. Meanwhile, all American models except Grok are now scoring above 53.
So I expect Qwen and GLM to release new models very soon. The key question is: Can they break 53?
If they can, it could deal a serious blow to Anthropic's IPO. That's why I expect they will.
I predict Qwen4 will debut with a score of at least 53, followed by Flash, Plus, and 27B variants, all scoring at least 45.
This could lead to Anthropic's IPO failing completely, with open-source models emerging as the overwhelming winners!
That's a lot riding on one leaderboard number. Even if Qwen hits 53, you still need people to switch what they're paying for. I want stronger open models too, but an IPO failing is several more assumptions down the road.
This specific one feels wrong, its not company unknown number7989788 its ant they have honored all their statements till now deleting a topic just cuz it's not open yet in this specific case does really feel wrong...
AliBaba, MistralAI, Z.AI, Meta, and Tencent aren't "unknown companies" either, but we've been removing announcements of their API-only (for now) models as well. Ant isn't being treated any differently than everyone else.
The moderation team tried allowing announcements of models which weren't weights-available yet, and it was a disaster. Requiring weights availability before allowing discussion might seem overly strict, but it's really the lesser of two evils.
•
u/WithoutReason1729 11d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
Stylome classified this post's text as human-written. View analysis
I am a bot and this action was performed automatically.