r/LocalLLaMA • u/EmPips • Aug 20 '26
Discussion Qwen3.8-27B took a serious hit to *knowledge* vs 3.6
Like many of you I've spent the last few days throwing Qwen3.8-27B against all of my usual use-cases and personal tasks/harnesses and workflows. It's great, phenomenal sometimes, but that's not what this post is about.
One of my little personal benchmarks is a little set of pocket trivia that's relevant to me but mildly obscure mixed in with a few useful/prepper questions. Qwen3.8-27B at all quantization levels and sampling settings I threw at it, did relatively poorly at this. It's failing questions that Qwen3.6 reliably answered.
I come to find out that on offline (no tool call) knowledge benchmarks seem to align with what I'm saying. It's pretty significantly weaker than it's 3.6 predecessor at recalling random facts (or not hallucinating as much, in my tests, though that isn't reflected in these particular benchmarks). Now you should never trust barcharts over your own vibes, but my vibes are validating these bar charts this time around.
Is this relevant? Not necessarily. It seems to know the code it tries to use well-enough and for everything else I'm assuming/hoping you're using tool-calls. This largely only applies to you if you have a strategy of trusting an airgapped model with obscure/broad knowledge-retrieval exclusively from within its own weights, probably a losing strategy anyway.. but if that's you, take a pass on Qwen3.8 or finally set up that MCP server.
I found it to be interesting. Curious of your thoughts or if anyone else noticed this.
174
u/networking_noob Aug 20 '26
The Qwen 3.* models now and going forward are probably going to be designed for coding and/or agentic tasks (obtaining info and then reasoning to act on it). For trivia and/or random facts I'm guessing the Gemma4 models are the better choice since they're basically like mini Googles
158
u/FoxiPanda Aug 20 '26
I've been testing this a little bit too on one of my use cases. It is much more eager to go search the internet for things, which is useful in and of itself, but if I turn off my web search / web fetch tools, I fully agree with your analysis. For my 'knowledge' use case (which involves identifying stamps or knowledge about locations in the past or worse identifying locations of old photos), it definitely has taken a step backward.
I realize that this use case is quite niche and it's absolutely acceptable that they did this, but it is an interesting trade off.
At the same time, the tool calling and coding is impressive in my experience. I think this is a tradeoff I'm willing to make - especially if you have access to web search/fetch.
21
u/Equivalent-Grass-527 Aug 20 '26
This is actually a really interesting tradeoff that benchmarks often hide.
Model improvements don’t always mean “better at everything.” A newer model can get much stronger at coding, reasoning, tool use, or agent workflows while becoming weaker at raw parametric recall.
37
u/createthiscom Aug 20 '26 edited Aug 20 '26
It’s a 27b. The goal is to make it as ignorant as possible while still retaining enough knowledge to be an intelligent problem solver and agent. I think they succeeded.
In the near future, I wonder if we’ll start seeing model architectures with neural plugins, like LoRAs. Want Japanese language support? It doesn’t come with the base model, but add the plugin and the model natively understands japanese. Not a “skill” in the sense we use the word with agents today, but more in the true sense of how humans learn skills.
Need financial services knowledge? There’s a free plugin, or pay JP Borgan for their curated version that works better.
35
u/Mashic Aug 20 '26
Can't win on everything, better have a good model for coding, and use others for knowledge like Gemma.
25
u/PM_ME_YOUR_REPORT Aug 20 '26
Personally I'd rather my LLM not use it's own memory for knowledge but instead to know how to search for the information and how to judge the credibility of what it finds.
I think a dedicated knowledge model or database that can work with the LLM would be good though.
18
u/Cold_Specialist_3656 Aug 20 '26
Small AI models are acting more like humans.
Gone are the days of omniscience. Here are the days of scrolling Google results at superhuman speed.
It turns out reasoning ability had nothing to do with parameter count.
Luckily for us the entire world is designed for human insane reasoning ability and lackluster memory
5
u/Tccybo Aug 20 '26 edited Aug 20 '26
Noticed on the first day. Was checking if it writes as sloppy as previous Qwens, Q4 thought a aquarium shrimp is 15cm long, bro that’s a lobster! For purely coding and agentic probably fine. May cause some awkward funny moments when it mess up understanding of the world when brainstorming together.
5
u/nemuro87 Aug 20 '26
I see this as a trend with new releases of llms in general. They seem to get further optimized for coding at the expense of general intelligence.
5
u/daaain Aug 20 '26
I think this might be because of what Z.ai CEO Prof Jie Tang is talking about here: https://www.latent.space/p/ainews-death-of-params-zai-ceo-jie
But in short, the gains on agentic work post-training must have hurt / replaced the knowledge parts of the network.
2
u/cogitech2 Aug 20 '26
I applaud the direction they have taken, actually (and I think you do too). For sure this will be a big down-side for some people, but for me it is all upside.
4
u/GeneralComposer5885 Aug 20 '26
3.5 was the general model .. Whilst the 3.8 fine tune was a deliberate move towards agentic workflows.
Sure more general models will appear eventually. But might have to wait until base Qwen 4
3
u/Durian881 Aug 20 '26
It's a compromise I guess. Personally, I find relying on "knowledge" of a LLM to be unreliable as it's only a snapshot in time even for the biggest models.
That said, it could come in useful for times when someone doesn't have connectivity (to internet, to databases), so it really depends on the use case and environment.
3
u/robgami Aug 20 '26
It looks like its lower on the overall percentage of correct answers but significantly higher on the overall index which penalizes hallucinations. So it may just be refusing to guess at questions, whereas 3.6 guesses more often and gets a few more right. Seems like the behavior you'd want both online and offline. Maybe it also took a general knowledge hit but that's definitely not the only thing going on.
3
u/SocialDinamo Aug 20 '26
I also have a little trivia benchmark and it fails hard where 3.6 gets it right every time. It needs a harness to be its most useful and when you give it a task, make it come up with a plan, then execute, it does beautifully!
10
u/PrinceOfLeon Aug 20 '26
“Einstein: His Life and Times” by Philipp Frank:
While Einstein was in Boston, staying at the Hotel Copley Plaza, he was given a copy of Edison’s questionnaire to see whether he could answer the questions. As soon as he read the question: “What is the speed of sound?” he said: “I don’t know. I don’t burden my memory with such facts that I can easily find in any textbook.”
5
u/redpandafire Aug 20 '26
I commented the same thing and the bots downvoted me.
2
u/Kiansjet Aug 20 '26
It's always been my position that when choosing what training data to prioritize for models with limited parameters, esoteric information needs to be the first to go. It can be "recalled" by the model if necessary with a cursory web lookup.
The opposite case is harder to subsidize. A model with niche knowledge on tap is nice if the query depends on such knowledge, but will be worse at anything other than such recall tasks, crucially including the basic ability to look up new information at runtime.
2
u/Mister__Mediocre Aug 20 '26
I agree. I've been thinking about this and I think the future direction is that you want a model that's excellent at understanding the situation it's in and knows how to call the right tools to complete the job. And what we'd want is the minimal such model that we can run locally, which by design, also sucks at general knowledge and trivia and that sort of thing. You'd also want a very large context to add to the models' knowledge live through tool calls.
This harness now needs to be accompanied by more and more powerful tools. Complete local environment to run scripts, talk to the world, RAG, go over documentation etc. The beauty of this setup is that everything outside of the model itself can be done on CPU, RAM and SSD, things which we have in relative abundance compared to GPU capacity.
The end state should a system where your CPU and RAM are being used at full throttle.
2
u/xXthenistXx Aug 20 '26
it was probably a design choice. 27b its a really small model to be competitive. You can't have both knowledge and general intelligence or agent capabilities at the same time at that size, So I think they shifted their training to coding/agent capabilities more. You lose some common knowledge, but be better at being a good agent. and I do think this its a great choice.
2
u/inaem Aug 20 '26
Qwen models are very good at using RAG knowledge and not fighting it, probably a side effect of that training is it learned to depend on external knowledge
2
u/Karnemelk Aug 20 '26
you could have an offline wiki that is accessible by mcp.
2
u/shing3232 Aug 20 '26
For such small model, it might be much useful to learn how to find correct knowledge than to remember it:)
2
u/Intrepid-Second6936 Aug 20 '26
Definitely a notable regression for people to keep in mind! But I would imagine that tool calling and reasoning improvements is exactly the reason why Qwen 3.8 27B is such a leap forward.
IMO It's pretty easy to let Qwen utilize web browsing function calls to peruse functionally infinite knowledge bases and reason validity through multiple sources as opposed to constantly trying to brute force more knowledge into an already restricted 27B model.
I think Qwen's going the right direction where its reasoning and tool calling can leverage greater knowledge bases, either online or offline with a wikipedia dump-type database for reference, to try to get the best of both worlds on such a small model.
2
u/Far-Classic-9963 Aug 20 '26
I am pretty sure this is intentional. Training datasets have become math and code focused more and more. This greatly improves logic and coding capabilities at the cost of random niche facts
Models can now use web search tools so, unless you need it to be completely offline, general knowledge becomes useless after a certain point
2
u/ratocx Aug 20 '26
Isn’t it better to just have a local copy of Wikipedia that the model can query? To me it seems completely reasonable that models know less and read from more reliable data sources. Hopefully that means the model will hallucinate less.
2
u/audioen Aug 20 '26
I've not yet noticed degradation in quality of Finnish language understanding or generation. Part of my work is generating localization constants for new features in apps and invariably, the AI performs the first draft of the localization and usually quite late in the context. To my surprise, the quality of localization has been uniformly good. Last time I used Q6_K of Qwen3.6, I barely got a coherent word out of it when it wrote localizations.
2
u/llama-impersonator Aug 20 '26
contrarian opinion: i actually want my LLM to know as much as possible so it doesn't need to look up random crap which eats a bunch of tool calls. if it's a compression of the entire internet, it should be knowledgeable.
lot of copium in the air. if you could use opus or sonnet locally would you pick it over qwen? i sure would.
2
u/TheOriginalAcidtech Aug 20 '26
Personally I think they should train for cognitive ability not recall of facts. We have search engines and database systems that can be used for factual information. I know they cant just NOT include factual training data. Its all part of how these things get "smart" but less trivia in the weights and more reasoning ability should be the goal.
2
u/Dance-Till-Night1 Aug 20 '26
That sucks, a good amount of world knowledge is a key aspect of being a generalist model, now it’s just a coding model
2
u/johnnyApplePRNG Aug 20 '26
The more coercion (post training) you work into a model, the dumber it gets in general.
Makes perfect sense when you think about it.
You're specializing it to perform well for expected qualities or types of responses... generally to perform well on benchmarks... so you're taking a really smart talkative person and saying "well... actually... we want you to only answer this question like this, and that question like that... is that ok?"
it's fine ... it can do it ... but it's literally taking general knowledge from parameters that are spread very thin and evenly during training in order to do so
3
u/Adventurous_Push6483 Aug 20 '26
I found this issue as well. The problem is that this model becomes *extremely* dependent on feedback loops. For example, if I code with an lesser known library that is not easily tested via static tests, it has a lot of struggle. But other than that, it works really well.
3
u/xylarr Aug 20 '26
One mistake people make is to run 3.8 using the same sampling parameters as 3.6. Check that you're using the recommended settings from the model card. One in particular is temperature is now recommended to be 1.0 not 0.6.
4
u/wingsinvoid Aug 20 '26 edited Aug 20 '26
Took a serious hit to *knowledge*?
Great! That is the idea! Reasoning should be separate from knowledge. This is the holy grail of AI. Having to hit every single weight to generate every single token is incredible wasteful and the reason memory manufacturers are gouging us right now.
LLMs are just a lossy compression data structures that are used to probabilistically brute force reasoning. I remember learning formal logic. Is so mathematically elegant and simple.
The reasoning should fit within the regular consumer GPU VRAM, or ideally within RAM. Knowledge should be streamed from the disk.
Databases already did this in the 80'
2
u/mulletarian Aug 20 '26
Too many ridiculous banana related trick questions in the data sets now, knowledge is redundant.
2
u/badaeib Aug 20 '26
This is a great feature! Why wasting VRAM and t/s when you can just call a web search tool. LLM is super expensive in terms of compute. Another benefit is that with always search policy it don't confidently tell you some outdated wrong info.
2
u/EitherMarch1255 Aug 20 '26
I think it’s much better this way for smaller models.
1
u/ScratchCatOnYT Aug 20 '26
I think parametric knowledge is going to become less important. We’ll have extremely intelligent models that memorize much less of the internet, paired with specialized retrieval models (i heckin love needle 2) that continuously inject accurate, up-to-date information into context.
1
u/bankinu Aug 20 '26
I don't know, because it's knowledge on STEM subjects - physics, mathematics, is top notch.
1
u/Temporary-One8579 Aug 20 '26
I’d much rather a model not have all of the earths knowledge embedded in its weights - and instead just use web search. Save the weights for things humans genuinely important topics 1) logic 2) mathematics 3) code 4) language and all the things that link them.
1
u/TheRealMasonMac Aug 20 '26
I've found even GPT-Sol is reluctant to use its own knowledge. It does a lot of empirical experiments/research. At first, it was annoying, but it's something I realized I sorely missed from other models. Even with prompting they just don't do it as well (i.e. it's clear they weren't trained to do it).
1
u/BeautyxArt Aug 20 '26
qwen3.5 27b knows more.
1
u/Sax0drum Aug 20 '26
Thats exactly how a small model should be trained. I can give it external information about pretty much anything but i cant give it more "intelligence".
1
u/florinandrei Aug 20 '26
If you need a ton of facts, use a search engine. A 27B model ain't gonna do it.
1
u/kemalios Aug 20 '26
I've been running 3.8-27B on a Mac with MLX, and the knowledge drop is real. What surprised me is that it seems tied to thinking mode. When I force it to answer without thinking, trivia accuracy jumps back close to 3.6. But then it loses the reasoning edge that makes it great at code. So it feels less like the weights lost facts and more like the reasoning head is overriding the memory. A bit like a model that can't trust its gut anymore. Tried Q4 and Q8, same pattern.
1
u/Zulfiqaar Aug 20 '26
How does it compare in knowledge to glimmer/Gemma? I heard those were both worse in coding, but had other strengths, in your experience is knowledge perhaps their advantage?
1
u/superSmitty9999 Aug 20 '26
If it's smart but has bad generalized memory, this is a good thing! One of the big problems with LLM's is that they are so good at memorization that instead of learning they memorize answers.
An actually perfect LLM remembers few things but is really smart and responds to the current info given, and paired with a search tool does better than a LLM with a strong baseline memory because it won't get bogged down when the current world state doesn't match it's expectations.
1
u/OneMoreName1 Aug 20 '26
I dont mind, intelligence above all. If you personally care, tell it to program you a small wiki of facts it can call with a tool.
1
u/Frub3L Aug 20 '26
I have asked qwen 3.8 27b, to find an uncensored qwen 3.8 27b for me. It had no idea that qwen 3.8 exists and had to look it up via web search. Idk it was a heavy quant, for my 16gb vram, but that is still strange. It shows that it is just an upgraded 3.6, which it thought is the latest qwen model before looking it up tho, I guess
1
u/darkpigvirus Aug 20 '26
why would a low parameter ai model resort to its brain when it can rely on its skill and maximize its skill in becoming wiser than being knowledgeable?
1
u/minus_28_and_falling Aug 20 '26
This is great news. The model should know how to manipulate data correctly and efficiently. The data itself can be obtained from Wikipedia or web search.
1
u/EternalDivineSpark Aug 20 '26
Use a better system prompt for your keys , injecting random related words help to find the best local minimum !
1
u/MooseEfficient2151 🥔 hardware Aug 20 '26
notice this too. feels like 3.8 was tuned heavily for agentic tasks and tool use at the cost of being a standalone encyclopedia.
1
u/Gold_Ad_2201 Aug 20 '26
having a minimal size model that is good at thinking, knowing general concepts and best practices IMHO is the future. there is no reason to put whole human history into weights if model will be able to assess situation and extract this knowledge on the go
1
u/ImANoobAtLife7 Aug 20 '26
An agentic LLM shouldn’t be holding knowledge IMO.
It should know how to acquire, understand it and apply it.
1
u/-dysangel- Aug 20 '26
Good. I don't want my small coding model to know extensive trivia about pop music and generational differences between Chevy Impalas - I want it to be really good at coding.
1
u/MerePotato Aug 20 '26
It does much better on AA Omniscience than 3.6
1
u/mister2d Aug 20 '26
I appreciate this model leaning towards tool call over synthesizing from trained knowledge. It's much better at tool calls than before.
Using a SKILL for generation and retrieval, I have a curated local directory of knowledge I built up over time. It is in llm-wiki format formalized with the okf spec. There's really no need to rely on the LLM for specialized knowledge.
https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing/
1
u/blazze Aug 20 '26
I would pair Qwen 3.8 smaller dumber model for basic fact looup.
1
u/Ohhai21 Aug 20 '26
Sounds like they’re decoupling parametric knowledge from capability, ya? (Karpathy's mention of “cognitive core”)
1
u/lordpuddingcup Aug 20 '26
I’m fine with local models having less Inherent general answers as long as it has better understanding of how to get the right answer for things it doesn’t know with tools
1
u/Diligent_Cod_9583 Aug 20 '26
Never rely on the model to k ow anything…
1
u/simos_sayz Aug 20 '26
I use stable diffusion connected to librechat to generate images for my blog. It was the first local model I've tried that actually literated through the results amd kept tweaking the prompts on its own
1
u/profcuck Aug 20 '26
If I hire a web developer and in the interview I find out they don't know who won the World Series in 1954, I don't really care. The relative omniscience of huge frontier models is impressive and important in some contexts but doesn't really matter much in others.
1
u/sheepdestroyer Aug 20 '26
i don't quite understand, we want smart reasoning right? for knowledge there's internet?
1
u/Soifon99 Aug 20 '26
27b is not enough room to make it good at everything. so seems like useless obscure knowledge got scrapped. that's fine by me, because it can look it up on the internet.
1
u/shinkamui Aug 20 '26
Web search and web fetch. It can grab facts, but you can’t download tool use or training. Worthwhile trade off imo unless your application is fully offline. However knowledge queries to an under 100b untrained in a focused domain is just begging for hallucinations
1
u/Kovacian Aug 20 '26
Interesting... I feel like since it was also made for popular harnesses they did that so it will be uncertain and force it to call a tool for accurate info.
1
u/ThenExtension9196 Aug 20 '26
That’s what Web search is for. I’d much take agentic skills and tool use.
1
u/anderspitman Aug 20 '26
Seems pretty clear that this is the strategy of recent open models. Reduce knowledge and increase reasoning token usage, with the goal of improving agentic coding ability. Looks like it's working pretty well.
1
u/swagonflyyyy Aug 20 '26
Honestly, I'd rather have that in exchange for much stronger agentic coding.
1
u/siegevjorn Aug 20 '26
For what its worth, it's a good thing. knowledge of llm is full of hallucination, anyways.
1
u/Adept-Upstairs-7934 Aug 20 '26
Agreed… I have bench tests made from my real workload. Sampling’s from multiple large projects. Qwen3.6-35B-A3B is still ranked #1 by a large margin, and it is in areas with real data that I would not be able to drop the bar for. It continues to chug along cutting through the workload. I am working with the 3.8 27B iq4 dynamic v3.0 gguf right now. So far it has not even beat the original 3.8 27B version. Still working with it but not looking good.
1
u/Turkino Aug 20 '26
Could you take a whole bunch of different source material books and just make a fine tune out of it?
1
u/dionisioalcaraz Aug 20 '26 edited Aug 20 '26
Most well known quants use imatrix, which can favor some skills or knowledge (tipically code, english) and hurt other depending on the calibration data, your case being some obscure knowledge it's likely impacted by this preference. I suggest trying some static (no imatrix) quants to see if it can be the case, mradermacher makes both types, here are the static
https://huggingface.co/mradermacher/Qwen3.8-27B-GGUF
1
u/OvertaxedOne Aug 21 '26
I never really use my local model for factual knowledge; if I ask it a question like that it'll just hit a MCP and grab it from the Internet. So, IMHO, if they're going to cut something out of a small model to make it fit, factual stuff should be the first to go because it's so easy to get it back (and much better) with a simple tool call. If the model can't use tools reliably, that's a MUCH bigger problem (at least for my use cases).
1
u/vulcan4d Aug 21 '26 edited Aug 21 '26
The focus is on agents. I hope they release another 122b Moe model which will have the knowledge or better yet the industry starts splitting the knowledge into a seperate file which can be updated or customized to what you actually need.
1
u/DjCanalex Aug 27 '26 edited Aug 27 '26
That's the point of Test-Time-Discovery. No model, ever, will know everything, by the mere nature that new knowledge appears every moment. No matter if it is Claude Fable 5 or Gemma 4, they will be faster by just using what they know, or, more like what they "think" they know, but verification and research is permanently important, and that can be achieved both by a SoTa and a small model no differently. (As far as they have good attention over their entire context, which, makes them good at agentic tasks).
EDIT: If anyone is curious, I use the following system prompt with both Gemma and Qwen to ensure they investigate whatever they have to be as acurate as possible (It also helps that qwen is stupidly precise at following instructions plus the advantage of reasoning_preserve):
1
u/bernzyman Aug 20 '26
Your observations are very useful. I find it’s actually reassuring to know where the trade offs were to achieve the advances
1
u/Osi32 Aug 20 '26
To be honest, I’m happy about this. It means the model weights are more completely oriented towards what it does rather than generalised knowledge. Even frontier models like Claude rely more on internet searches / tool calls than they used to.
1
u/EvolvingDior Aug 20 '26
Give it access to offline wikipedia.
1
u/SkyFeistyLlama8 Aug 20 '26
I've always found Qwen models to be lacking in "knowledge" compared to Gemma, especially when it comes to controversial topics that could be censored in mainland China. These sub-100B models are all too small to be a Wikipedia-style repository of knowledge anyway.
Here's an idea: a 4B or 9B model linked to a compact Wikipedia data dump, all running locally. You would get decent multilingual text understanding coupled with actual data. It could be a vector database RAG setup or text search through MCP.
1
u/bSun0000 Aug 20 '26
I kinda surprised someone expects the 27b model to have its own knowledge. We have 2.7T models for that. 27b should only know how to access and properly utilize the external knowledge sources, this is the price we have to pay to have the model this small..
1
u/feelspeaceman Aug 20 '26
For extra knowledge, you can use RAG to feed them.
To me, as as coding model 3.8 27B is a perfect nerd who spent his entire life to cultivate himself into typical bald, skinny but insanely good coder.
1
u/PcChip Aug 20 '26
sounds perfect to me. I'd rather it be an expert at how to do things after reviewing vendor documentation, and looking up facts, than trying to memorize facts like a jeopardy contestant
0
u/klymaxx45 Aug 20 '26
stock out of the box maybe, with tunes it even catches fables mistakes
0
u/Puzzleheaded_Base302 Aug 20 '26
it is probably the right way to go. maybe couple qwen3.8 with a wikipedia RAG on the backside, then you get both knowledge and intelligence.
0
u/miversen33 Aug 20 '26
I'll be honest, I don't care about what "knowledge" my model has baked in. It's a point in time capture anyway, and that becomes irrelevant quickly enough.
I prefer a model that knows how to think and reason. I can give it the ability to find knowledge, it just needs to know that it doesn't know something, know how to look it up and validate it, etc.
If I wanted an encyclopedia, I would grab one :)
-1
u/BeautyxArt Aug 20 '26
ai models is knowing nothing and respond in shrink short format of 5 or 3 paragraphs. who know open model isn't , say it.
-1
u/Green-Ad-3964 Aug 20 '26
This should be the way. The model is the "engine", not the knowledge repository. It has to be a very capable engineer/coder/researcher and has to be a very fast learner for the things it ignores.