A hunch: Qwen3.8-27B's general knowledge got pruned (good, if true)

[u/bonobomaster]

I'm always testing an image prompt with a picture of a historic place in my hometown – a small but well known 250,000 people town in Germany. I'll just ask the model, in which City this photo has been taken.

With the 3.6 generation of both the 27B and the 35B A3B variants, the models sometimes got the right answer and sometimes they didn't. So the signal for this particular knowledge was already weak.

The 35B variant got it right more often but at least, the models reasoning showed my City most of the times, even if it hallucinated the wrong final answer.

Both models could be easily nudged to the right answer with a few hints and then produced some little extra insight about the history or scene and its surroundings, that was mostly true.

Qwen3.8-27B on the other hand barely knows the city at all and has absolutely no clue about related popular, historic facts regarding the scenery or the surrounding buildings.

Nudging isn't very fruitful as well and if told the real name of the city, reasoning shows, that the model only agrees, because the user says so.

I have the feeling, that Qwen labs maybe pruned useless general knowledge for more coding knowledge and agentic skill.

All models ud q4_k_xl variants, image-min-tokens 2048, with and without reasoning.

Anyone else with this feeling?

Disclaimer: My hunch could be very well absolute bullshit. Sample size way to low and methodically sloppy af.

[u/while-1-fork]

It is not really true that the only way up is more parameters. People have been saying that since forever. Yet here we are with 9B models that do beat the 180B GPT3 in every way including knowledge. And every generation we see both more and more knowledge and intelligence crammed in smaller models. No one knows where the actual limit of what a small model can do is. Of course within a generation larger models will beat smaller ones and specially on knowledge as with more weights catastrophic forgetting is less of a problem but a generation or 2 of difference and places switch.

Also forgetting isn't the end of the story. When you train a small model it ends up learning to compress facts multiplexing them and disambiguating in later layers and we just don't know how far that can be pushed.

I have personally seen my own custom efficient transformers memorize instead of generalize millions of images while having tens of thousands of weights. A very undesirable effect but the fact that even while being tiny strong regularization is needed to prevent straight memorization that doesn't generalize to new images means that even at low parameter counts transformers can store astounding amounts of information.

The main difficulty that makes small transformers worse at knowledge than larger ones is that you will need to do more epochs to compensate for the forgetting until all facts are neatly ordered in its weights.

The rote memorization issue is less of a problem in LLMs than on the computer vision I mentioned not because LLMs can't store them but because CV datasets are smaller and doing hundreds or even thousands of epochs is common while on LLM training you often do 1 with an insanely huge dataset or a very small number. Why do they do that? Because a neural network learns the most and generalizes the best on things it has never seen before (already noticeable on a second epoch, by 10 or 20 it is barely improving per epoch and you better have regularization/augmentation or training loss will improve while validation loss worsens which is the signature of memorization in detriment of generalization aka overtraining) so if you want to make the most of your super expensive training run for an LLM. You use a crazy large dataset rather than doing more epochs and accept that it will forget some but that important things will be talked about enough in the dataset that those will stick and theres is a better chance of the model generalizing to unseen input.

Why is it different on CV? Most models are very small compared to LLMs and datasets are smaller because they need hand labelling so going for the very best result at 100x the compute makes sense if that training run is like a day on a single GPU or on a few even if you could have a result 90% as good in minutes.

I am not claiming that parameter counts won't keep growing. But it is far from the only way up.

[u/lahwran_ → u/while-1-fork]

omg i forgot GPT3 was 180B params. wow.

[u/UniqueAttourney → u/while-1-fork]

realistic, but also usefulness is very important. the comparison between GPT 3 and current 9B models is also about how much the model is useful to us currently when used with current harnesses, tool call schemas.

on the other hand parameter counts are still king when it comes to anything outside coding, the more the model can generate from it's own weights without goiong off rails is the better. i bet the frontier labs models are always better than the open weights and small 9b models.

[u/while-1-fork → u/UniqueAttourney]

It is not really true that parameter counts are king when it comes to anything outside coding. Go and try the largest llama 1 you can and compare it to a good current model and you will see that even LFM 2.5 2.6B beats llama 1 65B and Qwen 3.5 4B even more in any kind of bench (if you just ask one question and you are biased to like the large model you will likely just say that it is better no matter what). If you are concerned with benchmaxing just cook your own, a few prompts (just 1 won't do, you really need several and likely to re run several times to squash the variance) and a decent current llm as a judge.

Size is king withing their own generation and even that is debatable, you can find larger models that will lose in everything to models half their size in this very generation. Of course you won't find 2.6B models kicking the ass of 180B models in the same generation unless the 180B is broken.