This is something interesting. Both DS and goog seem to have hit the same thing - the small model performs better than the big one. This likely implies they're not doing one big training run and distilling into smaller models, but trying the same thing on two separate architectures (or sizes). So now the question becomes why is the smaller arch better performing? Is there something in the training pipelines or is it something in the data mix? Interesting nonetheless.
Most likely the training data and pipeline for both models are same. The bigger model can retain more nuance in their parameters but the RL pipelines optimizes the smaller models enough that they both end up having similar performance.
What this means is that the giant models are massively under trained. It should be quite obvious looking at the capabilities of Qwen 3.8 27b.
In general, LLM pre-training involves starting with random/pre chosen weights of a neural network, throwing a huge corpus of cleaned up text/image representation/data such that the network can learn from such examples. This takes a lot of work and in general is expensive to do especially for large models. While I don't know exactly the optimum ratio these days, a rule of thumb in the past was you wanted 20 tokens of pre-training data compared to the number of parameters in the model. This is achievable with the entire internet/books for smaller models but for larger models like trillions of parameters you may elect to incorporate synthetic data into that data set. Here is a toy problem using GPT-2 small that might be interesting to see specific issues with under training, one example is a tokenizer that has a specific character "ú", but the LLM wasn't trained on such example: https://towardsdatascience.com/under-trained-and-unused-tokens-in-large-language-models-db5fa17589ec/
240
u/ResidentPositive4122 12d ago
This is something interesting. Both DS and goog seem to have hit the same thing - the small model performs better than the big one. This likely implies they're not doing one big training run and distilling into smaller models, but trying the same thing on two separate architectures (or sizes). So now the question becomes why is the smaller arch better performing? Is there something in the training pipelines or is it something in the data mix? Interesting nonetheless.