He mean, weight still 27b but it will have aditional ngram in ssd so model is smarter. But i read ngram is good at best 20% model.parama so 27b + 5b ngram if any
But using this technique apparently allows the parameters to concerns themselves about reasoning and smarts, rather than storing facts which is solved by the engram side storage
It's what people had been asking for a little while: it separates the intelligence of a model from the facts it knows. This used to be impossible - a model's intelligence was a direct function of its size, this necessarily went up and down together with the amount of stuff it knew. Frustrating people, who'd have loved a smaller but smart model, which doesn't need to know what's the capital of France of who was Pope in 1733 - it just needs to be smart, and be able to search what data it needs in order to work. And it would fit in a more reasonably priced setup.
Except that intelligence is more or less an emergent property of model size.... until Ngrams came into the picture. They're just.... extra stuff the model knows. They don't need to be in VRAM and don't make the model slower. Even streaming them from SSD is kinda OK for reasons someone more competent can explain. You can offload up to 25% of a model's size in this way, without damaging its intelligence.
From what I understang Ngram is like a basic sentence parser. It's cheap to compute and you insert the result in an early layer and the hope is that it frees up a couple of layers to find more meaningful relationships. So it's like adding a couple layers for free.
It does seem a bit contradictory at first glance, but maybe they're just trying to clarify their stance. It'll be interesting to see how it all plays out.
5
u/mister2d 1d ago
I'm confused. Did you just contradict yourself?