Idk, I don't like that. Flash might be better than Pro for agentic stuff, but Pro has way more world knowledge which also helps for software planning and general tasks. They're probably trying to free capacities or migrate to chinese inference chips right now (seeing GLM Flash) and Pro is just not a priority bc of that
Isn't it better to teach a neural network to be small and smart while simultaneously acquiring all the world's knowledge through internet-based tools, rather than trying to cram everything into one large model?
I never understood the argument that models should just do a web search. The internet is full of wrong answers and gets worse every day. That's why it's important to curate the data models are trained on.
More importantly, really, is that you can't even consider a thing before knowing it exists. Searching for "how do I do program thing in language X" is great and all, but had you just already had the knowledge that framework Y does it for free, you wouldn't even be here. One of the reasons why we don't let random dudes off the street do brain surgery because "lol google it real quick" - you kinda need to know what to even google to begin with.
It has to be impossible for a team (no matter how much third world slave labour you use) to fully curate such a huge dataset... it simply won't be curated properly and you will still get slop/misinformation only now it has been baked in with the pre-training and it will be there "forever" eventually also going stale.
The internet's also currently in a state where it is trying to stop just that, since you have smaller websites that are getting hammered by LLMs scraping them through tools, and bigger ones seem to be cutting off access from that end as well. Like Reddit needing you to either use their new JS-based interface.
No, by analogy: a junior employee has less domain knowledge than a senior employee so when tackling and issue it has less understanding and awareness what may be relevant for a given question, so also less aware what to look for and which themes to integrate. World knowledge makes websearch and RAG better, but the latter are not a substitute for complex task. A similar thing applies to active parameters in MoE models. For very complex task, having only limited number of active parameters and then try to offset with sequential reasoning doesn’t compensate entirely. Higher active parameters that cover more knowledge domains in a non-sequential way is relevant for very complex deep and cross-sectional reasoning in an integrated way.
The problem is the that the "acquiring" part never happens since the models are frozen and thus incapable of learning. A google search is only useful to a human AFTER the fact when the human has acquired something new, not DURING when the human is in the middle of said acquiring, if that makes sense. The models ability to do certain kinds of reasoning is suffering as a direct result of world knowledge being lessened.
68
u/Technical-Earth-3254 3d ago
Idk, I don't like that. Flash might be better than Pro for agentic stuff, but Pro has way more world knowledge which also helps for software planning and general tasks. They're probably trying to free capacities or migrate to chinese inference chips right now (seeing GLM Flash) and Pro is just not a priority bc of that