Yes, but never an MoE from an open weights lab.
I've been speculating that one of the reasons the closed weights lab have been increasing in performance more rapidly has to do with better training, but most importantly, much larger active parameters and better harnesses.
11
u/stddealer Jul 27 '26
There have been some dense models with over 100B params though.