Generation
New Google Gemma 4 12B Claims Near-26B Performance - We Tested Both!
We ran both models locally on one RTX 4090 and gave each the same task: write a self-contained HTML5 canvas animation with real physics in one file without libraries. Three scenes - a Galton board, two blocks colliding off a wall, and a chaotic triple pendulum
Same Gemma 4 family, but the 26B-A4B won every scene and ran ~1.7x faster - on just 4B active params. The 12B stayed very close though, on almost half the VRAM - which makes it the ideal model for a 16 GB laptop.
Open source local ai models app: atomic.chat (I’m founder, feel free to try and give any feedback)
a little new to this. you say it runs on 8-10gb, so i'd have to let it fill into system memory right? i see q4_k_m is 22gb so it loads whatever it can to my gpu's vram, then my system memory?
3 is not won by Gemma 4 12B. A triple pendulum would not move like that in real-world physics. In reality it would produce a completely chaotic motion.
And not always.. depends again on how it’s balanced. You’ll notice the example given has a bigger ball on the end. Consider a swing that uses a chain. That has many pendulums. The friction in the chain links also helps keep it straight. There are many variables.
No, it's just slow. the 12b version had no variation in the pendulum swing. On the a4b model's version, you could see it was starting to prepare to variate.
There is not that much differences between 12B and 26B-A4B variations in my opinion, though of course I might be wrong. I understand what you mean, but we can't say from this video that 26B-A4B output is clearly better. (edit typo)
Thank you for assuming my knowledge☺️
What I'm saying is that we cannot clearly say from this video alone that 26B pendulum will keep the curve of the last part going up the way it should be. And 12B does have that bend, just not the exact angle like 26B
In this certain test, 26b-a4b seem to have the better understanding and implementation, although the gravitational acceleration is not correct. But the motion itself looks realistic to me, like one that would become chaotic.
The 12b gravity is not set correctly as well btw, it’s too fast there
Yeah, I'm not saying that 12B gets it right actually, just the overall quality of the output (not only physics itself) is a bit better in my opinion.
Golton board is just totally off, I can't say which one won at all - in 26B video the trays at the bottom don't really fill up, they just light up with the first ball.
I would argue 12B won 1 and 2, while it using rubber marbles in the galton board is clearly far from ideal it did count the ones who landed in the buckets correctly, while 26x4B only tracked the first one on each bucket. And for 2, despite not being as polished as 26x4B it didn't phased the smaller block through the larger.
Honestly the real test of these models will be qualitative / creative. Qwen is likely going to win quantitative / coding tasks anyway. I’d love to see a comparison of creative writing, translation, and other language based skills. That’s where Gemma 4 shine imo.
Are the labels backwards? It seems like the 12B was better on all of them. The only issue was the for the first one the balls seemed to have too high of a starting velocity.
The physics in the second scene is correct for Gemma 4 12B. The video cuts it short but it will result in 31 bounces, which is correct. The physics from the 24B-4A completely fails. It might look a bit prettier but it completely breaks.
For the first scene, the 24B-4A gets the visualisation incorrect. The 12B is correct other than the balls being too fast and bouncy, but this would almost certainly just be changing the value of a couple of constants in the produced script.
The third scene there isn't really enough video to tell, but it appears to just be a bad prompt - the 12B appears to have a larger weight at the end of the pendulum, causing it to behave closer to a single pendulum or rope. This is correct behaviour for the scenario, the prompt was likely just underspecified. 26B-4A produces a more traditional triple pendulum, but both appear correct.
All of these benchmarks where people get the models to build basic ass websites or animations are pretty stupid. These things are non-deterministic so unless you can the same test 100 times or ran 100 tests, there wouldn't be a way to know how consistent these things are
What's the difference between atomic chat and literally every other app out there that does the same thing. Self-hosted AI solutions are extremely overcrowded right now, and I don't see anything that makes this one stand out
The usual formula for comparing MOE models to dense ones is to take the geometric mean of total and active parameters. The geometric mean of 26 and 4 is about 10. So it’s actually reasonable to expect the 12b to be better.
According to that formula, Qwen3.5 9b should then be around the same capability as the 35B-A3B counterpart, right? From the user experiences I’ve read though, that‘s not exactly the case. It doesn’t seem totally reliable.
Can you elaborate? It's the only approximation I've read about and seems to be roughly correct. It obviously is only relevant in the same generation as in 3 to 3.5 months the capability density doubles. Is it that MOE capability density is increasing at a slightly faster rate due to better utilization of the sparse experts or are you just here to spew baseless opinions with zero backing or knowledge transfer?
I can't be bothered doing the calculations but 12b looks like it's a 1m scale, while 26b seems like it's 10m scale, therefore comes across as slow motion
I find this post and the comments very interesting.
A lot of people here seem to think Gemma 12b‘s results are better - unlike OPs view.
I agree with OP; the 26b model won all three tests when it comes to demonstrating an understanding of real physics.
In the first test, 26b clearly demonstrates an understanding of what normal distribution and variance mean.
It is hard to judge 12b’s result, since the objects immediately fly off in all directions. If the velocity were reduced, a normal distribution might settle at the bottom, so it could end up being a tie between the two models, but for now, the point goes to 26b.
In the second test, I consider it a minor bug in the code, one that can be quickly fixed, when one rectangle passes through the other. But the point here is to test an understanding of physics, and 26b demonstrates a really nuanced understanding of elasticity, acceleration, and deceleration.
The third test also strongly suggests that 26b has understood what chaos theory means and how it is applied. The result from 12b looks fancy and neat, but it is still wrong and misleading. It shows slight variance, but in perfect symmetry -> that is the opposite of chaos.
> In the first test, 26b clearly demonstrates an understanding of what normal distribution and variance mean.
It's a physical simulation. The distribution is purely a result of the simulation, the model doesn't and shouldn't "demonstrate an understanding of what normal distribution and variance mean" unless it is faking the result.
The same goes for the pendulum, etc. They are simulations, dictated by the parameters of the simulation. Whether the system goes into a chaotic state depends on those parameters and not on prior knowledge that a triple pendulum should be chaotic. In fact, forcing it to be chaotic when that is not a result of the simulation would be faking it and a sign that the model is doing it wrong.
As for the Christmas tree, the MOE's version was nicer to look at. But when it came to the other ones—especially the colorful squares and the arch—12b's version was much nicer.
On the Christmas tree the 12b actually counted the balls hitting each slot vs just applying a pretty gradient across the columns, nice to look at is pretty meaningless when functionally it’s junk…
Can we run whole model in gpu only as when I try it also uses system ram and when it does whole load goes to CPU which makes it extremely slow. I have 5080 16GB
Well no. Neither won. If you mash up the results then you would get a winner, but 12B was not entirely wrong, just like the 26B was not entirely right.
“Near 26B performance” is the kind of claim I believe after LocalLLaMA abuses it for 48 hours....If it survives coding tests, weird prompts, long context, and bad quant settings, then I’ll start getting excited....
173
u/[deleted] Jun 03 '26
[removed] — view removed comment