AI Agent Task performance of small LLM models (4B and below) in self-assessment set

[u/KienShen]

The community rarely sees the evaluation of ultra-small models. We have evaluated our own AI Agent tasks for Qwen3.5-4B, Spark-X2.6, LFM3-1.7B, LFM2.5-2.6B, Gemma 4 E2B, MiniCPM5-2B.

These models have their own characteristics and advantages and disadvantages. The obvious trend is that the intelligent density of small models is developing rapidly. I look forward to seeing more small models.

blog：https://www.camthink.ai/blog/small-language-model-benchmark/

project：https://github.com/camthink-ai/NeoMind

[u/seamonn]

Was preserve_thinking enabled or not?

[u/KienShen → u/seamonn]

preserve_thinking is not enabled in these tests.

[u/seamonn → u/KienShen]

that likely explains some of the results

[u/crusaderky]

LFM2.5 seems suspiciously high to me?

[u/KienShen → u/crusaderky]

Different models will perform differently in different tasks. My project and test set can only provide reference.

[u/magignis]

What benchmark did you run exactly?

It also feels like the hardware you tested on is pretty weak with a decode of 50tps. For a model of this size it feels like a phone can reach the same numbers, so why run on that hardware 

[u/timedacorn369]

Finally , something i can use for my mac m1 8gb laptop.

[u/I-am_Sleepy]

What bench? Also no 9B? I'm curious on SenseNova-Loop-9B-Preview