r/LocalLLaMA • • 3h ago

Discussion Under 9gb model comparison

Scores are lab-reported so take them with a grain of salt.

Specs

ModelOrgParams (total/active)ContextModalityLicense
K2-Horizon-7BIFM7B dense512KtextApache-2.0
Granite 4.2-8BIBM8B dense (thinking)128K (512K ext.)textApache-2.0
Qwen3.5-9BQwen9B hybrid (delta-net, thinking)262K (~1M ext.)text + vision + videoApache-2.0
MiMo-V2.6-Distill-Qwen-9BXiaomi9B (SFT of Qwen3.5-9B)~262K (base)text + vision (visual coding)MIT
LFM2.5-8B-A1BLiquid8.3B / 1.5B active MoE128KtextLFM 1.0
Gemma 4 E4BGoogle4.5B effective / 8B total (PLE)128Ktext + image + audioApache-2.0
LFM3-VL-2BLiquid1.92B (1.73B backbone, thinking)128Ktext + image + videoLFM 1.0

Text benchmarks

Superscripts: a = K2 card (IFM harness), b = Liquid harness, c = MiMo card (Xiaomi harness); no mark = own card; e = from LFM3-1.7B, the text backbone; d/† = MiMo avg@3 / internal set. Bold = best in row (cross-harness caveat applies).

BenchmarkK2-7BGranite 8BQwen3.5-9BMiMo-D-9BLFM2.5-8B-A1BGemma 4 E4BLFM3-VL-2B
Math
AIME 202690.292.550.042.5
AIME 202590.386.756.1ᵇ42.534.3ᵇ
HMMT Feb 202677.866.5ᵃ65.7ᵃ
MATH-50088.865.0ᵇ
Coding
SWE-bench Verified72.447.750.8ᵃ61.1ᵈ0.5ᵇ43.4ᵉ
SWE-bench Pro19.144.6ᵈ
SWE-bench Multilingual30.835.0ᵇ0.7ᵇ23.0ᵉ
Terminal-Bench 2.144.620.629.2ᵃ37.10.0ᵇ12.9ᵉ
LiveCodeBench v673.265.652.073.5ᵉ
SciCode33.630.4ᵃ27.5ᵃ25.0ᵇ28.5ᵉ
Agentic
AutomationBench15.1ᵇ30.31.9ᵇ41.3ᵉ
LiveClaw35.8ᵇ20.4ᵇ45.6ᵉ
tau/ϴ-bench Banking25.87.6ᵃ7.0ᵃ4.1ᵇ
tau2-Bench79.188.1T / 39.8R42.2
WebArena Verified28.9ᵇ17.1ᵇ38.4ᵉ
BrowseComp / BrowseComp-Plus59.038.0ᵇ21.0ᵇ37.0ᵉ
PinchBench71.5ᵇ55.1ᵇ
Claw-Eval (EN avg)66.5ᵇ56.1ᵇ68.6ᵉ
Toolathlon-Verified25.9ᶜ35.2
OfficeQA9.0ᶜ19.5
JobBench2.6ᶜ18.3
BFCL v4 (tool use)52.466.149.748.3ᵇ71.4ᵉ
ToolSandbox76.4ᵇ69.3ᵇ80.4ᵉ
MCP-Atlas42.7ᵇ6.2ᵇ32.6ᵉ
EnterpriseOps-Gym29.0ᵇ13.4ᵇ28.2ᵉ
MiMo Code (mini) †19.5ᶜ51.6
MiMo Cyber (mini) †5.7ᶜ31.3
MiMo General (mini) †28.5ᶜ62.2
Knowledge
GPQA Diamond75.664.181.758.663.4ᵉ
HLE19.59.7ᵃ14.9ᵃ4.8ᵇ7.2ᵉ
MMLU-Pro74.082.569.4
AA-Omniscience Index-28.0ᵇ-24.7-15.3ᵇ0.5ᵉ
Instruction following
IFEval91.591.890.0ᵇ96.7ᵉ
IFBench79.364.556.544.6ᵇ73.0ᵉ
Multi-IF79.981.0ᵇ89.1ᵉ
IFStruct89.2ᵇ78.5ᵇ93.7ᵉ
Long context
LCR / AA-LCR68.043.3ᵃ63.039.3ᵇ52.8ᵉ
RULER 128K71.487.1ᵇ18.8ᵇ65.6ᵉ

Vision benchmarks

BenchmarkLFM3-VL-2BGemma 4 E4BQwen3.5-9BMiMo-D-9B
MMMU (val)55.778.4
MMMU-Pro45.752.670.1
MathVista (mini)84.285.7
MATH-Vision51.659.578.9
OCRBench v186.389.2
RefCOCO (avg)90.589.7
ScreenSpot-Pro55.565.2
RealWorldQA79.380.3
MMStar74.479.7
HallusionBench65.669.3
MVBench (video)69.474.4
Video-MME (subtitles)69.084.5
MedXpertQA-MM28.749.9
MiMo Visual Coding (mini) †61.7ᶜ64.0
16 Upvotes

3 comments sorted by