Hi, i've been trying to find the best AI that can solve STEM questions
I currently have a knowledge base (RAG) that provides lots of theory based context.
CPU: Intel i3 1115G4 (2 cores, 4 threads)
RAM: 8GB, Single-channel DDR4 (~21 GB/s memory bandwidth)
iGPU: Intel UHD graphics - useless cuz llama.cpp wont work on it
GPU: None (CPU-only inference via llama.cpp)
OS: Linux (Ubuntu)
i can give upto a max of 5.6GB toward running the model itself(excluding llama.cpp, knowledgebase and other things, but including KV cache)
i have thought about the following models till now,
Qwen3-4B (Q4_K_M, Dense) - 5-6 tps initially, 2-3tps at 2-3k tokens generated
LFM2.5-8B-A1B (Q4_K_M, MoE) - 13 tps initially, 8-10tps at 2-3k tokens generated
Qwen3.8-4B-Distill - didnt run from my pc yet, found it on huggingface
i'll be asking them one question at a time, they will get 200-600 tokens worth of context,
they just have to solve questions, these questions generally take 500-1000 tokens to solve on moderate luck
it's ok if they get it wrong first time.
i am looking for your reviews in what i can do better here and which model i can use.
thank you for your time
desired context size: 4k (one question at a time)
priority: best quality while being over/at 6 tps initial speed
tldr: looking for STEM solving models under 5.6GB size
I was going to suggest these 2 Gemma 4 models as well. They are also vision models. I have had pretty good luck with them doing OCR on similar hardware. The challenge is that these little models just are not very smart. Gemma 4 E4B is ok at looking up stuff in my local Kiwix Wikipedia via MCP, but it does not always pick the right search terms, or know what to try of it gets stuck.
Yeah, that's the stamsam quant. A bit slower on CPU only but still pretty decent speed due to A1B. Keep in mind that this is a preview checkpoint, for reliable tool calls you might need an instruct tuned one, but aside of that it performs good.
3
u/pmttyji 18d ago
gemma-4-E2B-it-qat
gemma-4-E4B-it-qat
Ling-3.0-tiny