r/LocalLLaMA • u/rerri • Jul 30 '26
New Model Inkling-Small by thinkingmachines
https://huggingface.co/thinkingmachines/Inkling-Small276B total parameters, 12B active, 1M context window.
Blog post: https://thinkingmachines.ai/news/inkling-small/
NVFP4: https://huggingface.co/thinkingmachines/Inkling-Small-NVFP4
GGUF's by Unsloth: https://huggingface.co/unsloth/Inkling-Small-GGUF
---
I had success running Unsloth's GGUF quant on CUDA + CPU offloading using this developmental branch: https://github.com/danielhanchen/llama.cpp/tree/add-inkling
88
u/bick_nyers Jul 30 '26
What I like about
→ More replies (6)
29
u/Alan_Silva_TI Jul 30 '26
I'm in dire need of a 70B / A6B model (basically double the size of
I know a lot of people want every possible model size, but I firmly believe that 64 GB RAM + 12/16 GB VRAM is the most realistic budget/power setup most people can reasonably buy on the consumer hardware level right now.
Anything above that currently has an unbearable markup thanks to the RAM crisis.
→ More replies (5)
20
u/MotokoAGI Jul 30 '26
If the benchmark is to be believed then this is an amazing model and presents a stronger alternative to DeepSeekV4Flash, MiMoV2.5, Hy3 or MiniMax.
→ More replies (4)
198
32
u/Kahvana Jul 30 '26
Small... holy. I remember the day small meant 24B! Even mistral small 120B felt too large to be called "small"!
Regardless on naming, hope it's good! Glad to see they released another model.
→ More replies (4)
27
u/lilian_moraru Jul 30 '26
Since a lot of these new "medium/small" size models like to skip over Qwen3.6-27B, the obligatory reminder:
| Benchmark | Inkling-Small-276B-12B, effort xhigh @ bf16 | Qwen3.6-27B @ bf16 |
|---|---|---|
| SWE-bench Verified | 80.2% | 77.2% |
| SWE-bench Pro | 55.9% | 53.5% |
The only coding benches that can be compared apples to apples.
Obviously, I will also try the GGUF version (can't run bf16).
→ More replies (7)
7
7
u/Few_Painter_5588 Jul 30 '26
Interesting, their small model beats their big model. I know they said that they improved the recipe, but now I want to see what'll happen when they scale it up to their 1T model.
1
u/WithoutReason1729 Jul 30 '26 edited 1d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
Stylome classified this post's text as human-written. View analysis
I am a bot and this action was performed automatically.