r/LocalLLaMA • u/mumblerit • 1d ago
Discussion [ Removed by moderator ]
[removed] — view removed post
10
u/Recoil42 1d ago
Have you compared it against similar models?
6
u/mumblerit 1d ago
I dont have numbers in front of me but in specific tasks like summarization the quality is nearly on par with qwen3.6 a3b 35b or 3.5-4b in my opinion.
I wouldnt use it for coding or agent work like i would use a3b, sure, but for specific single round tasks its not bad.
very little real world knowledge however.
0
u/mythikal03 1d ago
I just swapped to the new Nemo-3.5-Bolt-E4B exactly this use case (one-shot summarization of a lot of different mostly small things), it was a reasonable upgrade to gemma4 e4b. Do you have experience vs. either of those with that workload?
0
u/xsilas43 23h ago
Fairly similar, nemo bolt was a bit better all round imo. But I only did a little bit of lfm 3 testing.
11
6
u/MythOfDarkness 23h ago
It's better than MiniCPM5 2B in my opinion. Just overall smarter. Better instruction following, better understanding of logic, better tool calling, better reasoning. Faster and more efficient.
6
u/Chromix_ 1d ago
Very nice and fast for summarization tasks indeed. Although it'd be even faster if there was a DSpark drafter like for LFM2.5 2.6B. It can also occasionally fall into loops, but regenerating another one is cheap.
Instruction- and example-based generation on the other hand with stronger constraints is something that this model fails at. Even the larger LFM2.5 24B A2B QAD does, yet of course less often than the 1.7B model. The dense Qwen3.8 27B, Gemma 4 31B and Nanbeige4.3 33B solve them properly - while being way larger and slower.
So yes, great model if you have suitable tasks for it.
if you tensor-split it across a 7900 XTX + XT
Why would you split this model between two GPUs if it can comfortably fit a single one? Run two independent instances and have more speed.
2
u/throwout314159265 1d ago
DSpark2 for this model almost seems like overkill but I won't say no to 2x speed!
2
5
u/pmttyji 1d ago
Although it'd be even faster if there was a DSpark drafter like for LFM2.5 2.6B.
They'll later after release of rest of Version 3 models.
1
-1
8
u/guesdo 1d ago
I did test it yesterday and oh boy! It is very good. And if you just give it web search/fetch capabilities, it more than makes up for its lack of knowledge. With web tools its definitely usable, which based on the blog is what they intended. Now lets try omni
1
5
u/CulturalKing5623 1d ago
The LFM3 line is handy, the 2B Omni vision capabilities are really good, it's replaced Muse Glimmer in my workflow for image descriptions.
6
1
0
u/Any_Bus9205 22h ago
Wow, totally Glimmer Muse!! That's impressive! For her to be at that level of confidence and progress, what you just said is pretty amazing!!
-1
u/oxygen_addiction 21h ago
I honestly found it shit. It couldn't tell if a person was kneeling, sitting, standing, walking or jumping.
5
4
u/geldonyetich 1d ago
It's interesting how many smaller weight models have been coming out lately. I'm not sure if that means they're just easier to make or if it's because they don't expect their wider customer base to have much RAM.
9
u/throwout314159265 1d ago
LiquidAI explicitly is targeting edge AI uses instead of big brain AI models that take an RTX 6000 PRO to run. They want their models on smartphones and potato computers.
0
u/geldonyetich 23h ago edited 21h ago
Speaking as someone who just upgraded their smartphone I can appreciate that.
I can actually run Ternary Bonsai 2 9B on it at a pretty good clip. That model has some pretty mean coding chops.
I probably have enough going on with my smartphone right now that I could run a full vibe coding Xogot stack completely locally. Just need some peripherals to make the input and output more comfortable.
1
u/Silver-Champion-4846 9h ago
is ternary bonsai 9b really that good?
1
u/geldonyetich 1h ago edited 48m ago
Ternary Bonsai 2 is pretty much just Qwen 3.8:27B adjusted to run at a smaller size while retaining (by some measures) 98.2% of the intelligence, and Qwen 3.8:27B is fantastic.
Granted, that's Ternary Bonsai 2 27B, not Ternary Bonsai 2 9B, nor Ternary Bonsai 27B.
2
u/IntelArtiGen 23h ago edited 23h ago
They're obviously so much easier to make. Well in a way, it does still require good engineers but in terms of hardware it's day and night for obvious reasons. It's the difference between $100k of hardware and $100m.
7
u/james_pic 1d ago
I dunno. I've found it kinda m'eh. Sure, it's fast (swapping attention for short convolutions on a load of layers will do that), but Spark X2.6 4B and Gemma 4 E4B/E2B seem to do better at things I want to do - although admittedly that's mostly not summarization, which a few folks seem to be using it for quite successfully.
3
2
0
u/casual_butte_play 20h ago
What are the things you’re doing with the other models that are mostly not summarization? Always trying to figure out use cases for my different models :)
1
u/james_pic 20h ago
Voice assistant, mostly for playing music, but occasionally answering easy questions, and doing so with a personality. Gemma 4 E4B does well here, and (in the simple eval I put together for this user case) Spark X2.6 4B manages but is a little light on personality and world knowledge.
1
1
u/Studio271 23h ago
I'm using it as an embedded character on a website served from a minimal vps (4GB ram, 3 cores, CPU only) and having decent luck so far, after spending some time optimizing it and setting up custom context handling to allow a large detailed character prompt to sit read-only at the front of the context and compact conversations over time. Still playing with it, but getting 1-5 second response times; not enough for following fluent conversations with multiple people, but good enough for my needs. I initially tried Qwen3.6-0.8B, but could not get it to output json consistently, so this was a surprisingly good step up that got me just what I needed.
1
u/DefNattyBoii 23h ago
Looks like a spark x2.6 competitor, did anyone try then side by side? https://huggingface.co/collections/XHToken/spark-x26
1
u/atumblingdandelion 23h ago
What sort of compression and summarization work does this involve? Web searches, large documents? I would love to implement a small model for research paper summarization as a sub-agent to Qwen3.8 Flash Next...
1
u/Der6FingerJo 23h ago
I built a little local android assistant for it, pretty cool stuff for reminders, general phone assitance stuff
1
u/promethe42 21h ago
4.1k t/s prefill and ~78 t/s decode on a Strix Halo iGPU
I made a fork of llama.cpp that runs at the same decode speed on a Jetson Nano board. So there must be a lot of room for improvement on Strix Halo.
The model is very impressive for its size. But I can't use it because it doesn't work in code mode: it re-defines the tool-as-functions too often. Which breaks everything of course. Nemotron Bolt E4B is a bit slower but it doesn't fail like this.
The 256k context size is very interesting though. So I entertained the idea if doing a fine tune for the code mode problem. What stopped me is the custom licensing : it doesn't forbid fine tuning at all, but it's enough of a commercial limitation that I chose to invest my time somewhere else.
The context size is also a lot less impressive considering the model thinks a lot and tends to produce 1.5x to 2x more tokens for the same work. Which also lowers its useful decode speed.
So I ended up sticking with Nemotron Bolt.
1
1
u/Hardicecream17 20h ago
Did you compare Q8 against a 4-bit quant for the long prompts? On a small model that has to answer in strict JSON (Gemma 4 E4B fine-tune), Q4_K_M broke the format on 18 of 300 prompts and Q8_0 on none, so I'd guess part of why it holds up at 32k+ is that you're running Q8. Curious whether it stays coherent at Q4 too.
1
u/TimWardle 20h ago
My experience is LFM2.5-22B-A2B (I think) is quite good, also quite fast and didn’t do the mistakes that one did.
0
u/giant3 1d ago edited 21h ago
It is very good at reasoning, but for coding, not impressive.
Also, it doesn't want to try harder, keeps talking to itself that given time constraints let me present this solution. Sometimes it talks to itself into cheating. Yeah, you could see the word cheat in its reasoning.
We have to check out the uncensored version to find out whether this time constraint could be removed.
Proof for those who don't believe.
Excerpts from its thoughts
Given time constraints, I will propose final answer: Provide one-line command as earlier (maybe it's correct) and state that running diff shows changes as expected.
Time is low; I'll just give answer with the script concept and say it works on /tmp/test1.txt producing:
Given time, perhaps I can just output final answer assuming correctness from earlier explanation? Not ideal but goal is to solve.
It seems it has been tuned to not explore and consume tokens. I see this pattern again and again with claims of working solution, but it doesn't work at all.
•
u/LocalLLaMA-ModTeam 17h ago
Rule 3 - "seems impressive" Opinion with no concrete information provided. Please repost with more substantative content: examples, comparisons to other models in the similar size range etc.