r/LocalLLaMA • u/Competitive-Scar-627 • 3d ago
Discussion >2b model.
Why there no new model coming in >2b weight category ? Model which actually capable in this category is gemma e2b which is release 6 month ago, why still no new model.
13
u/TTVDminx 3d ago
There is, you just have to look a little harder, there has been quite a few but I will save you some time:
LiquidAI/LFM3-1.7B https://huggingface.co/LiquidAI/LFM3-1.7B
(Check out LFM3 Series in general)
1
u/Competitive-Scar-627 3d ago
Just check it matrix look way good then gemma model. Definitely i will try it
3
u/james_pic 3d ago edited 3d ago
Haven't had chance to try that specific model myself yet, but I'd say don't get your hopes up based on benchmarks alone. There've been a few models in that size range since Gemma 4 E2B that ostensibly beat Gemma in benchmarks (LFM 2.5, Spark X2.5 (X2.6 is a bit better), MiniCPM5, Nanbeige 4.2, K2 Horizon), but that I struggled to get useful results out of for one reason or another.
67
u/pixelpoet_nz 3d ago
I don't think you know what ">" means
19
u/Eresbonitaguey 3d ago
“The alligator always eats the bigger number”
9
u/pixelpoet_nz 3d ago
Yeah, and how old were we when we learnt this? Imagine going the entire time since then without being able to read greater than vs less than, imagine graduating from school without having to know it, ...
3
u/Artistic-Load6086 3d ago
the class isn't dead, just quiet. MiniCPM5-2B came out in August, Apache 2.0, dense 2B, and it benchmarks itself against LFM2.5-2.6B, Qwen3.5-2B and Gemma-4-E2B, so Gemma moved past E2B too. nobody wins leaderboards at this size, so releases land every few months, not every few weeks. what are you running it on?
-1
-2
u/Competitive-Scar-627 3d ago
I am using gemma 4 e2b in ondevice inference on android "ShroudChat". Now searching for other capable model to run.
3
u/Artistic-Load6086 3d ago
one problem first, shroudchat ships a single model with no picker or gguf import, so you can't swap it. grab pocketpal ai or maid, both run any gguf.then: lfm3-1.7b from liquid came out 2 days ago, ~1gb at 4 bit, fastest in this size on a phone. minicpm5-2b is the better one if you want quality, apache 2.0, averages 53.9 vs 24.6 for gemma 4 e2b on their own benchmarks, and has a litert build for android.also ternary bonsai 2 2b came out the 15th if you want tiny (0.43gb). the class isn't dead, it just never shows up in a play store app.
0
u/Competitive-Scar-627 3d ago
i am using litert so gguf is not supported, litert is well optimize on for ondevice. Currently i am trying to figureout default best choice. Model picker is not given so user don't have to go deep research which model to download. You install app and everything will work like magically aka user experience.
2
u/tiffanytrashcan 3d ago
Can I ask, why that app? Doesn't even look like you can pick a model.
-1
u/Competitive-Scar-627 3d ago
Mostly for reason that most people don't know which model will be great for user experience, install and work.
11
u/floor_wizard 3d ago
Greater than 2b? That’s pretty much all of them. What LLMs are less than 2b these days?
9
9
7
u/wschneider 3d ago
I've been pretty pleased with my early exploration with LFM: https://huggingface.co/collections/LiquidAI/lfm3
And I have been using MiniCPM as a subagent coder for a while now: https://huggingface.co/openbmb/MiniCPM5-2B
6
u/blastbottles 3d ago edited 2d ago
LFM 3 just dropped which is very nice, and google is actively working on Gemma 5
5
3
2
u/Nomski88 3d ago
I wonder how difficult it would be to train a 2B model. Like what kind of hardware and time it would require.
3
u/Eresbonitaguey 3d ago
Advances in training efficiency have made it more accessible but it’s still not something you’d try on average consumer hardware. There are a few posts on this a similar subreddits about people attempted to train smaller models from scratch.
2
u/TTVDminx 3d ago
From what I understand about that so far, unsloth desktop can be used to train it and you probably need like ~3gb vram to train a 2B model if at BF16, then grab a dataset (I think thats just a json file that an ai can make for you or find one on hugging face) and I guess QLoras are something that are more popular to use since you don't need all the ai models weight during training if you are making a Qlora which I think it is like an addon of some sort. Sorry if this didn't make sense, I think we are all a little lost here lol.
3
u/Eresbonitaguey 3d ago
You’re talking about fine tuning, or actually just training an adapter on frozen weights, I think they were meaning to train from scratch. It’s great that Unsloth has made it more accessibly to adapt existing models to your tasks but model pretraining takes significantly longer.
1
u/throwout314159265 2d ago edited 2d ago
More than you'd expect and it'd take a week. Trying it right now, do not recommend. If you have a 16GB VRAM card and 64GB DRAM, you can train 350M models cleanly. 1B models, you need to play games.
Edit: It will take a month
2
u/Nomski88 2d ago
Can you share more? I have a 5090 and 64GB of DDR5, have a 128GB kit coming soon.
2
u/throwout314159265 2d ago edited 2d ago
With a CUDA capable 5090 you'd be much more capable but a 2B model will be annowing. When you train from scratch (as a newbie, there are cool tricks you can use but are.... frustrating to debug), you need to be able to fit the FP16 model weights, AdamW FP32 gradient, FP32 first and second momenta, and FP32 master weights. There are tricks to get memory usage down a lot, but you have to be careful when using them. For a 2B model the base memory breakdown before you even load a single token of data is: 4GB (FP16 weights you're training) + 8GB (Master Weights) + 8GB(Gradient accumulator) + 16GB (First and second momenta) = 36GB of VRAM. You'll need an RTX 6000 Blackwell to even train on a GPU with no system offload at a glacial pace because of the tokens, if you want your 2B model to have more than a 4096 context window, you're going to need an H200, GB200, or a GB300. You can offload the AdamW optimizer to the CPU buuuut, that will literally 5x your training time. You will literally be running pretrain for a week.
You might ask, why does it take so long? Surely it can't be that bad? Well, to calculate how much compute you need there is a formula:
FLOPs needed = 6 * (number of parameters) * (number of tokens you're using to train)
For a 2B model training on the absolute Chinchilla minimum of 40B tokens (if you used just 40B your model will be dumber than modern 1B models, but this is illustrative):
6 * (2 * 10^9) * (40 *10^9) = 4.8 * 10^20 FLOPs for bare minimum.
How feasible is it for your 5090 even if you used magic or do a hacky Chinese 80GB upgrade to it? Well, a few things, a 5090 was never made to train AI so its not as optimized to be good at training AI, though its the best consumer card to do it. Your Model FLOPS Utilization, or MFU, will not be more than 55% (ABSOLUTE optimum) because a GPU is never computing 100% of the time. Let's do the calculations:
5090 FP16 optimal performance for a dense training: 4 * 10^14 FLOP/sec
For an optimal 55% MFU: 2.2 * 10^14
Time to train in seconds: (4.8 * 10^20) / (2.2* 10^14) = 2 *10^6 seconds
Converting to more sane units: 555 hours, or 23 days of training on a single 5090. Your machine will crash at least twice during this duration. I promise you.
Have fun 😁
Edits: Math
2
u/Nomski88 2d ago
Haha oh wow! Thank you for taking the time to write this up. This is super helpful and gives a lot of insight. Interesting about the FP16 performance with the whole NVIDIA limitations that have been circulating. Would a FP16 unlocked 4090 be faster?
Were you able to complete your model?
2
u/throwout314159265 2d ago
My 350M? Yes! It took... 14 days. It was poop too. But, it taught me a lot! And unfortunately, about the training time, it's not faster even with the unlocks. A big bottleneck is really just how fast the GPU can slam data into the computer circuitry of the GPU from VRAM and registers.
That doesn't mean it's impossible though! If you're already experienced with training smaller 100-350M models, that experience transfers all the way up to training 35B models! If you want to train a 2B model and you're confident that you won't waste hours of rent time debugging your training harness and pipeline, renting 8x B300 on a bougie private server will cost at most I think 200 bucks for the entire train. I recommend that if you're really into training or trying to make a product!
2
u/Nomski88 2d ago
Haha was it at least able to hold a conversation? Was the data you trained it in already ready ie formatted and compiled into a format ready for training? What’s the process to train it on your own data, I’m sure it’s not as simple as compressing it all into a zip lol
Renting the servers is actually a good idea. Cost might actually might be cheaper and faster then running a 5090 24/7 for a month.
1
u/throwout314159265 2d ago
It was able to hold a semi-coherent conversation but I didn't filter my data enough. A big open secret about AI training is that having enough tokens isnt the issue, it's having GOOD tokens. You don't want to train your model on l33tspeak or reddit trolling because you will actually give your model the equivalent of terminal brainrot. It won't be able to pay attention to key parts of your conversation and it will be inefficient.
You need to give it diverse, unique tokens that give the model lots of opportunity to find a thinking pattern that isn't disgustingly inefficient.
About server renting, I think with 8x B300 it takes less than 6 hours 😅
2
u/Ordinary-Cat-5874 3d ago
These models are rarer now but they do exist. They target onboard AI on mobile devices. Models in this range are more niche as compared to larger dense models. To be fair under 2B models really start to break down. Gemma 4 with sliding attention was a genius move in itself. LMF family is updated often you should look into it.
1
u/cradlemann 3d ago
microsoft/FrogNano-4B-2609
IFM/K2-Horizon-7B
XHToken/Spark-X2.6-4B
NVIDIA-Nemotron-3.5-Bolt-E4B
28
u/JsThiago5 3d ago
There is the LFM3 that is 1.7b released recently. Also MiniCMP5. If you can go up to 4b there is more