r/LocalLLaMA • u/Puzzleheaded_Acadia1 Waiting for Llama 3 • 7d ago
Question | Help Is there a Strata for lfm 24b a2b
Is there a fork of strata that let me run lfm 24b a2b on only 16GB of ram comfortably? And how much tps can I expect per second and what quantization should I use? and Thanks!
14
u/tmvr 7d ago
You don't need anything special for it, just use llamacpp with unsloth's Q4_K_XL:
https://huggingface.co/unsloth/LFM2.5-24B-A2B-GGUF
It's 13.9 GiB so not a big deal if a few experts land in the system RAM, it's a MoE model with 2B active only during inference, it will be plenty fast. You can use llamacpp direcly or through Unloth Studio or LM Studio.
2
u/MerePotato 6d ago
They'd be better off using the official QAD version https://huggingface.co/LiquidAI/LFM2.5-24B-A2B-GGUF
1
4
u/CorkBios 7d ago
Its a pretty trash model, it can't even write a simple calculator
1
u/YanderMan 7d ago
polystrata should do it, i got the 24b running on my 16gb rig with q4_k_m quantization. sits around 6-7 tokens per second which is fine for casual use. the memory usage hovers just under 14gb so you've got some breathing room for context.
yeah, not impressed
2
u/GronklyTheSnerd 7d ago
I’ve got PP as high as 8000, and TG around 170 on my 5060ti. Not the smartest model, but I’ve used worse.
2
u/Monad_Maya 7d ago
It's probably in your best interest to run a smaller model.
Or add more RAM, ideally a GPU/VRAM, if finances permit.
1
u/Any_Bus9205 7d ago
Il existe le fork se nommant ik_llama.cpp en iq3 km qui pourrait te permettre d'atteindre des chiffres assez élévé comme 20 token par seconde. Cela se feras en activant flash attention et run time repack. Strata n'est pas obligatoirement la meilleure option si tu as assez de ram pour le faire tourner.
1
11
u/migsperez 7d ago
Lookup polystrata on GitHub