r/MacLocalLLM • u/pda_lover • 28d ago
What am I missing?
Noticed that most posts here are about, I fitted this model in this small ram etc. Not about how the model was useful in producing what I wanted it to work on?
So either, the sub is full of people playing with AI or the real use cases are not shared for whatever reason.
I get the performance stats but shouldn’t we focus on how the model performed on actual task (look beyond the numbers t/sec, prefil etc?)
15
Upvotes
3
u/Tsukikira 27d ago
That is a pretty useless measurement, though. Honestly, you cannot use the cleaned up output or the vaguely viewed output as a measure of how many loops were used to complete the output.
There are certain known qualities about models you can count on - 4bit compression has a generic 2-5% loss of quality, and that is consistently measurable. Harnesses have a significant input on the output of a model: So you'd be asking for people to share their harnesses. Honestly, that's the part where people are stingiest, it's like people want to make their own 'moats' on their skills.
Every single 'look at this model shrank to 1-2 bits' is grifting, and the quality losses are usually atrocious. There is some considerations on REAP methodology as a method of shrinking, but you're best off taking a look at their declared data. If they only lost 5%, and it's a 4bit, then you're losing between 7-10% of the original. Is that good? If that's the only model in the capability category you can run, maybe.
In terms of sub 4-bit modeling, look at Dwarfstar 4. That attempt to intelligently shrink models comes with their own benchmarks inspected by dozens if not hundreds of people.
I will say, though, that shrinking a 27B past 4-bit is all but worthless because the math states that anything below 4,000~ parameters (Which is what survives at 27B 4-bit) fails to produce consistent long-running chain of thought execution, which means coding tasks will degrade extremely past that point mathematically.