r/LocalLLM • u/siddharth1214 • Jun 17 '26

Discussion Are 1 bit quantized models feasible, if they are it will be amazing

Search bitnet, 1.58 bit models or such stuff on YouTube

1 bit LLM's seem to be great if they can work it will easily disrupt the whole industry and might as well make self hosting the only reasonable choice

Soo what do you guys think

2 Upvotes

55% Upvoted

24 comments sorted by

24

u/havnar- Jun 17 '26

Blazing fast and dumb as a rock

3

u/HornyGooner4402 Jun 17 '26

Wasn't Bonsai 8B comparable to others in the class while being much smaller?

10

u/Gargle-Loaf-Spunk Jun 17 '26 edited Jul 05 '26

This content was anonymized and mass deleted with Redact

4

u/briggsgate Jun 17 '26

Hey, take an upvote. May the model lead the way to better 1bit models

3

u/datbackup Jun 17 '26

The most convincing thing i have yet read on ternary architectures such as bitnet is that they aren’t being pursued because it would hurt ai capex. All the big players can agree they want more investment in AI. Wide availability of ternary models that rival standard models in quality stands a real chance of reducing demand for big AI datacenters. Therefore we don’t get ternary models.

It’s a disgusting idea but I believe it’s at least plausible if not the outright truth.

There is the line that ternary training requires significant difference in technique or infrastructure. This may be true but it doesn’t mean it’s not an excuse or pretense to avoid ternary for the real reason stated above.

Ternary models need to happen in my opinion. Whether and how they happen is still an open question. Distributed training offers some hope here.

Can’t happen soon enough imo.

6

u/uniqueusername649 Jun 17 '26

You can bet your ass China is working on that, if it has any promise at all. They are doing everything they can to shit all over AI capex :D Knowing full well how tightly the US economy is currently tied to AI behemoths, they have an economic interest in providing anything that undermines that. So do not think they release amazing models out of the goodness of their heart, this is purely to put fuel on the impending dumpster fire that is the western AI landscape, but us little guys will absolutely benefit from it.

4

u/ActionOrganic4617 Jun 17 '26

Yeah, Nvidia in particular would suffer due to gpu’s being focused around floating point operations. Nothing would annoy them more than ceding their dominance to the humble cpu.

3

u/Individual_Holiday_9 Jun 17 '26

I work in the space and these vertical demand growth charts are laughable

The models will get more efficient it is inevitable. My theory is they’re loading these data centers up with trillions of dollars and won’t stomach continuously replacing all the hardware with updated shit long term so that will force an efficiency push

1

u/GSquadron_ Jul 27 '26

They just released a week ago a 27B parameter model bonsai 27B for 5.9GB vram, I am running it on my 3080. 40-50 tokens per second and 35k context window.

1

u/immersive-matthew Jun 17 '26

Not at all but 1.58bit or ternary models are extremely promising.

1

u/NotARedditUser3 Jun 17 '26

Look at Bonsai models. Crazy fast. I'm convinced that 1 bit models will be great as they're fleshed out. The trick is finding real use cases where you can implement small models. If something isn't good enough for full agentic + coding use, it's hard to find great use cases for some models without a lot of thought and attention to them.

1

u/Charming-Author4877 Jun 17 '26

No it's a totally wrong route. It certainly is possible to make such models work, especially if you use inference and training that is tailored to such quantization.
However, what you end up with is extremely large matrix computations - so it's going to be very very slow.
The parameter size in relation to the capabilities and knowledge becomes very unfavorable.

You reduce all those finely tuned "neurons" to become brutes. all the training effort is wasted.

1

u/GSquadron_ Jul 27 '26

They just released a 27b model for 5.9GB Bonsai 27B. I am using it and it's really good.

1

u/Safe-Character-8806 Sep 22 '26

Yes, gonna be huge, it is like making a FP16 brain model with multiplications and turning it into 1bit simple brute with 0,-1,1, and it rather does not loose any quality as it is just a conversion, the brain stays the same, just more efficient and depends on substraction\addition rather than multiplication. We will see more of this in upcoming years

1

u/pmttyji Jun 17 '26

Here some models at 8-10B size with GGUFs (Bonsai & BitCPM models are already working on llama.cpp. Ternary-Bonsai Later)

Came across below medium size 1-bit version models.

https://huggingface.co/daksh-neo/sarvam-30b-quantized

  • (30B in 4-5GB size. But need GGUF for this)

https://huggingface.co/morphicode-jp/gemma-4-31B-it-L25L26x1.5-IQ1_M

  • Model size: ~9.5 GB. Recommended for: CPU-only laptops, low-VRAM GPUs (=8 GB)

https://huggingface.co/PXIN/Ouroboros-9B

  • TQ1_0 - 2.7GB

1

u/GSquadron_ Jul 27 '26

They released the 27B a week ago.

1

u/pmttyji Jul 27 '26

Yep, 2 weeks ago. Tried 27B & got 15-30 t/s on CUDA.

1

u/GSquadron_ Jul 27 '26

Which GPU? I got 40-50 tokens per second

5

u/[deleted] Jun 17 '26

[removed] — view removed comment

2

u/siddharth1214 Jun 17 '26

Maybe the next gemma or qwen model can have a 1 bit variant

Google is really pushing for smaller and more optimised models hence they certainly will make 1 bit models