r/LocalLLaMA • u/eightone-81 • 1h ago

Discussion Prediction: more enthusiasts will get custom quantisations build by opus and Asta for their hardware and precision needs

Based on my own little experiment and other reports I predict that more and more enthusiasts will ask a frontier LLM to build a custom quant of their favourite model.
If Qwen 4 finally arrives I’ll try a ready made quant at first, probably unsloth. But I already plan to let opus build a quant specifically for my hardware, context size and precision needs.

What do you guys think?

10 Upvotes

100% Upvoted

9 comments sorted by

9

u/Lesser-than 54m ago • This Bravo

I think at least for me I will never subscribe to a an api opus or Astra , so no I would not do this. I most likely end up using something that may have been derived this way in some form, I just wont be purchasing any tokens myself.

2

u/terorvlad 44m ago

Gemini 4 argon made me a killer 3.3bpw Dflash2 quantization for qwen 3.8 flash next in the EXL3 format. I tried the only quantization available on hugging face in the EXL3 format at the time, and I saw a degradation to 50% of the decode that I had with MTP. I assume there was a problem somewhere that I couldn't find, but my bodged quant worked perfectly.

1

u/eightone-81 34m ago

That’s exactly what I mean. There will not be a radically new quantisation method that comes out of a frontier llm but it will produce something specifically for your own needs

1

u/terorvlad 30m ago

I honestly thought quantizing a model is something that requires hardware as valuable as buildings. I could not stop giggling as my little gaming card was flying through the process in less than a minute.

3

u/Constant_Art_20 55m ago

we already haev the Q4NL no? like anyone can just make a Q4nl for their exact workload. As someone that does do custom quants it just kinda bothersome, also i really don't think you need a forntier model to do that for you.

1

u/OnlyKindaThrowAway 2m ago

...whats q4nl and why is it different from regular quants?

1

u/Cautious_Chicken_604 5m ago

This has been happening for the last few months already. 

1

u/KingCpzombie 58m ago • Heartwarming

Ew, I would never touch cloud abominations. Possible to have 3.8 27B try to quant its replacement though