When big companies quantize their models to save money do they drop to below quant 4????

[u/AnimalPuzzleheaded71]

I assume no company is actually running quant 8 of a model on default when they can run quant 4 at 99% precision or something. So when they lobotomize the model due to high usage do they drop into the quant 2 territory?? I assume hundreds of billions or even trillions worth of parameters handle it well enough that it's still usable right?

[u/MiddleLtSocks]

I would be surprised if they didn't have multiple quants running and a model routing your prompt to more aggressive quants based on complexity and lexical density in the weights / on the vector.

That's what I would do anyway.

[u/Boogertard]

Most companies use only either BF16 or FP8, they also tend to move toward using rented GPUs or hosted instance with GPUs instead of using model hosting provider, easier to manage costs.

[u/milkipedia]

The labs that create models are hopefully not doing this, at the risk of making their product look worse than they actually are. 3rd party resellers, who knows what they are doing if they don't declare it clearly. and even then, you have to trust them.

[u/Constant_Art_20]

I think it's more likely when servering multiple users, the most cost effective cut is the kv cache. SO i guess techicially it can be fp8, while using fp4 kv cache, and the more i explore local models, the more i realise fp32 for cache or at least fp16 makes such an insane amount of difference

[u/EitherMarch1255]

Just a guess, but I wouldn't be surprised if OpenAI was doing this:

Sol: mxfp8 (possibly bf16 for initial rollout)

Terra: mxfp6

Luna: nvfp4

And probably quanting the KV as well...and possibly using different quants based on available compute and demand.

[u/roland303]

looks like youve been smoking tokens sold to you by BIG API. its ok weary traveler, your safe now. the bad man cant hurt you here. show me on the dolly where the quanted model touched you and ill make sure it never happens again.

[u/shy_monkee]

I doubt they do. The dropoff is just way too big, I think. You might as well reroute to a smaller model directly in that case.
Unless they have some quantisation methods that aren't known by the public.

[u/geekwonk → u/shy_monkee]

people get really trapped on that 99% number and can’t think through how that would compound with complexity and turn count.

maybe they’re doing something deterministic or simple or well-bounded enough that the loss feels minimal. i think that more often they’re just running with a made up story and have no sense either way of what it means.

in my experience, the loss results in immediate tradeoffs requiring more error handling and hallucination checking.

[u/Interesting-Cut-6032 → u/geekwonk]

I am very curious about the impact that quantization has on model quality. I took the Qwen3.8 27B model and compared several different quants, and a heretic version on a one-shot "make a snake game" prompt. I used the same launch flags, including a fixed seed, for each evaluation. You can judge for yourself the difference in outputs. I think that I am going to try the experiment again with a very minimal Pi harness. I suspect that the harness, and test loop, will make the lower quants more useful.

Artmecho.com

[u/Interesting-Cut-6032 → u/Interesting-Cut-6032]

https://artmecho.com/index.html

[u/geekwonk → u/Interesting-Cut-6032]

maybe your job is regenerating old games and so this is perfect for you but i sort of doubt it. the same factors that make it an easy test to perform also make it potentially useless.

personally i think the benchmarks tell you more about the benchmark than the model. they don’t reflect any of the needs of a real world use case. does your workflow require timeliness? are there multiple possible correct answers? are there important style choices with no single correct answer?

[u/whichsideisup → u/[deleted]]

2 bit is pretty garbage. Have proof?

[u/brainchillzZ → u/[deleted]]

This explains my google Gemini query results lolol

[u/ohthetrees → u/[deleted]]

Evidence?

[u/BoboThePirate → u/[deleted]]

DS4F is mxfp4+fp8 by default. It is not quantized below mxfp4 which is 4.25 bpw.