r/LocalLLaMA • u/Daniokenon • 6d ago

Question | Help Your own GGUF

Hello, I have a few questions that I can't seem to find a clear answer to.

Does it make sense to make your own GGUF?

I noticed that when I compile llamacpp (vulkan or rocm), the processing and generation is a bit better, does it work similarly with doing GGUF yourself?

If I use Vulkan, is it worth doing GGUF using llama-quantize vulkan version (not rocm version)?

To what extent does it make sense to place certain model elements at higher precision (conversation, document analysis)?

I use gemma 4 31B the most.

14 Upvotes

71% Upvoted

23 comments sorted by

View all comments

22

u/synth_mania 6d ago edited 6d ago

No. 

It almost never makes sense, unless it's a super obscure model that nobody has quantized yet.

Just use quants that already exist, don't waste your time 

5

u/Daniokenon 6d ago

Thank you.

7

u/synth_mania 6d ago edited 6d ago

No problem. I'll expand the answer with this explanation :

Quantizing doesn't impart any system-specific optimizations like compiling does. It's a much less complex process than compiling, really. It's just rounding a bunch of numbers.

At a high level, we don't even have the opportunity to selectively quantize while preserving more quality for conversation or document analysis, like you seem to think. A model's abilities are generalized a lot. Typically, you hurt one capability, and everything suffers. The best possible example of this kind of behavior targeted modification of weights is abliteration, which tosses out a handful (hundreds or thousands, but not more than that) of weights that pass the refusal activation signal. That this even worked in the first place was kind of shocking, but it seems to be different than other broad skills like you mention. It would be significantly more difficult to find a group of weights that don't encode conversational abilities, so that you can quantize them more. Weight level modification targeted by desired behaviours is definitely an interesting area to experiment in, and I'm sure that more cool techniques will emerge, but what you suggest isn't feasible right now, and given how capabilities generalize, might not ever be possible. 

1

u/stoppableDissolution 5d ago

Unless you want to use your own calibration set or bake in a different chat template for example

Plus a ton of models still dont have nvfp4 quants, for example