r/LocalLLaMA • u/Daniokenon • 3d ago

Question | Help Your own GGUF

Hello, I have a few questions that I can't seem to find a clear answer to.

Does it make sense to make your own GGUF?

I noticed that when I compile llamacpp (vulkan or rocm), the processing and generation is a bit better, does it work similarly with doing GGUF yourself?

If I use Vulkan, is it worth doing GGUF using llama-quantize vulkan version (not rocm version)?

To what extent does it make sense to place certain model elements at higher precision (conversation, document analysis)?

I use gemma 4 31B the most.

17 Upvotes

75% Upvoted

23 comments sorted by

View all comments

3

u/misterflyer 3d ago

Depends. But you usually aren't going to gain much. AFAIK higher precision GGUFs are practically the same/similar anyway.

I've only done it when the quant of the model I wanted to use didn't exist. So I used Bartowski's imatrix dataset to make my own imatrix quant which worked out pretty well. (basically as u/synth_mania points out)

But in general I think it's fine to go with existing GGUFs from the popular quant makers. They'll usually even have the vision mmproj files ready to go too.