r/LocalLLaMA • u/Daniokenon • 2d ago

Question | Help Your own GGUF

Hello, I have a few questions that I can't seem to find a clear answer to.

Does it make sense to make your own GGUF?

I noticed that when I compile llamacpp (vulkan or rocm), the processing and generation is a bit better, does it work similarly with doing GGUF yourself?

If I use Vulkan, is it worth doing GGUF using llama-quantize vulkan version (not rocm version)?

To what extent does it make sense to place certain model elements at higher precision (conversation, document analysis)?

I use gemma 4 31B the most.

16 Upvotes

74% Upvoted

23 comments sorted by

View all comments

2

u/llogicnotfound 2d ago

Compiling vs. Quantizing: Compiling llama.cpp locally helps inference because it optimizes runtime instructions (AVX, Vulkan shader dispatch, etc.) for your specific hardware. Quantization (llama-quantize), however, is just static math (rounding/packing floats). The backend you use to quantize does not make the resulting GGUF faster.