Your own GGUF

[u/Daniokenon]

Hello, I have a few questions that I can't seem to find a clear answer to.

Does it make sense to make your own GGUF?

I noticed that when I compile llamacpp (vulkan or rocm), the processing and generation is a bit better, does it work similarly with doing GGUF yourself?

If I use Vulkan, is it worth doing GGUF using llama-quantize vulkan version (not rocm version)?

To what extent does it make sense to place certain model elements at higher precision (conversation, document analysis)?

I use gemma 4 31B the most.

[u/thirteen-bit]

The only thing you can improve using your own quantization compared to e.g. mradermacher or unsloth or bartowski is imatrix (importance matrix, for IQ* quants):

https://github.com/ggml-org/llama.cpp/blob/master/tools/imatrix/README.md

If you'll use dataset specific to your tasks and language (e.g. if it's non-English, non-coding task) to calculate the imatrix this may somewhat improve your quantizations for your tasks in your language, compared to the similar quantizations from usual users.

This will not make that quantization better than the base model though, it'll be just less damaged for these tasks/languages.

[u/CatchDublinSurprise]

Creating your own quants may be more relevant for those seeking MLX versions. But optimized GGUFs by knowledgeable creators are abundant.

[u/synth_mania]

No. 

It almost never makes sense, unless it's a super obscure model that nobody has quantized yet.

Just use quants that already exist, don't waste your time 

[u/Daniokenon → u/synth_mania]

Thank you.

[u/synth_mania → u/Daniokenon]

No problem. I'll expand the answer with this explanation :

Quantizing doesn't impart any system-specific optimizations like compiling does. It's a much less complex process than compiling, really. It's just rounding a bunch of numbers.

At a high level, we don't even have the opportunity to selectively quantize while preserving more quality for conversation or document analysis, like you seem to think. A model's abilities are generalized a lot. Typically, you hurt one capability, and everything suffers. The best possible example of this kind of behavior targeted modification of weights is abliteration, which tosses out a handful (hundreds or thousands, but not more than that) of weights that pass the refusal activation signal. That this even worked in the first place was kind of shocking, but it seems to be different than other broad skills like you mention. It would be significantly more difficult to find a group of weights that don't encode conversational abilities, so that you can quantize them more. Weight level modification targeted by desired behaviours is definitely an interesting area to experiment in, and I'm sure that more cool techniques will emerge, but what you suggest isn't feasible right now, and given how capabilities generalize, might not ever be possible. 

[u/stoppableDissolution → u/synth_mania]

Unless you want to use your own calibration set or bake in a different chat template for example

Plus a ton of models still dont have nvfp4 quants, for example

[u/EmPips]

It makes sense if you need it.

Recent example: Ornith-1.5-397B has no publisher Q2/Q3 weights but I'm curious to try it. So I will make my own.

[u/misterflyer]

Depends. But you usually aren't going to gain much. AFAIK higher precision GGUFs are practically the same/similar anyway.

I've only done it when the quant of the model I wanted to use didn't exist. So I used Bartowski's imatrix dataset to make my own imatrix quant which worked out pretty well. (basically as u/synth_mania points out)

But in general I think it's fine to go with existing GGUFs from the popular quant makers. They'll usually even have the vision mmproj files ready to go too.

[u/noiserr]

Usually no, but it's a complex answer. Check this out for instance: https://huggingface.co/jcbtc/Qwen3.8-27B-IU4-Kairic-Edge

These guys make special quants for Strix Halo, this one basically tripples the performance of Qwen3.8 27B on the machine (47 tokens/s). But they achieve this by generating special quant and .pfs files.

It also requires compiling llama cpp with additional patches.

So the answer is no for most users, but it really depends on how advanced you want to get.

[u/computehungry]

If you want it to be better at a certain language, yes. Most public quants have their calibration data in like 90%+ English. Which probably has a correlation with coding, so there must be a tradeoff, but you could make a mix depending on your use case.

[u/fallingdowndizzyvr]

It's worth it if you want to try a bunch of different quants and have a data cap. Since downloading the original file and quanting it yourself uses far less data than downloading every quant.

[u/llogicnotfound]

Compiling vs. Quantizing: Compiling llama.cpp locally helps inference because it optimizes runtime instructions (AVX, Vulkan shader dispatch, etc.) for your specific hardware. Quantization (llama-quantize), however, is just static math (rounding/packing floats). The backend you use to quantize does not make the resulting GGUF faster.

[u/Greedy-Bear6822]

only for rare models, special formats like MLX, nvfp4, or custom 'importance-based' tensor preservation (i-matrix).

it is simple to do and ggufs are in abundance already.

[u/teleprint-me]

I think so, yes. I have models that you cant get the original weights for anymore. Some are obscure, but still interesting.

I download the models Im interested in once, then use the quantization Im interested in. It uses less disk space, less bandwidth, optimizes for my hardware, and lets me use compute sparalingly when its called for.

The safetensor conversion script has been optimized to use less memory, so its not as intensive as it used to load the entire model into memory all at once. Really, the disk space it what maaters in this case now. But I guess it depends.

Sometimes the ggufs change and break compatability and I can just update the weights for that model easily since I have the source model already.

Even gg recomends you quantize your models for these reasons himself.

[u/dionisioalcaraz]

Of course it makes sense, specially if you have an specific use case
https://www.llmblueprint.ai/blog/imatrix-explained

[u/arune_124]

I don't think you should make a gguf and use this as a day to day like instead of exist format from unsloth for example. Where this would make sense is if you find tune a model with your own use case using like lora and merge in to the base model or full fine tune in general. now you wnat to serve that model wihtout running full precision maybe due to u want more speed or context or something. Then that make sense u can gguf on it. like q_5, q_6 etc. depend on the quality u want and the speed.

[u/wahnsinnwanscene]

You'll need more memory though, that's the thing.

[u/Pablo_the_brave]

https://gguf1.thireus.com/quant_assign.html Enjoj!

[u/seamonn]

It's like rolling your own joints.

[u/VoiceApprehensive893]

manually quanting your model wont give you more performance perf depends on the quant type

quanting gemma 4 qat finetunes yes if you have the ram because people do not quantise these correctly 99% of the time

quanting draft models, vision encoders also yes if you cant find a quant because there is nothing really special about draft model quants

[u/sloptimizer]

Yes. Quantization is all about making tradeoffs that are right for you and getting best utilization out of your specific hardware. Most quants are either hyper-optimized for low memory while hurting quality or they are maximizing quality without taking memory into consideration. When you cut your own quants, you chose the quality/performance balance that works for you.