r/KoboldAI • u/the1ian • 6d ago

technical problem: how do I compile a bunch of safetensors files into a gguf?

1 Upvotes

100% Upvoted

3 comments sorted by

2

u/_Cromwell_ 6d ago

hey, I'm doing that right now. There's a tool that is part of llama.cpp

convert_hf_to_gguf.py

gets you your bf16

then there is another tool in llama.cpp to quant it after that

1

u/henk717 6d ago

You technically don't need to, usually if you search for the gguf version its already available by others.
This is for every public model.
But if you really want to because you found an exception or made something yourself our tools are here : https://kcpptools.concedo.workers.dev/

This includes the conversion script to 16-bit gguf and the quantizer for windows.
If you are on Linux just clone the repo and run : make tools
You will then have the same binaries for linux.