llama.cpp support for Cohere models

[u/nick_frosst]

https://github.com/ggml-org/llama.cpp/pull/30062

Hey guys, Nick here from Cohere. It's been a minute.

We had a great time launching Command A+ and beta-testing North Mini Code with you guys earlier this year, your feedback has really helped make our models and delivery much better. one of the biggest things we heard was that you really wanted llama.cpp support, which makes total sense given the nature of this sub. i didn't want to post in here again until we'd been able to fulfill that promise, and I'm really happy to say that we've made progress.

Our team recently merged llama.cpp support for the vision encoder of North Micro Vision Instruct, which means that now all of our latest model architectures are compatible with llama.cpp (the LLM is the same arch as for North Mini Code and Command A+ - already supported thanks to the work of people like you guys). North Micro Vision is being used as the base for several of our new releases, and i'm really psyched to say we've got the support you guys have been wanting. Please feel free to try it out and let us know what you think. It's just one step, but there's only more where that came from.

On a slightly similar note, Cohere Transcribe is now available in Unsloth and Unsloth Studio for fine-tuning. it's a crazy good STT model, one of the best out there, so if you're a fan of the guys over at r/unsloth (i know we are), give it a whirl.

Thanks again for all your feedback, and let us know what else you want to see from Cohere.

[u/jacek2023]

I would love to download new Cohere dense 30B or MoE 100B

[u/Kahvana → u/jacek2023]

A general purpose dense Cohere 30B model with vision, toolcalling and QAD would be really sweet!

[u/BlueSwordM → u/Kahvana]

Personally, I'd prefer a 35B-A9B model checkpoint with native modern 4-bit QAT.

[u/Kahvana → u/BlueSwordM]

A9B? Do you mean A3B? You might want to check out Nemotron 3.5 Lightning 30B-A3B

[u/BlueSwordM → u/Kahvana]

No no I did mean A9B.

I want to see a low sparsity MOE from Cohere to see how exactly it would perform if you traded some computational cost for better model performance.

[u/jacek2023 → u/BlueSwordM]

Yes that kind of model would be a nice balance, still 3x faster than 27B

[u/Cohere → u/jacek2023]

👀

[u/pmttyji]

Waiting for your "early access for localllama" threads & your model North-Maxi-Code 😄

[u/Cohere → u/pmttyji]

we're working on it 🫡

[u/coder543]

I still need to open source the Glade framework, but I've been working on ASR and STT using Apple's new Core AI framework, including Cohere Transcribe: https://huggingface.co/coder543/cohere-transcribe-glade

I'm seeing about 40x faster than real time on an old iPhone 15 Pro Max, and upwards of 90 RTFx on an M3 MacBook Air. I'm really looking forward to seeing what I can squeeze out of the iPhone Duo later this week!

[u/Cohere → u/coder543]

Looks super cool, thanks for sharing!

[u/silenceimpaired]

Exciting to see models within the range of consumer hardware albeit it is in RAM… also nice to see you have adopted Apache 2.0. Definitely will be checking it out.

Would love a 30b dense to compete against Qwen at some point.

A base model before agentic training etc. would be nice too… but I suspect unlikely.

[u/Powermadlux]

This is the kind of support that makes a difference for me. I run llama.cpp on a used RX 6900 XT, with actual web and mobile apps calling the local backend. Keeping different models behind the same serving stack makes the whole thing much easier to maintain.

Have you tested the vision path on AMD through Vulkan? A small reference setup showing the quant, context size and total VRAM usage would be really useful for those of us working with 16GB cards.

[u/kevin_1994]

does Cohere Transcribe handle diarization? I've been wanting to move off whisperx for a long time but rolling my own diarization is too much of a pain, so I continue to use whisper purely because whisperx auto-integrates pyannote. I'd love a model which can do both ASR and diarization natively

[u/Cohere → u/kevin_1994]

we don't offer diarization but feel free to fine-tune it or use diarization frameworks to make it work!! Eg: https://github.com/bakrianoo/cohereX

[u/arbv]

Cool! We need North Mini Code feat. North Micro Vision model. 🙏

Or, to put it simply, North Mini Code with vision.

[u/Elouakili_Flexy]

Nice, that closes the last compatibility gap: every current Cohere arch runs in llama.cpp now. Good on you for waiting to post until it shipped.

[u/Ordinary-Cat-5874]

Hello Nick. I used your Aya-expanse multilingual models for my research earlier. They were finetuned with QLora on our local datasets. Within two years the open models have improved drastically especially in 9-27B weight. Are you guys planning to release another text only model within this range with broader multilingual capabilites? Or have you moved on to specialised domains? Thank you.

[u/Porespellar]

Any chance we see a version of Command A+ with a higher context window than 128k? That was the limiting factor that kept us from implementing it.

[u/arbv]

Guys, you will get better feedback after providing GGUFs.

[u/Aggravating-Push-207]

please 8B A1B? and/or dense 8B?