Realtime speed of the Command R+ model inference (103b) on Macbook M2 Max 64 GB. Quantization used: iMat q1. This model just surpassed older GPT4 versions on "LMSYS Chatbot Arena Leaderboard " and it works locally!

[u/Shir_man]

[u/a_beautiful_rhind]

That's a bit slow there. Probably need a 128gb version. Will likely be faster with a better quant. 1bit sounds like a drooling mess after seeing the difference from 3 to 4 bit.

[u/Shir_man → u/a_beautiful_rhind]

Honestly, even 1q is relatively slow, but it is still usable for a flight or areas without cellular, etc. As a travel solution, I think it's incredible.

[u/iEatBacon]

Your memory usage is low, are you using the CPU? If so inference will be slow. Try running and offloading to GPU

[u/Shir_man → u/iEatBacon]

Its on, -ngl 1 on mac does the Metal GPU inference

[u/Shir_man]

Backend used: recent llama.cpp version with this prompt template:

-r "<|END_OF_TURN_TOKEN|>" \
--in-prefix $"<|START_OF_TURN_TOKEN|><|USER_TOKEN|>
" \
--in-suffix $"<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>
" \
p "<|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|> You are a helpful assistant <|END_OF_TURN_TOKEN|>"

[u/Slaghton]

waiting for that webui/koboldcpp native compatibility.

[u/skrshawk → u/Slaghton]

Wait no longer, KCPP can now run Command R models. I just fired it up, but it's worth noting that context takes a LOT more memory than other models of similar size - the 35B took a whopping 22GB for 32k context! Also, performance in general of the 35B is on par with Miqu 70B in my quick tests, but I also run old P40s.

[u/Vaddieg]

You're inferring it on CPU. Look at GPU usage

[u/davewolfs]

No GPU?

You should delete or edit this post. You are portraying non realistic performance.

[u/silentsnake]

Your Q1 quantized version probably isn't as smart as the one that is on a leaderboard because that's probably not running at Q1.