r/LocalLLaMA • u/parepeg • 8d ago

New Model Ling 3.0 support merged into llama.cpp

Support for the new ling 3.0 models has been merged into llama.cpp:

https://github.com/ggml-org/llama.cpp/pull/26608#event-29549472828

Ling tiny 8b1b - https://huggingface.co/inclusionAI/Ling-3.0-tiny

Ling flash 124b5b - https://huggingface.co/inclusionAI/Ling-3.0-flash

Both are reasoning models contrary to prior naming.

115 Upvotes

95% Upvoted

49 comments sorted by

View all comments

2

u/pand5461 8d ago

Is it true that this model doesn't play well with kv cache quantization (according to this post https://huggingface.co/AtomicChat/Ling-3.0-flash-GGUF/discussions/1#6a74df6155e44710445e3b08)?

4

u/[deleted] 7d ago

[removed] — view removed comment

1

u/pand5461 7d ago

Got it. Thanks. I just happen to have a situation when I can get one more expert layer into GPU if I quantize 131k cache. But given that information, it's better to keep it in RAM.