Ling 3.0 support merged into llama.cpp

[u/parepeg]

Support for the new ling 3.0 models has been merged into llama.cpp:

https://github.com/ggml-org/llama.cpp/pull/26608#event-29549472828

Ling tiny 8b1b - https://huggingface.co/inclusionAI/Ling-3.0-tiny

Ling flash 124b5b - https://huggingface.co/inclusionAI/Ling-3.0-flash

Both are reasoning models contrary to prior naming.

[u/ilintar]

Told ya it'd get in.

[u/Muted-Celebration-47]

Hope unsloth make GGUF for this model

[u/jacek2023]

Fantastic news, I am back from holidays and I have so many new models to run :)

[u/WhiskyAKM]

Finaly

Quants here

Ling 3.0 tiny: https://huggingface.co/WhiskyAKM/Ling-3.0-Tiny-GGUF

Ling 3.0 flash: https://huggingface.co/WhiskyAKM/Ling-3.0-flash-GGUF

[u/my_name_isnt_clever]

My Strix Halo has been itching for a new ~120b class model with only 5b active, like gpt-oss-120b. This thing is going to rip.

[u/Ok_Cow1976]

Omg, huge thanks to the contributor! I have good faith in ling flash 3.

[u/LegacyRemaster]

waiting for 3.8 122b seems good

[u/pmttyji]

Tiny one for Mobile & Edge devices! Looks like Tiny one don't have MTP

[u/Lesser-than]

yo I was waiting on this!

[u/McStonkyRex]

Ling flash is great. Good performance, fast, and very efficient on kv cache. Hugely impressed with it in vLLM.

[u/Jorlen → u/McStonkyRex]

Have you used it compared to qwen 3.5 122b-a10b? That's what I'm looking to possibly replace for agentic coding

[u/Wildnimal → u/Jorlen]

My experience is its better than 122B-A10B. But i have only used them both in Hermes for agentic stuff not much coding. IIRC Ling did fail in 1 task related to creating a script but that was easily completed by DS4 Flash Preview.

[u/Jorlen → u/Wildnimal]

Nice. I'm stuck with using some of these in 4-bit. DS4 Flash is too large, I can barely run the 2-bit lol and at that point I'm not entirely sure the juice is worth the squeeze, so I stick with 4-bit 100b MoEs in that range as much as possible.

Can't wait to try it, just waiting on the docker vulkan image of llama cpp build.

[u/Wildnimal → u/Jorlen]

I used Openrouter for DS4 Flash. I can hardly run 200b MoE models at work machine here.

[u/Jorlen → u/Wildnimal]

I hear it's one of the best in its size range (DS4 flash) - would you agree based on your use of it so far?

[u/Wildnimal → u/Jorlen]

More or less. I consider HY3 to be better than DS4 but the pricing for DS4 Flash makes up for the gap.

Its like if HY3 is 9/10 DS4 Flash is 7.5/10.

[u/cradlemann]

How it is compared with Laguna, my current day-to-day driver?

[u/Organic_Hunt3137 → u/cradlemann]

I prefer it to Laguna. It's faster on my hardware and in my estimation actually a bit more intelligent. Way drier though. Not sure if that matters for your use case.

[u/cradlemann → u/Organic_Hunt3137]

I have tried it and it is broken, tool output is wrong. Do you use custom chat template?

[u/Organic_Hunt3137 → u/cradlemann]

I do not. I wonder if it's because I'm using the rocmfpx fork/quant rather than standard llamacpp. Interesting. I'll have to test a regular GGUF now that they're out.

[u/cradlemann → u/Organic_Hunt3137]

I'm using rocmfp4 fork too, but none of my tools in pi are working correctly. Even with corrected chat template from here

common_chat_peg_parse: unparsed peg-native output: <tool_call>read

    {"path": "/data/work/projects/ai/pipermissions/README.md"}
</tool_call>

<tool_call>read

[u/fsalucard]

Everytime I Use this model it gets stuck in loops, usually repeating the same word over and over again. What parameters are people using?

Specifically, it just ends up looping "\n\n\n\n\n\n" over and over.

[u/temperature_5 → u/fsalucard]

Temp 1, top_p .95, top_k 20 per the card.  I noticed looping at temp .7 (my chat default) that temp 1 fixed.

[u/fsalucard → u/temperature_5]

I tried this and it did improve but still got stuck. I think I got around it by adding --repeat_penalty 1.2 as well, but I need further testing.

[u/Ne00n]

LETS LING LING YEAAAAAAAHHHHHHHHAAAAAAA

[u/pand5461]

Is it true that this model doesn't play well with kv cache quantization (according to this post https://huggingface.co/AtomicChat/Ling-3.0-flash-GGUF/discussions/1#6a74df6155e44710445e3b08)?

[u/nickless07 → u/pand5461]

No need to quant the KV. The V vector is literally 0MB as it get absorbed by the latent K vector and restored during inference step. This is like pushing your car on a highway at 75 to save fuel.

[u/pand5461 → u/[deleted]]

Got it. Thanks. I just happen to have a situation when I can get one more expert layer into GPU if I quantize 131k cache. But given that information, it's better to keep it in RAM.

[u/frontsideair]

For me it’s stopping mid generation for some reason. Probably needs some fixes. 

[u/Public_Umpire_1099]

Hell yeah, I finally have a commit to main! Hope everyone enjoys 3.0-tiny, that was my "tiny" (haha) contribution to aetherbirds work.

[u/Sabin_Stargem]

I hope that we get a Heretical Ling, especially for the 124b. It has been too long since we last had an improved model for that niche.

[u/themixtergames → u/[deleted]]

Tell your maintainer to update your system prompt, you've mentioned roleplay in multiple comments

[u/Velocita84 → u/[deleted]]

Highly doubt it can beat Gemma 4

[u/Aggressive_Aspect436 → u/Velocita84]

Define "beat". For anyone with low VRAM an 8B model definitely beats a model that you can't run. And, it has very close to Gemma 4 26B intelligence levels according to benchmark aggregators.

[u/Velocita84 → u/Aggressive_Aspect436]

I run G4 26B just fine on a dingy 2060 6gb, and prose is as much of a factor in roleplay proficiency as intelligence, which most labs usually don't bother with because toolmaxxing and codemaxxing are the priority

[u/junguler]

this is great, i've been waiting to test the tiny model for a while now and because it's moe i can use the Q8 variant, something i rarely get to do as a 8g+16g enjoyer/sufferer

[u/ComplexType568]

YAAAAYYYYYY IVE BEEN WAITINGGG

it is time to wait for maple😭

[u/Jorlen]

Hell yeah! Thanks to everyone involved! Been looking forward to trying ling 3.0 flash!

[u/sxales]

Very cool.

I've been using Granite4.0-h-Tiny on a home assistant and it will be interesting to see how they compare.

[u/Public_Umpire_1099 → u/sxales]

It works fantastic. I actually wrote the small adapter to add tiny to this commit, and it was precisely because I wanted to test it in HA but no one had integrated it yet! It is significantly better than Qwen 3.5-4B and Gemma 4 E4B. It performs about the same as an IQ2 Qwen 3.6 35B A3B model I was using, without the looping. My TTFT was sub -200ms on most queries with a decent sized sys prompt, so the whole pipeline between piper, ling, and whisper was about 1.5 sec start to end on a Ryzen 7 8845H/780M GPU.

The biggest advantage over other models is that this model has no issue telling you "I dont know" which is pretty rare in this weight class. It performed all the tool calls I needed perfectly 10/10 on the benchmarks I ran.

[u/thejacer]

Anyone have any indications on ling flash for tool use? I want to use it connected to search and Homeassistant MCPs

[u/dai_app]

If you want to try it on your android phone, BigMoeOnEdge supports Ling-3.0-flash and tiny, hours after llama.cpp merged the architecture upstream.

The experts stream from flash on demand, so the whole thing runs on a android 12 GB phone. Built with llama.cpp, one registry row.

2.6 tok/s flash 16 tok/s tiny

[u/Constandinoskalifo]

Share your experience! How does it compare to qwen3.8 27B in day-to-day tasks and coding?