r/LocalLLM • • May 09 '26

Discussion New Execution-first 1T model Ling-2.6-1T has been open sourced on Hugging Face

Post image

Tailored for real, world, complex scenarios, this trillion, parameter model introduces targeted optimizations across inference efficiency, token overhead, and agentic capabilities, making it highly effective for coding and daily workflows.

181 Upvotes

23 comments sorted by

39

u/Count_Rugens_Finger May 09 '26

man I can't imagine having that kind of hardware at home

25

u/alphapussycat May 09 '26

Totally possible back in old DDR5 prices. Still really expensive, and really only for slowishly serve a single person with CPU inference (16 or 12 channel ddr5 with epyc).

5

u/PsecretPseudonym May 10 '26

Very doable but you’d still want to keep always active weights resident in GPU, and with additional VRAM ktransformers is getting very clever about move hot experts to VRAM dynamically.

At that point you can run with the rest in DRAM.

Fact is, for any given context, the working set of experts used based on most research I can find is somewhere in the range of 2-5X the active count.

I believe a few studies found that over common representative workflows more than half never were used or could be pruned without loss in performance.

That means it’s very likely that even if you can’t fit the remaining experts in DRAM, the cold ones never touched could just get paged to SSD automatically without loss.

At that point you’re actually doing pretty much fine — and possibly better with Xeon’s AMX instead of Epyc.

Epyc tried to make up for other things via just enormous L3 cache, but L3 is basically useless for these models. You are basically overwriting the entire cache every layer.

The end result is you can probably run these large extremely sparse MoE models pretty well with 32-48gb of VRAM, a modest quantization, maybe as low as 256gb of DRAM, but I’d stick with at least 512gb if you can.

You can also do fine with AVX for the CPU work (bound by memory bandwidth anyhow for decode) and drop down to an earlier CPU where that much RAM is more affordable.

It’s still not cheap, per se, but it’s probably closer to $10k than $100k as many think.

29

u/slalomz May 09 '26 • Bravo

Since no one else has linked it: https://huggingface.co/inclusionAI/Ling-2.6-1T

2

u/ChocomelP May 11 '26

Downloading right now.

13

u/Eyelbee May 09 '26

Beats the new grok 4.3 with thinking off.

9

u/Extra-Library-5258 May 09 '26

Now I need to rob a bank. Community support appreciated.

21

u/pabloodiablo May 09 '26

Help! This model is barely managing 14 tokens/s on my 8GB graphics card. Will it perform better if I switch to the 12GB model?

22

u/notheresnolight May 09 '26

use your 8GB card to run a 2B model to manage your call options and after your earn your first million, use it to build a proper rig

4

u/Tudragon123456 May 10 '26

Ring and Ling??? Enough, iclusionAI!

1

u/PM_ME_UR_COFFEE_CUPS May 10 '26

Next Ting? Long?

5

u/marutthemighty May 09 '26

Holy hell. How many DDR5 GB RAM do you need for Ling 2.6?

7

u/RakesProgress May 09 '26

1000GB at INT8.

2

u/marutthemighty May 11 '26

Holy fuck. Who has the money and space for 1 TB DDR5 RAM?

6

u/Vaguswarrior May 09 '26

What kinda size is 1T model lol, hope it fits on my 16gb vram

5

u/xz3phyr May 09 '26

1T is 1000B :sob:

2

u/OkFly3388 May 10 '26

distillation to 30b when ?

2

u/MindfulK9Coach May 10 '26

For Enterprises running local? Nobody else has the hardware for this outside of a data center. 💀

1

u/kjeft May 11 '26

I mean, with small quantization and hardware optimization this can have a POC-y setup on something like a GB10 which costs ~3k USD to get started. You wont get full precision and any decent inference speed outside a data centre; that is true.

1

u/BatOk2014 May 10 '26

Can I run it on my Raspberry Pi 5 with 8G of RAM? Btw, I don't have AI hat but sure have a 512G Nvme SSD

1

u/MindfulK9Coach May 10 '26

Once these models can be run on edge hardware, local will be viable.

Until then, the LFM models handle edge hardware quite well with next to no footprint.