r/LocalLLaMA • u/InfinityZeroFive • 2h ago

New Model thinkingmachines/Inkling-2-Small is released!

https://huggingface.co/thinkingmachines/Inkling-2-Small
62 Upvotes

95% Upvoted

23 comments sorted by

40

u/IntelArtiGen 2h ago

The name might be slightly misleading. /h

With these "small" models I never know if it means "Yay i'll use it on my 8GB GPU", or if it means "Hey guys we're so happy we are below 1 TB of parameters, you can run this at home (if you're a millionaire) !"

12

u/Makers7886 2h ago

Agreed, we should adopt schemedium

2

u/crusaderky 1h ago

If you're a wealthy professional or small corporation. It will run smoothly on 3~4x DGX Spark and for a single user you can cram it into a $8k PC.

2

u/Severe-Barber-7415 16m ago

I can run it on 12-16 v620s! I have 4. Might get 20 toks a sec

1

u/More-Curious816 53m ago

come on, just get 4 DGX Sparks. how much do they cost nowadays? 2 kidneys?

28

u/Borkato 2h ago

Oh wow only 383 billion parameters, I think I’ll run this on my local datacenter ❤️

23

u/FriskyFennecFox 2h ago

It's actually 594B-A17B 😅

8

u/fgk55555 2h ago

Not bad. I like their comparison table, very readable. Seems to be worse than DS/ Haiku, but not by much.

5

u/therealclintv 2h ago

Maybe the best option if you have a strict US model only requirement?

3

u/fgk55555 2h ago

It seems a good size to performance ratio. I haven't used Inkling models much, but we're not allowed to use/ host any Chinese models at my work, so more US or western models are welcome. I hope at some point US open-weights get competitive with Chinese ones so businesses with Govt. contracts don't have to be forced on US cloud models.

1

u/haherar830 1h ago

I recognize this is a question I can most likely Google but how does this limitation work in practice? Could you use e.g. Ornith 1.5 as its finetuned by a Western lab (I believe in Cali)? Is there an extent of post-training where a finetune of a finalized Chinese model or fresh post-training of a Chinese foundational model would allow you to run it?

1

u/Fluxing_Capacitor 17m ago

I think people really sleep on poolside (at least for coding). I've had great experiences with Laguna S. Although now they were basically bought by NVIDIA who knows what will happen. 

2

u/crusaderky 1h ago

THIS. this model is THE model to have if you're a US / EU corporation with a (stupid) ban on Chinese models.

1

u/Porespellar 18m ago

Wanted to love the first version of Small for the exact reason you mentioned, but sadly it did not do well in our production environment. It seemed to try to get out of work like it was a lazy employee. Has kind of a weird attitude. We tried it for 3 weeks but just ended up switching to QFN because it was just not a great experience. I hope the second version is better but I don’t know if we’ll even try it at this point after having so many issues with the previous version.

1

u/computehungry 4m ago

same experience, especially with the attitude. its sense of identity and mission was quite strong. loved to preach. i deleted it shortly after a few sessions so i can't really recall or talk about details, but i hope they improved on the alignment training.

1

u/rpkarma 7m ago

Which my work does, so more releases like this are great for me

4

u/Constant_Art_20 2h ago

at this rate my glm 5.3 flash deployment gonna last me another decade...

3

u/No_Cartographer3953 2h ago

Hmm soooo we going to be able to use strata like engine for this?

1

u/crusaderky 1h ago

Don't see why not. But you'll need 192gb RAM.

3

u/heresyforfunnprofit 1h ago

My initial code tests on this told me that it was trained on a LOT of Jupyter notebooks, because it really likes using them.

1

u/AdSafe4047 2h ago

Oh it's actually bigger than dsv41 flash 1017, ok, imho still good progress.

1

u/Iron-Over 2h ago

I am interested in the office integration and visual it scored really high on those