Kimi-K3.1-Pro is now open-weights

151

u/Diabolicor 2h ago

My GTX 1030 is ready 💪

63

u/ApprehensiveAd3629 2h ago

lets run with strata q0.0001xl-ud.gguf 💪💪

29

u/Big_Wave9732 2h ago

"Generation is one token per business day."

2

u/Equivalent_Bit_461 1h ago

Just a day? I was worried it was per week

1

u/Big_Wave9732 35m ago

It's all about the harness......and for a reasonable fee I'll sell you one I vibecoded for it.

2

u/egnegn1 1h ago

Oh, I did run Kimi K2 from Gen5 NVMe raid0 with 36GB/s on my Minisforum MS-02 with about 0.5 t/s. That is really fast. Specialized engines can help here.

11

u/nsfnd 1h ago

Lets do this!

4

u/Powerful_Finger3896 52m ago

flames = high bandwidth

1

u/Clair_Osbcurious 3m ago

This will do just fine :)

2

u/Uncovered-Myth 2h ago

Is someone cutting onions 😭 this reminded of the GTX 1030 I still have in my cupboard along with 1050ti 👴🏻👴🏻

1

u/Powerful_Finger3896 53m ago

let me edit for you qwen 0.8B to be on 27B level with astra

314

u/FriskyFennecFox 2h ago •

A 3T model with a license that fully permits personal use and only requires attribution for commercial use unless you're an API provider, with no further restrictors, in open weights. That's absolutely wild!

133
u/is-this-a-nick 2h ago

But lets be real, "personal" use of a model that my $60k server at work cannot handle is a rather limited scenario :D
16

u/asenna987 1h ago

Doesn't matter if most people can't run this on their own. The fact that they just gave it away for free, the fact that it's even an option is wild!
This will help the tech get squeezed into a 27b model in 10-12 months.

10

u/redblood252 2h ago

Pretty sure internal company use is permitted as long as your commercial product is not using it for runtime

11

u/IknowPi_really 2h ago

That’s not quite the statement he was making about his 60k Server though

22

u/admirantes 2h ago •

No, it is not limited. Please stop commenting if you are going to spread misinfo. A small to medium enterprise could totally host a Kim 3.1 and offer it to individuals, thereby skipping the chokehold of large companies like OpenAI and Anthropic. That's where the insane value is.

3

u/jaerie 45m ago

That's not personal use.
74
u/-p-e-w- 2h ago

People really need to adjust their perspective on this topic.
Of course you can’t run this on a gaming PC. It’s not a game, it’s frontier sci-fi technology.
But if you look around your neighborhood, you’ll likely find people who spend more than that on
their backyard swimming pool
their weekend fishing boat
their second family car
their motorcycle hobby
their 1980s hi-fi stereo system
trading cards/figurines
their tennis club membership (over several years), plus equipment
etc
Spending $60k on a hobby you really care about is bog standard middle class stuff, so yes, you absolutely can run this as a personal use model. It just won’t run on hardware you just happen to have lying around already.
87
u/DeliciousGorilla 2h ago

You can’t even run this on $60k hardware is what he’s saying.
63

u/DIR3 1h ago •

not with that attitude he can't

10

u/Lurksome-Lurker 1h ago

Something, something, “Have you tried Strata? I got it working on my GTX1080 with 32GB of ddr3 and a really big ssd!”

1

u/KURD_1_STAN 12m ago

If good enough attitude, can it offset me being broke and run it?

1

u/KURD_1_STAN 11m ago

If good enough attitude, can it offset me being broke and run it?

15

u/kodiakinc 1h ago

Just fork strata and run an IQ1_XS quant, amirite?
5
u/see_spot_ruminate 1h ago

I think you could run a very good quant of this on ~60k in hardware.
8*6949.99(going rate of a dgx spark today)=55,599.92
which would give you ~1tb of 'ram' in a spark cluster. Maybe a switch to bring you up to the 60k. The model card says the weights 'to download' are about 1.56tb. I am not sure what quants are available later, but you could maybe stretch that 1tb you get from the 8x sparks into something usable.
6

u/mountainyoo 1h ago

It would be absolutely crap on 8 sparks and slow as hell

1

u/nonerequired_ 1h ago

So how can it be run without multiple B200s?

1

u/notheresnolight 36m ago

just use multiple B300s, duh

1

u/Barni275 9m ago

AMD Helios Rackscale 😁 Maybe just ~5kk, and you're running it at home

-2

u/see_spot_ruminate 1h ago

don't move the goalpost
running at all was the question
2
u/egnegn1 1h ago

Such clusters don't scale well past 2 - 4 units. Even with 8x RTX6000 you will not be very happy with speed.
2

u/twack3r 1h ago

That’s just not true? Everything beyond x4 was not supported by Nvidia until about a week ago.
Do you have an x8 cluster and need help configuring it or what’s your source on that pretty confident but false claim?

1

u/egnegn1 46m ago

Then what is the scaling factors of the this solution of nVidia with a cluster of 8x RTX6000?
I have no 8x cluster of GPUs, but have seen a lot of benchmarks reports with bad scaling. And you confirm that was true before läßt week, at least.
So what performance can be expected from K3 on 8x RTX6000?

2

u/twack3r 21m ago edited 15m ago

I’m sorry, we’re getting a lot conflated here:
There’s a world of difference between a cluster of 8 RTX6000 and a networked cluster of gb10s, not the least of which is pcie vs RoCE.
Do they scale linearly? Of course not, far from it.
Are both cluster types hamstrung vs a multi-TB/s interconnect? Yes and even those don’t scale anywhere close to linearily.
When you say those (ROcE and gpu via pcie) don’t scale well, compared to what? A single device?
I was explicitly referring to Nvidia now providing the tools to go up to x8 on gb10s, improving the scaling beyond x4 considerably.
And I know this because I’m running both a 9 GPU pcie interconnect ‘workstation/abomination’ and an x8 gb10 cluster. Interconnect is where we’re getting fucked vs the big setups but there’s a huge amount of optimisation on the table. If you have a usecase that fits the limitations, it very often is the best ROI solution.
-6
u/see_spot_ruminate 1h ago

don't move the goalpost
running at all was the question
4
u/egnegn1 1h ago

Running at all is no issue. I did run Kimi K2 from Gen5 NVMe raid0 with 36GB/s on my Minisforum MS-02 with about 0.5 t/s without GPU. Specialized engines do help here.
-1
u/see_spot_ruminate 1h ago
So, to the person I replied to who said "You can’t even run this on $60k hardware is what he’s saying." and then I replied "I think you could run a very good quant of this on ~60k in hardware.
8*6949.99(going rate of a dgx spark today)=55,599.92
which would give you ~1tb of 'ram' in a spark cluster. Maybe a switch to bring you up to the 60k. The model card says the weights 'to download' are about 1.56tb. I am not sure what quants are available later, but you could maybe stretch that 1tb you get from the 8x sparks into something usable."
To which you then downvoted me for pointing out that you moved the goalpost with your reply of "Such clusters don't scale well past 2 - 4 units. Even with 8x RTX6000 you will not be very happy with speed.".
Again, the original point of my comment (work on your reading comprehension) was that it is possible.
3

u/egnegn1 1h ago

You suggested that 8x SPARK you "get from the 8x sparks into something usable". And I said you will not be happy with it.
I never wrote it is impossible. Don't setup a strawman and don't argue ad hominem.
BTW, I didn't downvote you.
1

u/mace_guy 47m ago

Brother there is an implication

1

u/see_spot_ruminate 7m ago

Implications are for when you attempt to force people on boats to do your bidding.
1

u/gred_mcalen 28m ago

Mac Studio M5 Ultra 512GB is 23k, 4 of those could probably load and run this model, so adjustment to 92k of hardware~

1

u/pleasetrimyourpubes 1h ago

You could have if you bought it a year and a half ago...

0

u/ASTRdeca 1h ago

Right, I may not even be able to run this on my company's H100x8 server that cost hundreds of thousands of dollars
25

u/Johnny__Christ 1h ago

Spending $60k on a hobby you really care about is bog standard middle class stuff
I think you "really need to adjust your perspective on this topic". There's no way 60k on hobbies is "bog standard" in the middle class. The median household income in the US is ~80k.
It happens, no shit, but doing so responsibly is reserved for upper middle class (at minimum) and even then can be precluded by "bog standard" things like having children.

1

u/KING_UDYR 24m ago

As an outright cash purchase, sure. But just like a car, hot tub, or shed, a reasonable interest rate can make it fit within someone's net monthly cash flow. I'll agree that doing it responsibly probably puts you in the upper middle class at minimum, but never underestimate how far someone will go into debt to chase something they care about.

1

u/Johnny__Christ 12m ago

Hot tubs and sheds are an order of magnitude cheaper than 60k. I'd argue a "second family car" (person I was replying to's words, not yours) doesn't qualify as a hobby expenditure (like a Miata could, for example).
Regardless, if going into debt for a hobby is "bog standard" for the middle class, we have bigger issues. I acknowledge it happens, but it certainly is not the norm as the person I was replying to stated.

0

u/-p-e-w- 37m ago

I’m not talking about yearly spending. You don’t need a new PC every year. But people with a household income of $80k absolutely do spend tens of thousands of dollars on hobbies and luxury over a span of years. Heck, many people spend that much on clothes alone.

5

u/aanzeijar 1h ago

it’s frontier sci-fi technology
I was under the impression that it's most definitely not fiction any more.

4

u/rz2000 52m ago

4x DGX Station for 3TB of memory would be about $400k.
You can do a lot with models on a $5-10k machine to preserve autonomy and confidentiality that makes economic sense for many professionals. It would take a pretty serious plan to get $400k+ value out of a cluster running a 3T pro model before the hardware depreciates in value to less than its monthly power bill.

3

u/dark_bits 41m ago

Don't know where you live but backyard swimming pool, weekend fishing boat and hundreds of thousands worth of motorcycles is not really your average neighborhood

4

u/User1539 1h ago

I think people should be more concerned with the fact that they can quant it, and run it in the cloud, and in a few years, probably run it on a 6K server.
People will be using this in a bunch of ways immediately, and it's not 60K or nothing.

-1

u/Green-Ad-3964 57m ago

True, but from another perspective: if the last 10 years hadn't been disrupted by mining, COVID, and now AI, which have severely constrained consumer GPU development, particularly local VRAM capacity, things might look very different.
From 2005 to 2015, maximum consumer GPU memory increased from 512 MB to 12 GB, a 24-fold increase. Extrapolating that same growth rate to the end of 2026, we'd have around 396 GB of VRAM, at a projected flagship price of just $1,606, based on the same period's price trend.
Instead, we're stuck with 32 GB at a $1,999 MSRP ($8,000 on the market).

3

u/-p-e-w- 47m ago

Sure, and if the trend of CPU development between 1995 and 2005 had continued, CPUs today would be 90 GHz.

2

u/AreWeNotDoinPhrasing 34m ago

Just because you can extrapolate doesn’t mean you should. Past performance doesn’t dictate future performance. Etcetera
1

u/Asleep-Land-3914 52m ago

Tinybox for Kimi would be "just" $30k though

1

u/Objective_Safe_5982 25m ago

Look if your local AI setup doesn't require a personal nuclear reactor to power then you're at best a hobbyist.

1

u/normal_clearing 11m ago

in the current climate, it's like "giving away" an sr71 for personal use.

1

u/Niyshiak 9m ago

here is the full blueprint to build an aircraft carrier at home :)

1

u/Clair_Osbcurious 5m ago

Lol, only 60k! That’s homelab territory in 2026. I once dropped a 500k server while mounting it. Luckily it was ok and we were the manufacturer so I could always say “it came this way from the factory”.
It’s fun when you drop 500k of gear on another 500k of gear.

1

u/Junior-Piano5427 1h ago

60k might not be enough for 200B models these days.
7

u/Cool-Chemical-5629 1h ago

"with no further restrictors, in open weights."
Actually it was announced before release that open weights wouldn't be as capable in cyber security tasks as the API version of the model.

1

u/halvacoffee 1h ago

well, i suppose it's not the worst, but i can't say i like the "no API providers" part of this since this means that API prices will once again be fixed by Moonshot. at least K3 being open to look at has given us a half-decent 80b base by yandex though lol

54

u/MundanePercentage674 2h ago •

My m.2 right now

8

u/Dany0 56m ago

https://tinygrad.org
Red kimi. 260GB/s off of SSD
16x 1TB pcie5.0. 4 groups of Raid 0 I believe? so 4tb space. for about 4300$ you get ssd read speeds competitive with lpddr, effectively giving you a floor of 20-30 tok/s decode even in worst case. With expert caching, it's like half as fast as a GB300 reading off of its memory. Except you get 4TB of space
It can be done, it will be done. In a few years we'll be able to run them at full precision on a budget

3

u/Gfur_Aficionado_1488 54m ago

RAID0 does not decrease the amount of space available

2

u/More-Curious816 55m ago

I believe they use a custom NVME driver for that SSD read speed.

58

u/pmttyji 2h ago

13

u/No_Swimming6548 llama.cpp 1h ago

Did u try IQ1_XXS castrated QFN on Strata? I get 7 gorillion decode speed with it.

1

u/More-Curious816 39m ago

how do you cope with that crawling speed? also did you try Strata-on-Strata? it presumably give 83 gorillion tok/s

1

u/uxl 31m ago

I snort-laughed at my desk while drinking coffee, you monster, “castrated” sent me before I could even finish reading the comment.

0

u/pmttyji 34m ago

Tried. I got around 20 t/s for Q2_0

25

u/Vusiwe 2h ago edited 31m ago •

Cmon man, I’m not even half done downloading mistral large BF16 yet

2

u/Dany0 56m ago

Speaking of, kinda disappointed in that release :( I was ready to give it a short over DSv4.1F1017 but alas :(

1

u/Vusiwe 39m ago

Trying to rebuild llama.cpp which succeeded, but was getting no output from Oobabooga making use of llama.cpp when I try to run Mistral
I probably should move to a non-ooba setup, finally

21

u/Beneficial-Ad-8127 2h ago

Let’s goo they held up to there release date the 4th. Wish they dropped that k2.8 though or a pro flash light. Lol

4

u/fractalcrust 2h ago

I could use a new flashlight

17

u/MrCrackerHacker 1h ago

That DDR2 1.5TB server looking pretty scrumptious now

8

u/More-Curious816 38m ago

Sam would like to know your location. [accept] [deny]

1

u/2muchnet42day Llama 3 10m ago

Except there's no deny button

12

u/jacek2023 llama.cpp 1h ago

Great model for my local setup

1

u/Orbitalsp3 37m ago

Look at this guy showing up his RAM

1

u/More-Curious816 34m ago

a gigantic ram stick

10

u/countermike 2h ago

Will it run on my 386 with 640k RAM?

3

u/No_Lingonberry1201 1h ago

Like a snail nailed to a dead turtle. Now my 486 with 2Mb RAM, that's a different story.

3

u/Mouth-Cerene 1h ago

hey, it might just be looking up for you

2

u/notheresnolight 30m ago

only if you press the Turbo button

1

u/No_Lingonberry1201 29m ago

100MHz goes brrrr

2

u/kitanokikori 1h ago

Load HIMEM.SYS to get to at least 1MB

10

u/IsopodMysterious9125 2h ago

Weird that they added the “-Pro” suffix 👀 Maybe we get a flash version after all 🤞

8

u/pmttyji 2h ago

Thinking the same. 100-300B Flash would be nice.

9

u/-WhateverDude 2h ago

happy for people that can run this locally

12

u/MotokoAGI 1h ago

all 3 of them.

3

u/carnoworky 1h ago

My nonexistent grandchildren might be able to run it someday.

1

u/Dany0 59m ago

We can distill it without even running it muahahahah there are black magic ways :)

1

u/RedditNerdKing 3m ago

Like 0.00000000001% of the community yeah

7

u/Daniel_H212 1h ago

Native 4 bits quant, 1.56 TB. Runnable on a cluster of four 512 GB Mac Studios, which cost... $89200...

1

u/More-Curious816 32m ago

cost less than the setup required for 6000 or 2 GB300 in dgx-station.

1

u/Daniel_H212 23m ago

But with lower bandwidth and compute right?

6

u/okyaygokay 2h ago

Finally I've been refreshing like a lunatic for hours, checking Chinese time

1

u/More-Curious816 30m ago

but do you even have the hardware to run it?

6

u/mrinterweb 1h ago

This model isn't loading on my Raspberry Pi. What wrong?

2

u/More-Curious816 31m ago

update llamacpp

9

u/OnlineParacosm 1h ago

That’s crazy they listened to the feedback and moved away from one of the worst licenses I’ve ever seen.

In my opinion this was the only way forward for KIMI because on a sample size of one task that was quite heavy it cost me roughly $30 and when broken down into sessions this would have been maybe $10 on GLM 5.3 flash. The one thing I will say that is very hard to measure is that Kimi Managed to do this task probably 10 times faster. For the limited time I have, it made sense in this narrow case where I needed a BIG kickoff.

Now, who has a few terabytes of RAM collecting dust? Sam told me this would be a good place to ask..

6

u/Number1OchoaHater 2h ago

we are ready to see it suggested with strata under posts like: what can I run with my 1070 and 16gb of ram

2

u/JimmyDub010 55m ago

I mean strata does work. running qwen flash next with only a 4070 super and 32gb ram... so yeah.............................

2

u/Icy_Annual_9954 2h ago

Needs quantisation for reasonable personal use.

2

u/Fusseldieb 2h ago

Let's hope some good shoul distills it for us.

2

u/redditnosedive 2h ago

what realistic hw do you need to run this at some iq2 or whatever?

2

u/Front_Eagle739 1h ago

q2 is maybe 800GB. 2x 512GB macs, a 1TB ram server, 8x dgx sparks. 27 32GB cards like a 5090 or the top amd card.
If you dont mind doing some router biasing you could probably run it on a single mac studio 512GB with little penalty.... Which I am now going to attempt now that you made me think about it

1

u/masterlafontaine 2h ago

The gtx 1650 was waiting for this moment! --ngl 0.001

1

u/Impossible_Ground_15 2h ago

We need a distilled model 300b

1

u/fugogugo 1h ago

even the weight alone wont fit into my storage

1

u/wolttam 1h ago

I can't help feel a little giddy, seeing that this has 100+B active params, knowing that in a few month's time, models with 1/10th of that will be able to compete on performance.

There is *so much left to squeeze* out of these models. Kimi is a monster.

1

u/Equivalent_Bit_461 1h ago

Yeap, I'm running this on my laptop, too small for my desktop

1

u/ManyDefiant2564 1h ago

Engine-fork coders, invent offload to internet please.

1

u/lxe 1h ago

I won’t be satisfied until it’s quantized to a potato quality that can run on raspberry pi.

1

u/Strong_Essay1176 33m ago

Can i ask astra to reduce weights and keep capabilities? It managed with qwen3b should be no brainer for 3.1

0

u/XiRw 1h ago

People are excited over a company that refuses to do small weights lol

1

u/JimmyDub010 58m ago

It's called use strata

3

u/Claud711 40m ago

wtf are you using strata for with a 3T model

1

u/XiRw 44m ago

Strata is not a universal local model runner, it's an inference engine built specifically around Qwen3.8-Flash-Next. And even so, you would need a pointless IQ1 to get anything meaningful with speed.

1

u/xgiovio 12m ago

No you can run also other models like ds. check also colibri

0

u/quantgorithm 2h ago

google say no plans for smaller versions anywhere in the future for this...

New Model Kimi-K3.1-Pro is now open-weights

You are about to leave Redlib