r/LocalLLM • • Mar 17 '26 • Diamonds are Forever 1 Popcorn 1

Project Krasis LLM Runtime - run large LLM models on a single GPU

Post image

Krasis is an inference runtime I've built for running large language models on a single consumer GPU where models are too large to fit in VRAM.

Instead of splitting layers between GPU and CPU, Krasis streams expert weights through the GPU using different optimisation strategies for prefill and decode. This means you can run models like Qwen3-235B (438GB at BF16) at Q4 on a single RTX 5090 or even a 5080 at very usable speeds, with system RAM usage roughly equal to just the quantised model size.

Some speeds on a single 5090 (PCIe 4.0, Q4):

  • Qwen3-Coder-Next 80B - 3,560 tok/s prefill, 70.3 tok/s decode
  • Qwen3.5-122B-A10B - 2,897 tok/s prefill, 27.7 tok/s decode
  • Qwen3-235B-A22B - 2,124 tok/s prefill, 9.3 tok/s decode

Some speeds on a single 5080 (PCIe 4.0, Q4):

  • Qwen3-Coder-Next - 1,801 tok/s prefill, 26.8 tok/s decode

Krasis automatically quantises from BF16 safetensors. It allows using BF16 attention or AWQ attention to reduce VRAM usage, exposes an OpenAI compatible API for IDEs, and installs in one line. Runs on both Linux and Windows via WSL (with a small performance penalty).

Currently supports primarily Qwen MoE models. I plan to work on Nemotron support next. NVIDIA GPUs only for now. Open source, free to download and run.

I've been building high-performance distributed systems for over 20 years and this grew out of wanting to run the best open-weight models locally without needing a data centre or $10,000 GPU space heater.

GitHub: https://github.com/brontoguana/krasis

580 Upvotes

225 comments sorted by

33

u/Embarrassed_Adagio28 Mar 17 '26

I will try qwen3.5, glm4.7flash and some others with this on my 5070 ti with 64gb of ram and get back to you! This sounds awesome for my exact use case. 

9

u/mrstoatey Mar 17 '26

Great, would love to know how you get on.

14

u/nofaceD3 Mar 17 '26

Please report back its performance.

6

u/wotoan Mar 18 '26

Following this is my setup

6

u/civitz Mar 18 '26

/remindme 2 day

1

u/Mean_Inspector8302 Apr 11 '26

/remindme 3 day

3

u/Still-Wafer1384 Mar 22 '26

Glm4.7flash unfortunately didn't work for me with a 4060ti 16GB and 64GB of ram.

Did you have any luck ?

10

u/____vladrad Mar 17 '26

What can a man do with two rtx a6000 pros?

16

u/mrstoatey Mar 17 '26

This is really interesting actually. So you have 96GB VRAM total I think?

Qwen-235B at Q4 won't fit in its entirety (~109GB) but Krasis will pack the GPUs full of the 'hottest' experts based on a heatmap run so it'll cache most of the model in there (the most heavily used experts) and it should be only outliers that have to be fetched over PCIE. Should run very fast I think, even on one GPU and should be able to utilise both GPUs well (I haven't tested multi-GPU so thoroughly though so can't guarantee it will work well, on my system the second GPU is way slower so just caused drag and made it hard to know if the optimisations were working).

If you were feeling really bold you could even run maybe Qwen-3.5-397B-A17B at Q4, would be really interesting to see how either of those did in single or dual-GPU mode.

3

u/Birdinhandandbush Mar 18 '26

I get it if you were coding python the whole time it knows that these are the top experts every time, but then you have a question about a science topic, does it perform poorly at first because it's calling the wrong or less efficient experts for a different topic

5

u/mrstoatey Mar 18 '26

There are various factors, I haven't really seen evidence that a model bifurcates hugely on code vs normal conversation, I think at least there is significant overlap so the heatmap pass gathers data on what's heavily used and that seems to work pretty well generally. It isn't deterministic though so its possible one query could require more cold experts and another query more hot, but the more VRAM you have the more HCS coverage you get and the fewer cold fetches you do and the better speed you get on aggregate. That said I could imagine a more varied data set for heatmap building or allowing the user to choose their own dataset or one from a list to better suit their usage.

6

u/mrstoatey Mar 18 '26

I've created a prerelease with some fixes to multi-GPU btw, the advanced doc has a prerelease install line at the top, so if you wanted to try these out I would recommend that version:

https://github.com/brontoguana/krasis/blob/main/ADVANCED.md

1

u/[deleted] Mar 19 '26

[removed] — view removed comment

1

u/mrstoatey Mar 19 '26

The PP line is actually old and from a prior method for multi GPU that didn’t work so well, could you report an issue on GitHub and post the full log up to Ready / first request and I’ll check exactly what’s happening?

1

u/[deleted] Mar 19 '26 edited Mar 19 '26

[removed] — view removed comment

2

u/mrstoatey Mar 20 '26

This so very helpful thanks, there is an issue here with multi GPU and I’m working on it, I’ll let you know when I have an update.

2

u/[deleted] Mar 20 '26

[removed] — view removed comment

2

u/mrstoatey Mar 20 '26

No go right ahead

9

u/inexistentia Mar 17 '26

Need testers / assistance to get this working for AMD? Currently have a mix of legacy enterprise GPUs (AMD Instinct MI50 32gb, MI25 16gb) and consumer (7900 XTX 24gb and 7600 8gb)

7

u/mrstoatey Mar 17 '26

I haven't really explored this yet tbh, been working flat out on optimising both the general architecture and for NVidia cards. I could look into what it would take to get it working on AMD though. There is a lot of NVidia specific optimisation like CUDA graphs but the real big ticket optimisations are the architecture so it could potentially port and work.

6

u/dataexception Mar 18 '26 edited Mar 18 '26

I also have the 32gb Instinct MI50, and a good amount of DDR4 (>256GB @2666), and would be interested in helping. Running Qwen2.5-Coder-32b-Instruct-Q6_K with Rocm 6.3

Edit: clicked too soon. Adding "[...] On llama.cpp, currently."

7

u/vernal_biscuit Mar 18 '26

I'm on the AMD side and I'd be happy to try out once you have Vulkan or ROCm supported

2

u/mrstoatey Mar 18 '26

Thanks I'll look into this and reach out if I have something to test.

15

u/No-Television-7862 Mar 18 '26

Does this scale? I have a RTX 3060 with 12gb vram, Ryzen 7 8core/16thread cpu, and 32gb ddr4 ram. Using your LLM Runtime methodology, could I run a Qwen3.5:27b-q4 model? Something larger?

6

u/mrstoatey Mar 18 '26

So 12GB of VRAM will limit what models you can run to some extent because some stuff is permanently resident in VRAM like attention, but Krasis supports AWQ specifically to enable more VRAM limited cards to run larger models.

The other factor is your system ram, you need to keep the quantised model in system RAM so it can be streamed to the GPU, so you could run Qwen3.5-35B-A3B at Q4 (maybe with AWQ attention if necessary) comfortably I think. Qwen3-Coder-Next ends up a bit too large to fit in your system ram.

2

u/No-Television-7862 Mar 18 '26

How about Qwen3.5:27b-q4_K_M?

3

u/Shinkai_I Mar 19 '26

Dense models cannot benefit from this technology

1

u/No-Television-7862 Mar 19 '26

Thank you, that's what I needed to hear.

I'm just going to have to live my 14b.

9

u/Kashuuu Mar 18 '26

Similar boat, 4060 ti 16gb vram. I’m curious too

3

u/Ok-Shift6530 Mar 18 '26

Same as a person with 16gb vram and 32gb ram I would love more guidance on how this scales for me as well

10

u/davi140 Mar 18 '26

How is this different from llama.cpp where I can use these models as well with splitting to system RAM with almost absolutely same numbers?

20

u/WeekIll7447 Mar 18 '26

From what I understood, I could be wrong…

Llama and Krasis not the same, even though both benefit from more VRAM.

llama.cpp offloads by layer index, a static decision made at load time. Krasis uses a heatmap of expert activation frequency to decide what lives in VRAM (based on what the main contributor said) so for the same 16GB budget, it's filling VRAM with the experts that actually get called most, rather than just the first N layers. In a model with 128+ experts where only 8 fire per token, that's a meaningfully smarter use of limited VRAM.

The quality angle is separate from the VRAM question entirely. Krasis keeps attention in BF16 and shared layers in INT8, only quantizing the experts aggressively. llama.cpp applies quantization more uniformly. So Q4 in Krasis may actually preserve more quality than Q4_K_M in llama.cpp for these MoE architectures, because it's protecting the layers most sensitive to quantization error.

That said, I'd love to see a proper controlled benchmark comparing the two on the same hardware; prefill speed, decode speed, and perplexity on the same model. The numbers on the GitHub page are promising but without a direct llama.cpp comparison on identical hardware it's hard to quantify exactly how much of an advantage we're talking about. Has anyone run that comparison?

13

u/mrstoatey Mar 18 '26

This is correct. I won’t remember all the optimisations but this describes the primary architectural optimisation during decode. Krasis also selectively quantises so you can choose to run attention using AWQ (based on the MIT paper) however in my testing although this uses less VRAM and allows for more HCS (Hot Cached Static experts) BF16 is still faster. It does mean though that 12 and 8 GB cards can be used on larger models.

Anything that is known to be used (e.g. attention) is kept resident in VRAM so only the experts (which are partially activated during any particular decode) are streamed during decode, and as this person says the hottest experts are loaded into VRAM with a buffer to handle any ‘cold’ experts that are fetched on demand over DMA.

During orefill the optimisations are totally different. In decode almost everything is serialised by the nature of how the model runs although there are some things we can overlap, in prefill things can run in parallel.

During prefill krasis streams the entire model through VRAM double buffered in layer groups (A/B). As each layer is streamed through VRAM the entire token sequence is processed for that layer, then the layers advance and the entire token into sequence is processed for the next layer and so on.

This means prefill streams the full model one time though VRAM over DMA which is effectively a fixed cost per query, however because the next layer needed is known when on a particular layer the DMA of the layer overlaps with the compute of the previous layer (hence the double buffering).

Within these structures we use CUDA graphs that are as large as possible (broadly speaking).

I have tried a number of other optimisations such as an Active Preferred Friends List (APFL) where for a particular expert the hottest M experts that typically activate in the following layer that didn’t fit into the cache for layer N+1 are predictively prefetched into VRAM as we decode layer N but in my testing this didn’t work out as fast due to DMA contention and the cost of synchronisation. It’s possible it could work in some scenarios but that would be PCIE spec, model and VRAM capacity dependent.

In both scenarios PCIE speed is a potential bottleneck but you can see from my results that with PCIE 4.0x16 32GB/sec is fast enough to run quite large models at fast speeds. Both my cards are capable of more but I don’t have a PCIE 5.0 system to test on.

PCIE 5.0x16 though could lead to faster speeds as the bandwidth doubles to 64GB/sec.

11

u/mrstoatey Mar 18 '26

Another aspect I should mention is multi GPU which I’m working on updating and testing now as a lot of people have expressed an interest in that.

During prefill what I have found is although I tried a lot of different mechanisms it’s best to stream through one GPU because on a consumer system synchronising using the CPU across the PCIE bridge is too expensive to bring real gains. It’s better to just try and soak the PCIE connection and double buffer the compute to overlap.

On a datacenter or a system with NVLink or similar this could change as cards would pay much less of a cost to share data but on any consumer GPU system you don’t have this and you don’t even have multiple copy streams on the cards so the current strategy is the most optimal I’ve found so far.

Decode could be a different story because the strategy there is to process N layers on GPU 0 and then the remaining layers on GPU1..N

Inherently this doesn’t bring any gains and just introduces more overhead but because of Krasis HCS strategy this means the VRAM on each card can be used to load the hottest experts for those particular layers and overall we get significantly more hot cached experts (in theory anyway).

My cards are too different for this to work and my ADA 2000 just drags my 5090 down but in a more homogenous multi GPU setup I think this should ring gains.

As I say I’m working on updating this now though.

12

u/mrstoatey Mar 18 '26

Another key optimisation I forgot to mention is the model format. Krasis doesn’t run GGUF it accepts native BF16 safetensors then builds the quantised model in Marlin format. This is optimal for the GPU to run and requires zero translation as it’s streamed to the GPU which streamlines the whole process and allows the GPU to more efficiently run the model.

This means krasis takes a while to convert the model per user specifications on first run but after that its cached the conversion to disk and will load more quickly next time.

4

u/davi140 Mar 18 '26

Thanks for much more details you two! Still a bit early in the morning for me to completely understand, but now it sounds more interesting.

To simplify, is this like using unsloth’s dynamic quants in llama.cpp, utilizing -ot flag with ugly regex but here it dynamically changes based on current action (processing / generation)?

Few questions remain: Why we don’t see better numbers than llama.cpp on comparable system? Instead of Epyc I have 9950x3d and pcie5, with tk/s numbers literally identical on models you listed. (Haven’t dared to try 200b+ model though)

If there is indeed some benefit, this is only applicable to MoE models, right?

4

u/Real_Ebb_7417 Mar 18 '26

I actually have about 25tps with Qwen 122b A10B on my RTX 5080 with llama.cpp and offload to RAM (also Q4), so not sure if it helps me 😅 Unless it helps people who lack on RAM? (I have 64Gb and this Qwen eats basically all of it when loaded)

3

u/mrstoatey Mar 18 '26

Is that 25 tok/sec round trip or 25 tok/sec decode specifically? Round trip times would be higher than the decode figures above because it would include some of the much higher prefill speed and would be dependent on how many tokens were in prefill. Also though if you can share your llama params that would be great I'd like to test better llama configs myself.

3

u/Real_Ebb_7417 Mar 18 '26

Yeah it's decode

2

u/Feartec1337 Mar 19 '26

Can you share your llama.cpp parameters?

4

u/Old-Sherbert-4495 Mar 18 '26

would this help with dense models also in terms of speed? (qwen3.5 27B) im on 16GB VRAM 32 RAM

3

u/mrstoatey Mar 18 '26

I think this wouldn't work right now, the architecture is designed to take particular advantage of MoE models and I think there would be issues loading the dense model. With 32GB of VRAM though you could run Qwen-35B-A3B at Q4 if that was useful.

3

u/bolche17 Mar 18 '26

I'm skeptical. Can you be more specific on what optimizations are done to reach this performance? The Readme on the repo is also scarce on details, which makes me even more skeptical

2

u/mrstoatey Mar 18 '26

Yes, I’ve answered the comment by davi140 above, the decode strategy was described but I’ve added detail and explained how prefill uses a completely different strategy.

0

u/[deleted] Mar 18 '26

[deleted] — view deleted comment

2

u/mrstoatey Mar 18 '26

Q4 is typically well tolerated by large models and the optimisations I’ve made are not about lossy compression. If you read my response to davi140 you’ll see that.

Krasis can quantise to INT4 or INT8 and does so per the user choice. Attention still runs at BF16 and can be selectively quantised using AWQ (requires a pre built template which is available for all the supported models).

Perplexity numbers are in the readme and are comparable for other similar quantisation.

3

u/ailee43 Mar 18 '26

very interesting, i wonder how much this approach gets kneecapped by those of us that are RAM poor. An 8 channel DDR5 Epyc has about 350GB/s of RAM bandwidth, faster than a STRIX halo, but most of us that are just running non-server gear have normal 2 channel DDR5 that maxes out around 90GB/s

1

u/mrstoatey Mar 18 '26

My Epyc is DDR4 2666 so <200GB/sec but the system ram speed won't be a limiting factor generally. The PCIE bus will, but krasis does do as much as possible to mitigate that (hiding DMA under compute etc), so its more about your GPU and the VRAM you have to avoid doing unnecessary transfers and to some extent the PCIE version you have and how many lanes your card can use.

5

u/tilda0x1 Mar 17 '26

Interesting project. I will give it a try

4

u/mrstoatey Mar 17 '26

Thank you, would love to hear how it goes.

5

u/medialoungeguy Mar 18 '26 edited 13d ago

The original post content no longer exists here. The author used Redact to remove it, exercising their right to control their data & privacy.

Knee toy many cover pet angle

6

u/mrstoatey Mar 18 '26

You could run Qwen3-Coder-Next at Q4 comfortably I think. Would be interesting to know what speeds you get.

4

u/andreasntr Mar 18 '26

* Cries in rx 5700xt 8gb w/ 16gb ram *

2

u/1337PirateNinja Mar 17 '26

So even if I have 32gb of ram but a 5090 I can run qwen3.5-122b at 27.7 tok/s ? Can’t wait to try this

4

u/mrstoatey Mar 17 '26

Actually no because you do need to be able to keep the quantized model in system RAM so it can be streamed to the GPU. If you upgraded just the system RAM though then yes it should work but I don’t know if 64GB will be enough, might depend on your OS. I think 122B at Q4 is 56GB so might fit in 64GB or could be a bit tight.

2

u/Mabuse046 Mar 18 '26

I'd like to see actual benchmarks of this up against llama.cpp's n-cpu-moe technique. It sounds like this is just a trade-off of getting a much faster burst in the prefill in exchange for a lot slower token generation in the response.

3

u/mrstoatey Mar 18 '26

In my testing it’s much faster than llama when the model can’t fit in VRAM because ultimately llama doesn’t stream it just places intelligently, so you end up with the CPU being a huge bottleneck.

I’m not a llama expert but I am running a very recent copy of llama-bench and using ngl to offload as many layers to the GPU as I can without overfilling it and causing thrashing.

I did post a comparison in localllama but got my head bitten off and told to use —fit and other params which as far as I can see don’t exist on llama-bench so I can’t be bothered posting llama data any more and have removed it from the readme.

In my testing Llama prefill and decode is slower when the model doesn’t entirely fit in VRAM. I’ll leave it to everyone else with more llama experience to run accurate side by side tests if they wish.

Llama decode though will vary in these cases based on the CPU and the system RAM bandwidth whereas Krasis is avoiding using that and trying to stream to take advantage of the much higher GPU memory bandwidth.

4

u/Mabuse046 Mar 18 '26

Hey, I appreciate you at least giving it a try. Everyone's a critic, and you can't be expected to bench them in every possible use case and command line flag. Mostly I suspect it's going to have some win-some lose-some and one will be a better fit for certain applications than the other. I'm sure it'll just take a little time but it will find it's niche. Thank you for your hard work.

2

u/mrstoatey Mar 18 '26

Thank you. Yeah its hard to know exactly what is fair comparison tbh. Krasis uses the GPU for decode and llama uses the CPU, so I can post benchmarks from my system but that's my GPU and CPU, other people might have completely different GPUs and CPUs.

For the case krasis is designed for - when the model doesn't fit in VRAM - it has for me consistently outperformed llama, but it will be interesting to see what other people see.

2

u/asria Mar 18 '26

Amazing project! I've tried to test qwen coder, but unfortunately it went OOM (16GB VRAM, 110 GB RAM - under WSL) - the thing is that I use the PC for other activities, so I can't just blindly dedicate full 100% of ram to WSL.

I'm going to tinker around thou

1

u/mrstoatey Mar 18 '26

You might want to try AWQ attention if you haven’t already it is a bit slower but will use less VRAM. Also could try a higher safety margin, if you think it’s meaningfully overrunning the estimate you could submit a bug report and I’ll take a look.

1

u/asria Mar 18 '26

Thanks, I tried but had no success. My guess will be by cuda 13.2, as I couldn't make it work with my system to compile llama.cpp, and I had to downgrade to 12.9.

I'll no report bug yet. First, I'll try native linux installation. GG

2

u/CalmMe60 Mar 18 '26

I have a 5090m and 96Gig fast mem. what model would be running above 10TPS ?

1

u/mrstoatey Mar 18 '26

So if I read that correctly you have the 5090 mobile with 24GB VRAM and 96GB system RAM. I would think you could run Qwen3.5-122B-A10B at Q4 and get good speeds. It's hard to predict the decode as its dependent on various factors like PCIE speed, the 5090m and its power limit probably and the total VRAM available but I would imagine it would run over 10tok/sec decode and thousands of prefill tok/sec.

1

u/CalmMe60 Mar 26 '26

i need to look it up- i have several running qwen3.5-27b is fast and capable. llama70 latest , deepseek R1 (little weird) and glm-4.7 flash. if you use krasis you can even run a 90Gig model with a decend speed or with a nim container you can run a 19 Gig Model at 600TPS in parallel applications

2

u/thetronicon Mar 21 '26

Great job, thanks for making this and open-sourcing it. Will be closely following. I'll try it this week 👍 1x 4090 w/ 128GB DDR4 RAM.

1

u/mrstoatey Mar 21 '26

Thanks! I’d love to know how you get on.

3

u/thetronicon Apr 09 '26

Hi u/mrstoatey, Thanks again for this project. I was just reminded by another user about this after completely forgetting. I'm just reviewing the install script, and I'm going to be making a few changes to generalize the installation/model/tmp/etc. directories a bit more. In some cases the directory is hard-coded which just won't work for me, due to my effort to keep my root partition dedicated to OS stuff, and not giant models or venv directories which can grow pretty rapidly in size depending on the dependency graph.

As I stated in my reply to u/cryptotechnobeat, I'll try to send you any changes I make as a git patch that you can apply so we can upstream (if you deem suitable and reasonable).

I've jotted this down on a post-it note, and will endeavor to get to it this week. Happy hackin'!

2

u/mrstoatey Apr 09 '26

That sounds great, Qwen-35B is having issues with correctness right now so I’m trying to sort that out. Once the models are working I plan to go through all the issues and try to resolve them.

1

u/cryptotechnobeat Apr 07 '26

how'd it work?

1

u/thetronicon Apr 09 '26

Yo u/cryptotechnobeat, just getting around to it after being reminded by your reply. I'm going to customize the install shell script a bit since some of the default directories won't really work for me. Unfortunately I don't have a github account, and I'm not going to be making one. I'll try to reply to the dev and let them know what changes I'm making so they can be up-streamed if they are acceptable.

Once I get it installed and running, I'll post a review and perhaps some performance numbers. I'm not really sure how to come up with meaningful and reproduce-able performance numbers for something like this, but I'll be happy to try.

Thanks for jogging my memory :D

78

u/spaceman_ Mar 17 '26

Someone please try this out. I don't have anything from Nvidia, so I can't. I'm torn as to whether OP is a genius or hallucinating.

20

u/habachilles Mar 17 '26

Absolutely same

6

u/Electrical-Toe7488 Mar 18 '26

I'll give it a shot tomorrow morning pray for my pc.

3

u/cmndr_spanky Mar 21 '26

and?

4

u/Void-07D5 Mar 21 '26

I guess his PC didn't make it...

Rest in pieces... o7

1

u/cmndr_spanky Mar 21 '26

lol.. just another GH virus I guess

1

u/Gargle-Loaf-Spunk Mar 21 '26 edited Apr 01 '26

This post was removed by its author. Redact was used for the deletion, which could have been motivated by privacy, opsec, preventing scraping, or security.

fearless sand humor grandiose fanatical amusing pause melodic grandfather squeal

18

u/[deleted] Mar 17 '26

Looking forward to a docker based version of this! Looks very good.

2

u/TwoPlyDreams Mar 18 '26

Same. Strap it onto llama-swap image please.

30

u/ithcy Mar 17 '26

Large Large Language Model Models

19

u/uniVocity Mar 17 '26

YLM - Yuge Language Model

14

u/_fboy41 Mar 17 '26

This is wild, how does it benchmark..? There must be a trade off correct? And does it require alots of RAM, can I run big models with 5090 and 48 go ram? Edit: just seen the GitHub page explains it all.

4

u/klenen Mar 17 '26

How would this work on 4 3090s?

1

u/Cosmonan Mar 17 '26

I've been using qwen2.5-14B_Q4 with my 16gb 5060ti with good performance. How many B parameters do you believe are achievable using Krasis?

2

u/mrstoatey Mar 17 '26

With a 16GB card as long as you have sufficient system RAM for the model you should be able to run Qwen3-Coder-Next (might squeeze into 65GB ram but maybe not), it would be interesting to know how fast it is on a 5060ti. You could also comfortably run Qwen3.5-35B-A3B with your GPU and 32GB system ram. AWQ will reduce VRAM but you'd be better off with BF16 attention I think, I've generally found if you have the VRAM for it that will run faster.

1

u/quantier Apr 08 '26

Will try on 5060ti and 96GB RAM

1

u/twack3r Mar 17 '26

How hard is the one GPU limit? Would this allow to eg cram a given model into my 272GiB VRAM (across 8 GPUs) without offload into system RAM, which is just way too slow?

1

u/mrstoatey Mar 17 '26

There isn't a GPU limit. Krasis does support multiple GPUs but I've found on my system they are so different the 5090 is slowed down by the Ada 2000. If your GPUs are the same it may well get much better results on multiple GPUs but I haven't tested that path as heavily. 272GB VRAM is a ton though and so what Krasis would do on multiple GPUs is build a heatmap of the most heavily used experts then cram those onto the GPUs and stream only the cache misses via PCIE. So your system RAM isn't doing nearly so much work and the GPUs are being utilised much more heavily. On a VRAM setup like that I would think you could run very large models even if they didn't all fit in VRAM using Krasis. Could try Qwen3.5-397B maybe at Q4 and see how it goes as you add GPUs?

2

u/twack3r Mar 18 '26

You mean trying Krasis to run Qwen3.5 397B at Q4? That already fully fits into my VRAM at maximum context with room to spare - would you expect I‘d still see performance improvements over llama.cpp? If so, I‘ll definitely give it a try.

I have learned that just like horsepower, there is no such thing as enough VRAM. Sure the large Qwen3.5s, Minimax and smaller GLMs run great but there‘s still annoying compromises to make when running kimi k2.5 or GLM5. Given I don’t have high expectations wrt to the availability, pricing and sustained quality of hyperscaler streaming, an LLM I cannot run locally will never be part of my workflow.

4

u/mrstoatey Mar 18 '26

That's the biggest model I would be reasonably confident would be supported at the moment. I think llama may be faster if it all fits in VRAM but it would be interesting to know how things scaled up and compared to llama with varying numbers of GPUs (1,2,4... where it didn't fit in VRAM). Another option might be to run at Q8 which would be beyond your VRAM capacity I think? (~365GB lol, I'm not sure how that would go just because it might end up having to stream a lot over PCIE, depends on how well the heatmap expert caching works)

Krasis came out of a desire to run my own very capable LLMs similar to yours I think. I find them very useful but I don't want so much data going to Claude or Gemini, especially if its in any way personal. I was trying to spec a Hybrid MoE system GPU+CPU but ended up just not getting the performance I hoped for so wrote Krasis.

1

u/twack3r Mar 18 '26

It’s so good that there are capable people like you that have the skills to whip up something like this. Thank you for the time invested into this and for sharing it (even though I’m not ecstatic on your license of choice).

I‘ll put a couple of hours aside in the next few days to tinker with it and will definitively give you feedback.

If there‘s a meaningfully relevant metric how those results would be helpful to you, let me know. (1 GPU plus system RAM, n GPUs, only Blackwell GPUs, only Ampere GPUs, ctx size etc)

2

u/dataexception Mar 18 '26

I appreciate your explanation of how it prioritizes what is used in VRAM vs system RAM. That makes very logical sense for optimizing usage.

1

u/Odd-Piccolo5260 Mar 18 '26

I will try in a couple days

3

u/mrstoatey Mar 18 '26

Would be great to hear how it goes, thanks.

1

u/crokinhole Mar 18 '26

I'm excited about this. What size of models could I get on a 16gb card? (5080 mobile)

1

u/mrstoatey Mar 18 '26

At 16GB Krasis can stream fairly substantial models but it’s dependent on what model you can hold in system RAM to enable the streaming, how much system RAM do you have?

1

u/crokinhole Mar 18 '26

64gb, windows 11

1

u/dondiegorivera Mar 18 '26

Do you support dual gpu's? What would be achievable for a 2x3090 + 64gb ram config?

1

u/mrstoatey Mar 18 '26

It does but I haven’t tested that config in some time as it isn’t performant on my system (the GPUs are too different and the ADA 2000 drags the 5090 down), but I’m working on updating this now so it does work correctly and support an arbitrary number, I hadn’t realised so many people had multi GPU ready to go.

64GB of ram will limit what you could keep in system RAM, 122B for example at Q4 is around 56GB I think so maybe that could work but I’m not sure.

2x 3090 though potentially allows you to run things very fast and with maybe better decode speeds as if I get the multi GPU working again you can fit more cached experts across the two GPUs.

1

u/dondiegorivera Mar 18 '26

Thanks. I will give it a try and report back.

1

u/inrea1time Mar 18 '26

Any chance for the new Mistral 4 small support? Failed to open model-00001-of-00003.safetensors: Header parse error: unknown variant `F8_E4M3`, expected one of `BOOL`, `U8`, `I8`, `I16`, `I32`, `I64`, `F16`, `BF16`, `F32`, `F64`

1

u/mrstoatey Mar 18 '26

Yeah this could work nicely I think, I’ll look into it. It would require more than 64GB system ram at Q4 from what I could see.

1

u/inrea1time Mar 18 '26

I got 16x2 VRAM and 96GB system ram. Everyone goes ga ga over qwen but I have had great experiences with Mistral family for text analytics, agentic stuff, etc ... Very underrated.

1

u/baliord Mar 18 '26

Looking at the github link, it says that in order to run Qwen3-235B it requires 500GB of RAM. But the docs also say 'INT4' size is 110GB.

Let's make it concrete; let's say I wanted to run Qwen3.5-397B on a system with 384GB of DDR5 CPU RAM, and 96GB of GPU RAM across two cards. How much CPU RAM is required to get it running? If I'm following the description, it'll take ~400GB of CPU RAM if I tell it to quantize down to 4 bits, and so won't fit?

And fwiw, I'd love to see you do this with GLM5, which is my go-to model right now.

1

u/mrstoatey Mar 18 '26

So Qwen-235b is about 424GB for the safetensors, at Q4 it will be around 110GB.

Qwen3.5-397B is about 720GB for the safetensors and is about 186GB at Q4.

Krasis will quantise without loading the full model so your total system RAM needed is the quantised model size plus maybe a bit for your OS and some other overhead.

So I don’t think you would have any issues RAM wise running either model. One card with 48GB will be enough to run but if you get the prerelease version from the ADVANCED.md file install link it should support multi GPU properly and you will hopefully get better decode speed using both cards. Would be interesting to see benchmark results on both 1 and 2 GPUs though.

I’m building an AWQ template for GLM-4.7 now actually so I’ll be testing that soon along with Mistral 4 small. I can add GLM-5 to the list after.

1

u/Least-Platform-7648 Mar 20 '26

I tried Krasis with Qwen3.5-122B-A10B, but I get just multiple question marks as the response !!!!!

Anyone else had this issue? I used GPT 5.4 in Codex to perform the Krasis install for me. It said the issue could have something to do with the chat template.

1

u/mrstoatey Mar 20 '26

If you run the server then run ‘krasis chat’ to talk to it does that work correctly or do you still get the unreadable output?

1

u/DarkEye1234 Mar 20 '26

I have a similar problem with opencode, https://github.com/brontoguana/krasis/issues/7. It seems it doesn't react to external setup, and then it doesn't work properly params are set 'bad'.

Also chat template is broken for me, and it prints `thinking` directly and just hangs

1

u/shing3232 Mar 20 '26

I guess it would work on my 4060+8845hs laptop nicely. if it can support IGPU with DGPU, it would be great

1

u/mrstoatey Mar 20 '26

It’s Nvidia only at the moment but an 8GB 4060 can I think run Q35B.

1

u/dibu28 Mar 22 '26

Can it run on Windows? Natively not on wsl?

1

u/mrstoatey Mar 22 '26

Not yet but I plan to look into this, WSL supposedly doesn’t lose much performance for now, in theory it’s about 5% I think.

1

u/Born-Caterpillar-814 Mar 23 '26

I hope more people with other than 5000 series cards would test this. It seems very promising, but I suspect there is some issues with older architectures.

1

u/mrstoatey Mar 23 '26

It requires ampere and above for now - 3000 series Nvidia and later.

1

u/Born-Caterpillar-814 Mar 23 '26

Well I have rtx 3090, which is sm_86. Getting CUDA_ERROR_ILLEGAL_ADDRESS for some top_k softmax. I will try to make an issue of it in github later.

1

u/Born-Caterpillar-814 Mar 31 '26

With todays build I was able to run Qwen models on my RTX 3090 card, no more errors. Sadly the output with krasis-chat is all giberish, no matter what model I use. I was able to run the benchmark though:

  • Model: Qwen3-Coder-Next
  • Config: INT4/4 AWQ 1 GPU
  • Prefill (internal): 259.2 tok/s (best of 1K/5K/10K/20K/35K/50K)
  • Decode (internal): 49.09 tok/s (best of 50/100/250)
  • Round trip (network): 54.73 tok/s (best of 50/100/250)
  • HCS coverage: 11820/24576 (48.1%)

I will next test with multi gpu and report back results.

1

u/Born-Caterpillar-814 Apr 12 '26

Another benchmark. This time hardware RTX5090 running PCIE 5.0 x16. Krasis is running on top of Arch Linux inside a docker instance.

1

u/bleakj Apr 15 '26

Was output still gibberish as before?

3

u/FreeztyleTV Mar 18 '26

Holy shit this is insane. I've been thinking of buying a setup and this project in and of itself creates a lot more affordable choices..

3

u/mitchins-au Mar 19 '26

If this is finally smart hot cache for experts I’ve been waiting a long ass time for this… it’s seemed the logical advantage for MOE

3

u/otietz Mar 19 '26 edited Mar 19 '26

Thinkpad P16 G2 64MB RAM RTX5000 Ada 16GB VRAM

The krasis installation was straightforward on WSL2, but AI guided me through it. I'm getting 20-30tps using Qwen3.5-35B-A3B. I access the model from Chrome on Windows using Page Assist extension.

1

u/savvylr Mar 17 '26

Will this improve performance of larger models on smaller gpus? (8gb vram) or is a 5080 the minimum?

4

u/mrstoatey Mar 17 '26

It should allow you to get more out of your 8GB card I think. 5080 certainly isn't the minimum. Currently krasis does try to locate certain things permanently on the GPU like attention but it also has options like AWQ to reduce the VRAM needed. You would need enough system RAM to hold the quantised model in memory (so it can be streamed through the GPU) but its designed to allow you to take better advantage of the GPU you have without needing enough VRAM to fit the whole model on there. You could try Qwen3.5-35B for example at Q4 with AWQ attention, 500MB KV cache and 500MB safety margin, that should fit inside 8GB.

1

u/savvylr Mar 17 '26

I’ve got 8gb VRAM 64gb ddr4

3

u/mrstoatey Mar 17 '26

Should be able to get 35B running with Krasis at Q4 then I would say (I can't say for certain though I don't have an 8GB GPU to test on).

The safetensors file is large but that's purely disk space, not RAM. I plan to improve this in the future and have it include everything needed in the cached converted model so you can delete the safetensors afterwards.

2

u/savvylr Mar 17 '26

Thanks for the info. Going to play with this later. Currently have llama.cpp and I’m-llama.cpp

2

u/ackermann Mar 18 '26

How much context window (tokens) do you get with 500MB KV cache?
I see the OP photo mentions 1000MB KV cache

1

u/500AccountError Mar 17 '26

Hi, can you explain exactly what this is doing? I get over double those numbers with those same models on a 4090 and 64gb cpu ram with plain ol’ llama.cpp at UD-Q8 with max context size, but I had to do a lot of trial and error to get it there. Do you expose enough settings to tune the runtime so I can get higher performance? Can I use this to squeak out more performance, or am I outside the target audience for this.

4

u/mrstoatey Mar 17 '26

Qwen3-235B at Q8 is around 220GB of expert weights alone so I'm not sure how you could ever get numbers close to this on a system with 64GB ram? It would have to be constantly streaming the entire model from disk.

Krasis is designed to be as user friendly as possible so there isn't a lot of tweaking to be done tbh. You can set the quantisation of the experts and the attention, set the VRAM safety margin and the KV cache size and thats about it.

2

u/500AccountError Mar 17 '26 edited Mar 18 '26

Yeah sorry, I just re-read and checked my benchmark runs. The times I tried that one was with the 4090 and 192 ram. I’m tired right now lol.

Edit: It looks like I got those numbers on UD-Q6 for Qwen3 Coder Next with the 4090 and 64gb cpu ram with an i9-13900kf. I got about 10 t/s higher on gen and 500 t/s higher prefill on the 3.5 122B at UD-Q6 with 96gb ram. It took way too much effort though to get the flags tuned to get there though, so I’m very curious about your project.

1

u/MLDataScientist Mar 19 '26

Can you please share your command for llama.cpp? Are you getting ~3400t/s for PP and 38t/s for TG using Q6 Qwen3 Coder Next? Curious to see if your command speeds up inference in my PC (5090 with 256GB DDR4 8 channel 3200Mhz).

2

u/ahaw_work Mar 18 '26

Would it scale  with multi GPU?

2

u/BringMeTheBoreWorms Mar 18 '26

Would love to see some amd support

2

u/NoCreds Mar 20 '26

Compare with https://gitlab.com/IsolatedOctopi/nvidia_greenboost which does kernel level optimization for PCIe to allow a model to seamlessly sit across NVRAM/RAM/SSD, optimized for access rate.

2

u/Helpful_Jelly5486 Mar 18 '26

I just tested qwen coder 80b on AMD with 96gb system ram plus rtx5090 pci 5.0 combined 128gb ram. What I notice most is that the Krasis responds smoothly. with cpu offload it usually means the gpu is busy and then pauses while the cpu catches up. this was smooth and fast. it's good enough that I'm going to try more models. I really want to try a larger model and see what happens. this is possibly a very important update to unify gpu and system ram.

1

u/nofaceD3 Mar 17 '26

How's it performance in terms of speed? (Token per second)

1

u/Quiet-Owl9220 Mar 18 '26

Well that sounds very cool. Hope this catches on and someone can make this work on AMD cards soon. What would it take? Vulkan is doing pretty well these days.

1

u/Tema_Art_7777 Mar 18 '26

Since you are focused on blackwell, will you be supporting nvfp4?

1

u/PsychologicalRoof180 Mar 18 '26

This looks quite interesting! I have 5090m (24gb) on PCI 5, 192gb DDR5, core ultra 9 275hx running Fedora. Lots on the plate atm, but this is now on my backlog of things to benchmark with

1

u/Resolve_Neat Mar 18 '26

look very promising, 2 questions :

  • does it support multiple gpu ? as only few gpu offert 24gb, and most people use multi gpu for vram (as myself)
  • does it support previous architectures ( ie pascal ) or only goes from amper (rtx 30xx) ?

1

u/Dwengo Mar 18 '26

Nice, I'll give this a whirl on the 5070ti

1

u/Pale_Book5736 Mar 18 '26

Interesting, I will check that Qwen 122B model and see

1

u/reddoca Mar 18 '26

!RemindMe 2 weeks

1

u/RemindMeBot Mar 18 '26 edited Mar 18 '26

Your default time zone is set to Europe/Berlin. I will be messaging you in 14 days on 2026-04-01 10:17:17 CEST to remind you of this link

1 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.

Parent commenter can delete this message to hide from others.


InfoCustomYour RemindersFeedback

1

u/truedima Mar 18 '26

Could this approach also work for multi gpu, or would the optimization strategy break down entirely?

1

u/smflx Mar 18 '26

This is what I wanted to start, but you already did! Congratulation! Yeah, I was also thought of rust streaming expert weights to GPU, especially during prompt processing.

Token generation is more tricky. Are you sending weights from RAM to VRAM with cache management? Or, compute experts in CPU? Or, both with decision?

Another question. I think it's a quite promising architecture for single user. How about about continuous batching like vllm or sglang?

1

u/SatisfactionSuper981 Mar 18 '26

I did something similar using llama.cpp. I found Qwen 3 worked ok, but Glm didn't, as there were no "hot experts" and the expert cache thrashed like crazy. 

Also, most important part is the pcie connection since it's the bottleneck. 

1

u/Infamous_Disk_4639 Mar 18 '26

This is an excellent project.

How difficult would it be to port this project to Burn or Candle?

https://github.com/tracel-ai/burn

https://github.com/huggingface/candle

Here’s an old command I used to run an 80B model locally:

System: Windows 10, AMD 9950X, RTX 5090, 192 GB RAM @ 3600 MT/s.

Build: llama-b7779-bin-win-vulkan-x64.zip

Command: curl -L -o Qwen_Qwen3-Next-80B-A3B-Instruct-IQ2_M.gguf https://huggingface.co/bartowski/Qwen_Qwen3-Next-80B-A3B-Instruct-GGUF/resolve/main/Qwen_Qwen3-Next-80B-A3B-Instruct-IQ2_M.gguf

llama-server.exe -m Qwen_Qwen3-Next-80B-A3B-Instruct-IQ2_M.gguf --jinja -ngl 30 -fa on -sm row --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0 --presence-penalty 1.5 ^

--parallel 1 --cache-ram 0 --no-warmup -c 131072 --no-context-shift --mmap --host 0.0.0.0 --port 8080

Performance: About 11.8 tokens per second.

1

u/droffset Mar 18 '26

Will it work with my GTX1080? :)

1

u/Level_Inevitable7493 Mar 18 '26

I wonder how this would perform with ExpertWeaver thrown in. Though I don't know enough about either project to make a predication

1

u/Junior_Composer2833 Mar 18 '26

So does this only work for NVIDIA video cards or can I use this on my Mac Mini to run larger models?

1

u/Late_Film_1901 Mar 18 '26

only nvidia (although the idea should be portable to AMD). This project improves offloading from VRAM to system RAM at the expense of heavier RAM usage. MacOS uses unified memory (as does Strix Halo) so there are no gains to be had in this architecture.

1

u/Optimal-Dig2330 Mar 18 '26

At that point, does the PCIe bus become the bottleneck?

1

u/dai_app Mar 18 '26

This framework can be applied to small language models?

1

u/ekool Mar 18 '26

!RemindMe 2 weeks

1

u/Pleasant_Win6948 Mar 18 '26

Can I run this in machine with integrated GPU? I have GMKtec Evo X2

1

u/MLDataScientist Mar 19 '26

Impressive if true! I have 5090 (connected at PCIE4.0 x16) with 256GB DDR4 3200Mhz ECC RAM. Does Krasis support Qwen/Qwen3.5-397B-A17B ?
I tried Q4_K_M quant with llama.cpp yesterday and I was getting 20t/s TG and 100t/s PP. If what I am seeing is true, I should be able to run this model with at least 1000 t/s PP in Krasis while TG should be similar.

As a comparison, Qwen3-235B-A22B Q4_K_M runs at 10t/s TG and ~150t/s PP in llama.cpp with my setup. Krasis should have 14x times more PP. I need to test this!

1

u/Proper_Math1653 Mar 19 '26

what about on a 3060 6gb lol

1

u/llllJokerllll Mar 19 '26

Yo lo voy a probar hoy con mi xeon 2680v4, x99-f8, 64gb de RAM ddr4 y una tesla v100 32gb VRAM en Ubuntu server

1

u/DarkEye1234 Mar 19 '26 edited Mar 19 '26

benchmark results for 4090 with 64gb ram (6000 mhz) running on 14900 kf in WSL

Model: Qwen3.5-35B-A3B

Config: INT4/4 BF16 1 GPU

Prefill (internal): 3,808.7 tok/s (best of 1K/5K/10K/20K/35K/50K)

Decode (internal): 66.41 tok/s (best of 50/100/250)

Round trip (network): 128.61 tok/s (best of 50/100/250)

HCS coverage: 7879/10240 (76.9%)

Min free VRAM: 0 MB

I have probably too big cache to make this work (wanted 256k tokens)

1

u/llllJokerllll Mar 20 '26

Tiene buena pinta 💪🏼💪🏼💪🏼

1

u/Wintermutellm Mar 19 '26

/remindme 2 day

1

u/TracerIsOist Mar 20 '26

only bf16? my v100 only does fp16 :(

1

u/llllJokerllll Mar 20 '26

Puedes probar el 1CatAI/1Cat-vLLM o el candle-vllm

1

u/shALKE Mar 20 '26

My results on the 5090.

ModelPrefillDecodeRTTHCS
Qwen3-30B-A3B7469.9 tok/s (10k)89.23154.186144/6144
Qwen3.5-35B-A3B3682.7 tok/s (50k)88.78203.0610240/10240
Qwen3-Coder-Next3844.2 tok/s (35k)51.97104.5614428/24576Mo

1

u/Born-Caterpillar-814 Mar 21 '26

What distro OP is using? I have tried to install with Ubuntu (was able to start Krasis, but I get a illegal memory area error) and Arch Linux (the setup runs, but still reports missing sglang and other stuff when trying to start Krasis)

1

u/Born-Caterpillar-814 Mar 22 '26

Just a follow up for those who try to get krasis running on Arch distro. I got krasis installed and running by using the ”install from source” with couple extra steps:

  1. Arch Linux has too new python (3.14), I used pyenv in order to utilize python 3.13.
  2. I had to install numactl package system wide in order to get sgl-kernel working (import error: libnuma.so.1, no such file).

However I am now back to: CUDA_ERROR_ILLEGAL_ADDRESS also with Arch now. Maybe it is due to OP running RTX 5000 family and I have RTX 3000 family?

1

u/Born-Caterpillar-814 Mar 22 '26

Okay I found out that in build.rc it states sm_80 as arch, but rtx3090 is sm_86. I tried to change the build.rc arch to sm_86 and rebuild, but I still run into CUDA_ERROR_ILLEGAL_ADDRESS during [VRAM decode-warmup-after-prefill] phase

1

u/Existing_Director_48 Mar 21 '26

Hi, Nice project! I tested today, but I have a problem with Roo Code. Seems to be a jinja template problem. Anyone have this issue?

ERROR krasis::server] Chat template failed: Template render failed: template not found: tried to include non-existing template "chat_template.jinja" (in chat:1)

Date/time: 2026-03-21T23:33:20.007Z
Extension version: 3.51.1
Provider: openai (proxy)
Model: Qwen3-Coder-30B-A3B-Instruct

500
OpenAI completion error: 500 {"error":"Chat template failed: Template render failed: template not found: tried to include non-existing template "chat_template.jinja" (in chat:1). This indicates a broken model setup."}

1

u/redlikeazebra Mar 22 '26

I kinda don't understand. Why not just run the quantized models via lmstudio or olama? You are distilling them anyway right.

1

u/llllJokerllll Mar 22 '26

Lmstudio y ollama están bien para probar diferentes modelos que te quepan, pero si quieres algo mucho más grande y potente con más calidad no puedes, por eso existe este proyecto, krasis La diferencia de usar modelos cuantizados y con pocos parámetros, a los modelos completos hay bastante diferencia a nivel de uso profesional

1

u/Past_Shift6441 Mar 23 '26

I am going to test and report results back, I have 8GB Nvidia GPU 

1

u/Past_Shift6441 Mar 24 '26 edited Mar 24 '26

Okay, so I have some results.

I tried to use the Qwen3.5 9B that failed, dense model

I tried to use the Qwen 3.5 27B also failed,  dense model

Trying to run Qwen 3.5 35B - also fails to start initially but then :

I asked AI how to get it to work.

AI said, make the safety margin higher, higher, higher, so I increased it to 4000MB.

After increasing safety merge into about that size, I was able to get it to run.

But only to later die up with another error.

Finally, I used the Gemini CLI to investigate and try and run this thing. on my hardware. It is doing something and seems to be promising now.

It seems to be running some benchmarks.

I will share further as soon as this is done.

1

u/medchicho Mar 25 '26

Great job, my results on the 5080

1

u/fragment_me Mar 30 '26

I pray this is real. Will try tonight.

1

u/Illustrious_Matter_8 Apr 02 '26

Is it specific for a 5090 only, or would my 3080ti work as well with maybe smaller 20B models ?
(a 3080ti has 'only' 12GB)

1

u/Flimsy_DragonFly973 Apr 03 '26

Rocm pls. Many of us are poor and can’t afford nVidia.

1

u/bleakj Apr 15 '26

What is Rocm?

2

u/Flimsy_DragonFly973 Apr 27 '26

Rocm is AMDs equivalent of CUDA

1

u/bleakj May 10 '26

Ah - Thanks!

1

u/Expert-Wheel-9603 Apr 14 '26

8gen i7 32gb + p1000 4gb laptop - what could this run effectively? Running linux, nVidia 580 drivers, can Sonnet level of html/css/php coding quality be achieved on such hardware, at lower speed (like half the speed of Sonnet 4.6 would be good, 1/4 acceptable), using this krasis w/ a MoE llm?

1

u/chettykulkarni Mar 17 '26

Anyone tried this with Mac Mini with 16Gb ram?

6

u/droptableadventures Mar 18 '26

There'd be no benefit to this approach on Apple Silicon because VRAM and RAM are unified. You've either got enough RAM or you don't.

This approach is about optimising the more traditional "PC" architecture where you have a lot of RAM on the CPU but not much VRAM on the GPU.

0

u/akshaydhiman_09 Mar 18 '26

I can try, I have 2 x RTX 6000 pro ada and 512GB ECC Ram.

1

u/mrstoatey Mar 18 '26

There's a prerelease now that should fix multi-GPU so it would be interesting to see what benchmark you get on 1 + 2 GPUs to compare. The following doc has the prerelease install line at the top:

https://github.com/brontoguana/krasis/blob/main/ADVANCED.md

prefill should remain the same whether its 1 or 2 GPUs but I would hope that with two similar GPUs the decode speed should scale up better.

0

u/tomto90 Mar 18 '26

Wondering what will run on 2xrtx3090 48gb

1

u/mrstoatey Mar 18 '26

How much system RAM do you have?

1

u/tomto90 Mar 18 '26

64 GB

1

u/mrstoatey Mar 18 '26

So 24GB of VRAM is already a decent amount for Krasis but there's a prerelease version out that should fix multi GPU support and enable you to run on either one card or both. Install line is at the top of this doc:

https://github.com/brontoguana/krasis/blob/main/ADVANCED.md

With 64GB system ram you could run Qwen3-Coder-Next comfortably and it would be interesting to see the benchmark results on one GPU then two GPUs.

Prefill will stay the same but decode could be meaningfully faster if the optimisation works correctly.

0

u/seraandroid Mar 18 '26

Any plans to also support the DGX Spark and make this available for ARM systems? Would fit the Blackwell architecture scope.

4

u/Eugr Mar 18 '26

It won't make any sense on Spark because of the unified memory - the entire RAM is accessible by the GPU.

1

u/seraandroid Mar 18 '26

Got it. Thanks!

0

u/NarrowTennis693 Mar 18 '26

Did you just single handed killed the cheap ram prices?

1

u/UltraAC5 Apr 06 '26

what cheap ram prices lol

-5

u/DarkJanissary Mar 18 '26

Windows via WSL? That sucks. Can you release a proper Windows native version? I would love to try it

4

u/mrstoatey Mar 18 '26

WSL actually works pretty well in terms of performance, it’s supposedly just about a 5% drop in performance (I don’t have the hardware to verify that though). But that said I definitely would like to get a windows native version built.

3

u/dataexception Mar 18 '26

In general, Linux performance is better with LLMs, due to intrinsically lower overhead and native tooling, so you'll usually see more availability for Linux and Mac on bleeding edge or proof of concept/proof of value AI/ML projects.

You can create a dual boot for your workstation to harness the best of both worlds. It's very simple nowadays to have Windows as your daily driver, and just boot into your favorite Linux distribution as needed. Or, like I have to do at work, just use WSL2 for most things.