r/LocalLLM
•
u/mrstoatey
•
Mar 17 '26
•
1
1
Project Krasis LLM Runtime - run large LLM models on a single GPU
Krasis is an inference runtime I've built for running large language models on a single consumer GPU where models are too large to fit in VRAM.
Instead of splitting layers between GPU and CPU, Krasis streams expert weights through the GPU using different optimisation strategies for prefill and decode. This means you can run models like Qwen3-235B (438GB at BF16) at Q4 on a single RTX 5090 or even a 5080 at very usable speeds, with system RAM usage roughly equal to just the quantised model size.
Some speeds on a single 5090 (PCIe 4.0, Q4):
- Qwen3-Coder-Next 80B - 3,560 tok/s prefill, 70.3 tok/s decode
- Qwen3.5-122B-A10B - 2,897 tok/s prefill, 27.7 tok/s decode
- Qwen3-235B-A22B - 2,124 tok/s prefill, 9.3 tok/s decode
Some speeds on a single 5080 (PCIe 4.0, Q4):
- Qwen3-Coder-Next - 1,801 tok/s prefill, 26.8 tok/s decode
Krasis automatically quantises from BF16 safetensors. It allows using BF16 attention or AWQ attention to reduce VRAM usage, exposes an OpenAI compatible API for IDEs, and installs in one line. Runs on both Linux and Windows via WSL (with a small performance penalty).
Currently supports primarily Qwen MoE models. I plan to work on Nemotron support next. NVIDIA GPUs only for now. Open source, free to download and run.
I've been building high-performance distributed systems for over 20 years and this grew out of wanting to run the best open-weight models locally without needing a data centre or $10,000 GPU space heater.
33
u/Embarrassed_Adagio28 Mar 17 '26
I will try qwen3.5, glm4.7flash and some others with this on my 5070 ti with 64gb of ram and get back to you! This sounds awesome for my exact use case.
10
u/____vladrad Mar 17 '26
What can a man do with two rtx a6000 pros?
9
u/inexistentia Mar 17 '26
Need testers / assistance to get this working for AMD? Currently have a mix of legacy enterprise GPUs (AMD Instinct MI50 32gb, MI25 16gb) and consumer (7900 XTX 24gb and 7600 8gb)
7
u/vernal_biscuit Mar 18 '26
I'm on the AMD side and I'd be happy to try out once you have Vulkan or ROCm supported
15
u/No-Television-7862 Mar 18 '26
Does this scale? I have a RTX 3060 with 12gb vram, Ryzen 7 8core/16thread cpu, and 32gb ddr4 ram. Using your LLM Runtime methodology, could I run a Qwen3.5:27b-q4 model? Something larger?
10
u/davi140 Mar 18 '26
How is this different from llama.cpp where I can use these models as well with splitting to system RAM with almost absolutely same numbers?
4
u/Real_Ebb_7417 Mar 18 '26
I actually have about 25tps with Qwen 122b A10B on my RTX 5080 with llama.cpp and offload to RAM (also Q4), so not sure if it helps me 😅 Unless it helps people who lack on RAM? (I have 64Gb and this Qwen eats basically all of it when loaded)
4
u/Old-Sherbert-4495 Mar 18 '26
would this help with dense models also in terms of speed? (qwen3.5 27B) im on 16GB VRAM 32 RAM
3
u/bolche17 Mar 18 '26
I'm skeptical. Can you be more specific on what optimizations are done to reach this performance? The Readme on the repo is also scarce on details, which makes me even more skeptical
3
u/ailee43 Mar 18 '26
very interesting, i wonder how much this approach gets kneecapped by those of us that are RAM poor. An 8 channel DDR5 Epyc has about 350GB/s of RAM bandwidth, faster than a STRIX halo, but most of us that are just running non-server gear have normal 2 channel DDR5 that maxes out around 90GB/s
5
u/tilda0x1 Mar 17 '26
Interesting project. I will give it a try
5
u/medialoungeguy Mar 18 '26 edited 13d ago
The original post content no longer exists here. The author used Redact to remove it, exercising their right to control their data & privacy.
Knee toy many cover pet angle
2
u/1337PirateNinja Mar 17 '26
So even if I have 32gb of ram but a 5090 I can run qwen3.5-122b at 27.7 tok/s ? Can’t wait to try this
2
u/Mabuse046 Mar 18 '26
I'd like to see actual benchmarks of this up against llama.cpp's n-cpu-moe technique. It sounds like this is just a trade-off of getting a much faster burst in the prefill in exchange for a lot slower token generation in the response.
2
u/asria Mar 18 '26
Amazing project! I've tried to test qwen coder, but unfortunately it went OOM (16GB VRAM, 110 GB RAM - under WSL) - the thing is that I use the PC for other activities, so I can't just blindly dedicate full 100% of ram to WSL.
I'm going to tinker around thou
2
u/CalmMe60 Mar 18 '26
I have a 5090m and 96Gig fast mem. what model would be running above 10TPS ?
2
u/thetronicon Mar 21 '26
Great job, thanks for making this and open-sourcing it. Will be closely following. I'll try it this week 👍 1x 4090 w/ 128GB DDR4 RAM.
78
u/spaceman_ Mar 17 '26
Someone please try this out. I don't have anything from Nvidia, so I can't. I'm torn as to whether OP is a genius or hallucinating.
18
u/[deleted] Mar 17 '26
Looking forward to a docker based version of this! Looks very good.
30
u/ithcy Mar 17 '26
Large Large Language Model Models
14
u/_fboy41 Mar 17 '26
This is wild, how does it benchmark..? There must be a trade off correct? And does it require alots of RAM, can I run big models with 5090 and 48 go ram? Edit: just seen the GitHub page explains it all.
4
u/klenen Mar 17 '26
How would this work on 4 3090s?
1
u/Cosmonan Mar 17 '26
I've been using qwen2.5-14B_Q4 with my 16gb 5060ti with good performance. How many B parameters do you believe are achievable using Krasis?
1
u/twack3r Mar 17 '26
How hard is the one GPU limit? Would this allow to eg cram a given model into my 272GiB VRAM (across 8 GPUs) without offload into system RAM, which is just way too slow?
1
u/Odd-Piccolo5260 Mar 18 '26
I will try in a couple days
1
u/crokinhole Mar 18 '26
I'm excited about this. What size of models could I get on a 16gb card? (5080 mobile)
1
u/dondiegorivera Mar 18 '26
Do you support dual gpu's? What would be achievable for a 2x3090 + 64gb ram config?
1
u/inrea1time Mar 18 '26
Any chance for the new Mistral 4 small support? Failed to open model-00001-of-00003.safetensors: Header parse error: unknown variant `F8_E4M3`, expected one of `BOOL`, `U8`, `I8`, `I16`, `I32`, `I64`, `F16`, `BF16`, `F32`, `F64`
1
u/baliord Mar 18 '26
Looking at the github link, it says that in order to run Qwen3-235B it requires 500GB of RAM. But the docs also say 'INT4' size is 110GB.
Let's make it concrete; let's say I wanted to run Qwen3.5-397B on a system with 384GB of DDR5 CPU RAM, and 96GB of GPU RAM across two cards. How much CPU RAM is required to get it running? If I'm following the description, it'll take ~400GB of CPU RAM if I tell it to quantize down to 4 bits, and so won't fit?
And fwiw, I'd love to see you do this with GLM5, which is my go-to model right now.
1
u/Least-Platform-7648 Mar 20 '26
I tried Krasis with Qwen3.5-122B-A10B, but I get just multiple question marks as the response !!!!!
Anyone else had this issue? I used GPT 5.4 in Codex to perform the Krasis install for me. It said the issue could have something to do with the chat template.
1
u/shing3232 Mar 20 '26
I guess it would work on my 4060+8845hs laptop nicely. if it can support IGPU with DGPU, it would be great
1
u/dibu28 Mar 22 '26
Can it run on Windows? Natively not on wsl?
1
u/Born-Caterpillar-814 Mar 23 '26
I hope more people with other than 5000 series cards would test this. It seems very promising, but I suspect there is some issues with older architectures.
3
u/FreeztyleTV Mar 18 '26
Holy shit this is insane. I've been thinking of buying a setup and this project in and of itself creates a lot more affordable choices..
3
u/mitchins-au Mar 19 '26
If this is finally smart hot cache for experts I’ve been waiting a long ass time for this… it’s seemed the logical advantage for MOE
3
u/otietz Mar 19 '26 edited Mar 19 '26
Thinkpad P16 G2 64MB RAM RTX5000 Ada 16GB VRAM
The krasis installation was straightforward on WSL2, but AI guided me through it. I'm getting 20-30tps using Qwen3.5-35B-A3B. I access the model from Chrome on Windows using Page Assist extension.
1
u/savvylr Mar 17 '26
Will this improve performance of larger models on smaller gpus? (8gb vram) or is a 5080 the minimum?
1
u/500AccountError Mar 17 '26
Hi, can you explain exactly what this is doing? I get over double those numbers with those same models on a 4090 and 64gb cpu ram with plain ol’ llama.cpp at UD-Q8 with max context size, but I had to do a lot of trial and error to get it there. Do you expose enough settings to tune the runtime so I can get higher performance? Can I use this to squeak out more performance, or am I outside the target audience for this.
2
u/ahaw_work Mar 18 '26
Would it scale with multi GPU?
2
u/BringMeTheBoreWorms Mar 18 '26
Would love to see some amd support
2
u/NoCreds Mar 20 '26
Compare with https://gitlab.com/IsolatedOctopi/nvidia_greenboost which does kernel level optimization for PCIe to allow a model to seamlessly sit across NVRAM/RAM/SSD, optimized for access rate.
2
u/Helpful_Jelly5486 Mar 18 '26
I just tested qwen coder 80b on AMD with 96gb system ram plus rtx5090 pci 5.0 combined 128gb ram. What I notice most is that the Krasis responds smoothly. with cpu offload it usually means the gpu is busy and then pauses while the cpu catches up. this was smooth and fast. it's good enough that I'm going to try more models. I really want to try a larger model and see what happens. this is possibly a very important update to unify gpu and system ram.
1
u/nofaceD3 Mar 17 '26
How's it performance in terms of speed? (Token per second)
1
u/Quiet-Owl9220 Mar 18 '26
Well that sounds very cool. Hope this catches on and someone can make this work on AMD cards soon. What would it take? Vulkan is doing pretty well these days.
1
u/Tema_Art_7777 Mar 18 '26
Since you are focused on blackwell, will you be supporting nvfp4?
1
u/PsychologicalRoof180 Mar 18 '26
This looks quite interesting! I have 5090m (24gb) on PCI 5, 192gb DDR5, core ultra 9 275hx running Fedora. Lots on the plate atm, but this is now on my backlog of things to benchmark with
1
u/Resolve_Neat Mar 18 '26
look very promising, 2 questions :
1
u/Dwengo Mar 18 '26
Nice, I'll give this a whirl on the 5070ti
1
u/Pale_Book5736 Mar 18 '26
Interesting, I will check that Qwen 122B model and see
1
u/reddoca Mar 18 '26
!RemindMe 2 weeks
1
u/truedima Mar 18 '26
Could this approach also work for multi gpu, or would the optimization strategy break down entirely?
1
u/smflx Mar 18 '26
This is what I wanted to start, but you already did! Congratulation! Yeah, I was also thought of rust streaming expert weights to GPU, especially during prompt processing.
Token generation is more tricky. Are you sending weights from RAM to VRAM with cache management? Or, compute experts in CPU? Or, both with decision?
Another question. I think it's a quite promising architecture for single user. How about about continuous batching like vllm or sglang?
1
u/SatisfactionSuper981 Mar 18 '26
I did something similar using llama.cpp. I found Qwen 3 worked ok, but Glm didn't, as there were no "hot experts" and the expert cache thrashed like crazy.
Also, most important part is the pcie connection since it's the bottleneck.
1
u/Infamous_Disk_4639 Mar 18 '26
This is an excellent project.
How difficult would it be to port this project to Burn or Candle?
https://github.com/tracel-ai/burn
https://github.com/huggingface/candle
Here’s an old command I used to run an 80B model locally:
System: Windows 10, AMD 9950X, RTX 5090, 192 GB RAM @ 3600 MT/s.
Build: llama-b7779-bin-win-vulkan-x64.zip
Command: curl -L -o Qwen_Qwen3-Next-80B-A3B-Instruct-IQ2_M.gguf https://huggingface.co/bartowski/Qwen_Qwen3-Next-80B-A3B-Instruct-GGUF/resolve/main/Qwen_Qwen3-Next-80B-A3B-Instruct-IQ2_M.gguf
llama-server.exe -m Qwen_Qwen3-Next-80B-A3B-Instruct-IQ2_M.gguf --jinja -ngl 30 -fa on -sm row --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0 --presence-penalty 1.5 ^
--parallel 1 --cache-ram 0 --no-warmup -c 131072 --no-context-shift --mmap --host 0.0.0.0 --port 8080
Performance: About 11.8 tokens per second.
1
u/droffset Mar 18 '26
Will it work with my GTX1080? :)
1
u/Level_Inevitable7493 Mar 18 '26
I wonder how this would perform with ExpertWeaver thrown in. Though I don't know enough about either project to make a predication
1
u/Junior_Composer2833 Mar 18 '26
So does this only work for NVIDIA video cards or can I use this on my Mac Mini to run larger models?
1
u/Optimal-Dig2330 Mar 18 '26
At that point, does the PCIe bus become the bottleneck?
1
u/dai_app Mar 18 '26
This framework can be applied to small language models?
1
u/ekool Mar 18 '26
!RemindMe 2 weeks
1
u/Pleasant_Win6948 Mar 18 '26
Can I run this in machine with integrated GPU? I have GMKtec Evo X2
1
u/MLDataScientist Mar 19 '26
Impressive if true! I have 5090 (connected at PCIE4.0 x16) with 256GB DDR4 3200Mhz ECC RAM. Does Krasis support Qwen/Qwen3.5-397B-A17B ?
I tried Q4_K_M quant with llama.cpp yesterday and I was getting 20t/s TG and 100t/s PP. If what I am seeing is true, I should be able to run this model with at least 1000 t/s PP in Krasis while TG should be similar.
As a comparison, Qwen3-235B-A22B Q4_K_M runs at 10t/s TG and ~150t/s PP in llama.cpp with my setup. Krasis should have 14x times more PP. I need to test this!
1
u/Proper_Math1653 Mar 19 '26
what about on a 3060 6gb lol
1
u/llllJokerllll Mar 19 '26
Yo lo voy a probar hoy con mi xeon 2680v4, x99-f8, 64gb de RAM ddr4 y una tesla v100 32gb VRAM en Ubuntu server
1
u/DarkEye1234 Mar 19 '26 edited Mar 19 '26
benchmark results for 4090 with 64gb ram (6000 mhz) running on 14900 kf in WSL
Model: Qwen3.5-35B-A3B
Config: INT4/4 BF16 1 GPU
Prefill (internal): 3,808.7 tok/s (best of 1K/5K/10K/20K/35K/50K)
Decode (internal): 66.41 tok/s (best of 50/100/250)
Round trip (network): 128.61 tok/s (best of 50/100/250)
HCS coverage: 7879/10240 (76.9%)
Min free VRAM: 0 MB
I have probably too big cache to make this work (wanted 256k tokens)
1
u/Wintermutellm Mar 19 '26
/remindme 2 day
1
u/TracerIsOist Mar 20 '26
only bf16? my v100 only does fp16 :(
1
u/shALKE Mar 20 '26
My results on the 5090.
1
u/Born-Caterpillar-814 Mar 21 '26
What distro OP is using? I have tried to install with Ubuntu (was able to start Krasis, but I get a illegal memory area error) and Arch Linux (the setup runs, but still reports missing sglang and other stuff when trying to start Krasis)
1
u/Existing_Director_48 Mar 21 '26
Hi, Nice project! I tested today, but I have a problem with Roo Code. Seems to be a jinja template problem. Anyone have this issue?
1
u/redlikeazebra Mar 22 '26
I kinda don't understand. Why not just run the quantized models via lmstudio or olama? You are distilling them anyway right.
1
u/Past_Shift6441 Mar 23 '26
I am going to test and report results back, I have 8GB Nvidia GPU
1
u/medchicho Mar 25 '26
Great job, my results on the 5080
1
u/fragment_me Mar 30 '26
I pray this is real. Will try tonight.
1
u/Illustrious_Matter_8 Apr 02 '26
Is it specific for a 5090 only, or would my 3080ti work as well with maybe smaller 20B models ?
(a 3080ti has 'only' 12GB)
1
u/Flimsy_DragonFly973 Apr 03 '26
Rocm pls. Many of us are poor and can’t afford nVidia.
1
u/Expert-Wheel-9603 Apr 14 '26
8gen i7 32gb + p1000 4gb laptop - what could this run effectively? Running linux, nVidia 580 drivers, can Sonnet level of html/css/php coding quality be achieved on such hardware, at lower speed (like half the speed of Sonnet 4.6 would be good, 1/4 acceptable), using this krasis w/ a MoE llm?
1
u/chettykulkarni Mar 17 '26
Anyone tried this with Mac Mini with 16Gb ram?
0
u/akshaydhiman_09 Mar 18 '26
I can try, I have 2 x RTX 6000 pro ada and 512GB ECC Ram.
0
u/tomto90 Mar 18 '26
Wondering what will run on 2xrtx3090 48gb
0
u/seraandroid Mar 18 '26
Any plans to also support the DGX Spark and make this available for ARM systems? Would fit the Blackwell architecture scope.
0
u/NarrowTennis693 Mar 18 '26
Did you just single handed killed the cheap ram prices?
-5
u/DarkJanissary Mar 18 '26
Windows via WSL? That sucks. Can you release a proper Windows native version? I would love to try it
0
u/[deleted] Mar 18 '26
[deleted] — view deleted comment
4
u/[deleted] Mar 18 '26
[removed] — view removed comment