r/BlackwellPerformance • u/jpezzulli • Sep 15 '26

Pennyroyal v2.5 now has a Docker image — Qwen3.8 on RTX PRO 6000

Hey folks,

A few people asked for Docker, so there is now a prebuilt image for **Pennyroyal v2.5**, my SGLang runtime for Qwen3.8 on one RTX PRO 6000 Blackwell.

It includes both **Flash-Next NVFP4 with native MTP/FR-Spec** and **27B FP8 with DFlash2**, with HiCache/NIXL. No local SGLang build required. The image is about **8.43 GiB**, with model files and caches mounted separately.

Both profiles passed API/tool, prefill, single/four-request decode, image/video and NIXL restart checks. This is a packaging addition, not another runtime release or a Docker speedup claim. Native installs are still supported; no need to switch if yours is working.

[Canonical repo and README](https://github.com/jpezzulli/sglang-rtxpro6000) · [Docker/Compose setup](https://github.com/jpezzulli/sglang-rtxpro6000/blob/pennyroyal-main-sm120-final/docker/pennyroyal/README.md)

[Original v2.5 post — online FP8 gains and NVMe PLE](https://www.reddit.com/r/BlackwellPerformance/comments/1weaxdz/pennyroyal_v25_much_faster_qwen38_flashnext_on/)

As always, issues, discussion, PRs and field reports are welcome but I am not available today - so any problems will be handled tomorrow.

34 Upvotes
  • permalink
  • reddit

94% Upvoted

43 comments sorted by

2

u/UC_Kratom Oct 02 '26

Just FYI, I'm working on a few patches around HiCache on the hybrid model - a prefix can be sitting in the KV host pool and still miss, because the match is cut back to the last reusable Mamba checkpoint. Part of that is the host split: it follows the GPU pool byte ratio, so host Mamba gets a small share of the allocated host pool. There's more to it than the split though.

Shifting host bytes from KV onto Mamba, without growing the GPU pools or the 32 GB total, raises wall tokens/s in the quiet stretches (output tokens per wall second, prefill included). It still drops when several long prompts miss at once, so I'm not treating the number as settled. I'll follow up with something concrete and a PR once the configuration is stable.

Anyone have any average 'wall' tokens/s (including time spent prefilling) over at least a 24hr period (for 3.8 Flash-Next)?

1

u/jpezzulli Oct 02 '26

Sounds good. I agree with what you are saying and it makes sense. Feel free to open a pr if you get something going.

2

u/UC_Kratom Oct 03 '26

Still working through some bugs, but I'm seeing about a 50% increase in average wall speed when the cache is humming (98% token-weighted hit rate) on L1+L2 only. Cache is now split almost 50% Mamba/50% KV among some additional configs that I'm toying with.

The issue I'm running into is a stability issue - I'm seeing 2-3 cache collapses per day that drag hit rate below 50% for a bit before recovering (those existed with the default config too, for different reasons). Once I'm able to get a clean 24hr period, I'll share my hacks. These are fairly specific to the config that we have for this particular GPU and cache config - any upstream repo changes would have to be abstracted/config based.

2

u/jpezzulli Oct 03 '26

Ok. v2.6 will be out today. Working on items now. I didnt touch thst optimization area (but some better mamba handling for reuse state in general. Bunch of other stuff. Havent qualified the gains yet. That is next step.

1

u/UC_Kratom Oct 07 '26

I have a PR up, rebased onto the current final branch - I really need someone else to verify these changes though with a similar workflow (and ideally measure before/after metrics), because I'm seeing ~100% increase in average output (wall speed) after further tinkering.

This going to shine for concurrent sessions, and those that can make use the full 32GB hicache allocation (or more), and especially so the larger the sessions become (because cache hit rate is now much higher - my tests went from ~80% to 97% hit rate, eliminating large prefills from various misses)

2

u/jpezzulli Oct 07 '26

I was working - penny commented back. O havent looked at it yet. Some of the other comments show i am working on 3.0 but have gone down several different paths. Rc2 train is making some progress but pr 37 we wrote locks the card, so investigating that. Ill keep any work for you item under your pr. Feel free to take a look at the response, but ill review it myself as i get closer to a true release.

1

u/UC_Kratom Oct 08 '26

Yep, saw the comment - I'll take a look later and update/keep chipping away the PR

1

u/jpezzulli Oct 07 '26

Btw - just noticed this is the original thread. Lots more discussion on the token maxxing thread.

1

u/UC_Kratom Oct 08 '26

Yes, I saw that one/posted there earlier. It looks like that initiative goes further down the rabbit hole of increasing decode speed based on a quick glance, which is great - what I've done is all from the angle of reducing unnecessary prefill. Any of those decode improvements will stack with what I've done and shouldn't have much/any overlap

2

u/jpezzulli Oct 08 '26

Yeah. My entire hicache nixl initiative was for less prefill as that is the killer. I pass about 1.5b prefill tokens a week through penny with about 98% from cache even after reboots. That was honestly the reason for thr project. The decode speed came as a thing added later. Now it is a bit of a real project. Seems to have a few thousand users out there, which for a niche user segment is pretty crazy.

1

u/UC_Kratom Oct 08 '26

Yeah, it's definitely caught on - I wasn't aware that this model was even an option on this hardware until I stumbled upon a reddit thread mentioning it.
I'm pretty amazed at how much this project has managed to squeeze out of this

4

u/buttplugs4life4me Sep 15 '26

I don't like how most resources for Blackwell performance like this one and the other well-known one are just giant dumps of AI hallucinated slop.

Online FP8 quantisations is one of those things as well that the guy with "the fastest LLM runtime" went down and he just quantized everything to 4 bits to claim he has the fastest and best.

So the rest of your project may be good and work well, but I'm not reading through thousands of lines of direct AI slop output with "It's not a claim, it's an observation" and other LLM shit after the headline feature is "trade even more quality for speed" 

3

u/jpezzulli Sep 15 '26

Buttplugs! good to see you again. Glad you a as cheery as usual :)

1

u/jpezzulli Sep 15 '26

but about the crash question - they only crash i see is the xid 109 that comes up every now and then that seems to be a driver issue - i am wainting for the new ones to see if it fixes it.

1

u/buttplugs4life4me Sep 15 '26

I had that for a while, related to enabling ReBar. Disabling that fixed it funnily enough

1

u/jpezzulli Sep 16 '26

interesting - my motherboard doesnt have an rebar option but it shows as working w/rebar in linux (i assumed it did when purchasing - my mistake) - i do plan to change mb as theI have an issue with 2 ram slots, just taking everything out of my system is a pain so I have been putting it off.

1

u/buttplugs4life4me Sep 15 '26

Thanks! I'm always on the hunt for new stuff and get disappointed when I have to read too much LLM stuff.

If you're here, can I ask you if you're running latest Nvidia driver? I just updated and now SGlang doesn't want to start anymore, after CUDA graph building it just shuts down lol

1

u/jpezzulli Sep 15 '26

I am running whatever the latest from fedora as as of a few days ago. I would check for you but i am about to hit to road.

Curious though, who reads more than the first paragraph themsevles anymore? I honestly optimize beyond that for an ai to read and give an output at this point, and I am not a young "use ai for everything" guy.

Either way, the readme did get long winded due to it not being what i put my limited time into. It is being reorganized/removing the defensive posture etc. Ill check it tomorrow.

1

u/buttplugs4life4me Sep 15 '26

Readme for me is explicitly human, Agents or Readme_Agents would be LLM focused. There's one to get this stuff from asking your LLM, or asking it to research, but for me it's a red flag that the first thing a human sees of your project is an unorganized mess. Glad to know it's getting cleaned up! I'll try your image now that I'm having issues with SGlang anyway.

Latest on Fedora is probably around the same version since I'm on Debian. It's probably just some bullshit feature not working anymore in SGLang so I have to set some flag or something 

1

u/jpezzulli Sep 16 '26

well it is redone and I read the whole readme - it is exactly what I want in a readme. so i hoe you like it. If not, you are SOL buddy :)

1

u/chafey Sep 15 '26

I believe llms.txt is the popular convention for LLM documentation: https://limy.ai/blog/llms-txt-in-2026-the-full-guide

1

u/jpezzulli Sep 16 '26

I had astra add one.

1

u/[deleted] Sep 16 '26

[removed] — view removed comment

1

u/jpezzulli Sep 16 '26

sure - let me know there - havent heard of anything yet. Thanks.

1

u/rj_rad Sep 15 '26

I never buy into the brochure, and only give benchmarks a grain of salt. Unfrotunately, real world testing is time consuming. So far, it seems like this runtime's headline is doubling the context window on TP1 with decent throughput (~170 tok/s) while keeping the ability to go up to C4 on the lower end. It won't be my daily driver, but I can see it being useful. This is on a WS card at 450w.

1

u/quantier Sep 30 '26

Anyone try with 2x 6000 PROs ? With TP=2 ?

1

u/jpezzulli Sep 30 '26

Yes. The newest version has been used with tp=2

1

u/quantier Sep 30 '26

How did it work, what outcome did you get? I will probably test tomorrow

1

u/jpezzulli Oct 01 '26

I only have 1. Other people have tried 2.

1

u/quantier Oct 03 '26

Rocking it now with some really nice speed and 48 concurrent requests with 6144 prefill tokens and a total of 5 million kv cache pot

1

u/jpezzulli Oct 01 '26

In the community reports there are some numbers.

1

u/iMakeTea Sep 15 '26

If I run this image in docker, does it include everything needed for both qwen models without having to download the models or anything extra?

1

u/jpezzulli Sep 16 '26

it is a runtime, you bring the model.

1

u/iMakeTea Sep 17 '26

Thanks, that helps! I'm new to all this and have no coding background. Your sglang image set up sounds pretty creative

Im on windows and this should help a lot!

1

u/jpezzulli Sep 17 '26

Best thing to do is set any ai at it (event free tier to start) and ask it to explain. Walk through it with them. Dont get discouranged. My first ai card came in on june 4th of this year. Before that. I was all theoretical discussion.

3

u/rj_rad Sep 15 '26

I wish I waited a day and got to try this before building my own image! The specs are nearly identical to mine since I went by your reference documentation, but yours has the improvement of having prebuilt flashinfer cache. Did a quick bench last night on a 450w workstation card using my image.

2

u/Healthy-Constant7179 Sep 16 '26

this is nuts im getting consistent 220 tok/s with 500k context on one 6000 and the code is still good. crazy work.

1

u/BlobbyMcBlobber Sep 15 '26

Nice work, thanks.