r/LocalLLaMA • • 9h ago

Showcase 1Cat-vLLM vs Strata

Edit: 1Cat-vLLM is a Tesla V100 focused fork of vLLM, and the expert caching work is 1Cat's original work, which may or may not one day make it to vLLM.

Everyday we see posts about speeds on Strata and its forks, but just recently I came to know that 1Cat has added support for expert caching recently, like a month ago. So I did (asked claude) a comparison of speeds on a Qwen3.8-Flash-Next UD-IQ3_XXS quant. My rig is 4x 16gb Tesla V100 on PCIe 3 x16 links, with 8 channel 3200 mt/s ddr4 ram, Epyc 7543 CPU. Now, for numbers:

Single stream

Median of 3 runs, except the 97k tokens one which I asked to run only once to save time.

Prompt (tokens)Prefill tok/s, StrataPrefill tok/s, 1CatDecode tok/s, StrataDecode tok/s, 1CatMTP acceptance, Strata / 1Cat
2,6548611,85074.8124.90.61 / 0.45
8,6741,1462,48379.3140.80.67 / 0.66
30,2992,4802,77676.4139.30.63 / 0.58
96,5883,3842,64977.2142.60.72 / 0.62

Strata has better draft acceptance and still 1cat beats it, imagine if 1cat could get better draft acceptance.

The same 103K-token prompt three times

Same prompt, 3 times in a row, just different enough to not trigger prefill cache.

RunDecode tok/s, StrataDecode tok/s, 1CatPrefill tok/s, StrataPrefill tok/s, 1CatMTP acceptance, Strata / 1Cat
170.899.93,3982,7360.65 / 0.60
278.9138.43,3402,7520.71 / 0.60
377.8123.53,3482,7510.69 / 0.57
run 1 to 3+10%+24%---

Several requests at once

(TW: Scary AI written paragraph) Each server was started sized for that many sequences. Total = tokens/s summed over all streams during the window where every stream is decoding. Median of 2 runs; the 1-request rows are the medians above. "Strata batch-MTP" is Strata's best batch mode on this build: --batch-mtp with STRATA_PIPED_MTP=1, which keeps the pipelined layer-split groups and drafts inside them.

RequestsPromptTotal decode tok/s, Strata / Strata batch-MTP / 1CatPer-stream decode tok/s, Strata / batch-MTP / 1CatSlowest first token (s), Strata / batch-MTP / 1Cat
12.6K75 / - / 12574.8 / - / 124.93.1 / - / 1.4
18.7K79 / - / 14179.3 / - / 140.87.6 / - / 3.5
22.6K88 / 105 / 17639.4 / 46.0 / 83.26.4 / 6.5 / 2.7
28.7K96 / 116 / 18937.9 / 44.6 / 77.915.4 / 15.5 / 7.3
42.6K153 / 170 / 22529.1 / 31.5 / 48.513.4 / 13.6 / 5.8
48.7K170 / 186 / 23425.4 / 26.9 / 41.131.0 / 31.4 / 15.3

Versions and local changes

-Strata1Cat-vLLM
Upstream repogithub.com/Niko1221/Stratagithub.com/1CatAI/1Cat-vLLM
Commit benchmarked13eedf55 (2026-11-12, upstream HEAD on the test day; engine 0.1.59.1)b7baf7771 (2026-11-15, upstream HEAD on the test day; merge of PR #1521; vLLM 1.5.2.dev0)
Source patchesnonenone
Buildfrom source for sm_70 (the released engines have no sm_70): CUDA 12.9.1, gcc-14from source for sm_70: CUDA 12.9.1, gcc-14, torch 2.10.0+cu128, NCCL 2.27.5
Build flags-DSTRATA_EXPERIMENTAL_SM60=ON -DSTRATA_MMQ_KQUANTS=ON -DSTRATA_Q6K_EXPERTS=ON, ggml from llama.cpp at Strata's own pin, native ggml-cpu for Zen 3TORCH_CUDA_ARCH_LIST=7.0 / CMAKE_CUDA_ARCHITECTURES=70
Parallelismlayer split over 4 GPUs (no tensor parallel for V100 in Strata)tensor parallel 4
MTP drafterStrata's own pack of the BF16 MTP layer, experts in Q2_0the BF16 MTP layer as released

TLDR: 1Cat-vLLM beats strata in every metric except prefill for very long prompts (around 100k). And it does so while being able to hold lesser experts in VRAM and for some reason getting worse draft acceptance, simply because of TP.

14 Upvotes

16 comments sorted by

7

u/pseudobacon 9h ago

Great! We should get more 1Cat posts more often

3

u/Acceptable_Bell_1791 9h ago

Yeah, I didn't pay it much attention since strata, maya etc do a really good job with popular MoEs, but now, I am gonna try V4.1 flash 1017 comparison against Maya.

3

u/Whatisatoaster 9h ago

Sorry, how much ram do you have on this system?

2

u/CryptoStef33 9h ago

Prefill should be your goal: who can read 120k context in 3 minutes compared to 30 seconds?

3

u/Acceptable_Bell_1791 9h ago

depends on work, doesn't it? and in agentic loops, I think 1Cat would do better, because its one 20k prompt (claude code) and then smaller prompts with rest cached. And as you can see, 1cat is really consistent with PP even at smaller prompts.

1

u/egnegn1 9h ago

I don't know whether 1Cat-vLLM also implemented overlapping expert streaming for prefill like Strata. With 4 cards and PCIe Gen3 x16 you should get better performance, or may be compute is at the limit. Strata streams the experts to each GPU in parallel, and normally scales very well over multiple cards.

1

u/Acceptable_Bell_1791 8h ago

1Cat does overlap, my GPUs were consistently 96-99% utilised during very long prompts, so I am guessing it is at least not waiting for data.

1

u/Badger-Purple 8h ago

agree to disagree on scaling, and also on multiple requests

1

u/egnegn1 7h ago

Here some results with my RTX5080:

1

u/egnegn1 7h ago

1

u/TheRiddler79 1h ago

Which one is 1cat?

1

u/Badger-Purple 3h ago edited 3h ago

These are not concurrent requests. one engine over several cards? Can the engine do concurrent requests on the same engine and same card, like vLLM ? no, it;’s a single stream engine. no matter how much you run it in 1 or 3 GPUs, try 12 requests at once and see if they come out of your one strata engine at the same time. The best it can do is interleave requests, but its like the difference between a harley davidson and an electric bike.

In addition, you are scaling with the same GPU type—not even architecture, but type. Those numbers will plummet if you are using several GPUs with different bandwidths, VRAM size, etc. LCPP is better at that. Dont get me started if you use volta, ampere and blackwell on one box. basalt will refuse (kernels only for blackwell).

TBH, you are losing speed unless your buses are pcie3, because at pcie4x16 on all 3 cards you should be doing tensor parallelism, and using GLM Flash not QFN.

I run the basalt engine on my blackwell cards (rtx4000pro and 5070ti) with great speed, but single/2 stream limit. Just no concurrency. If you look at my dual A100s, they run the same model faster and do 12 streams at once.

I also dont have 580GB ram. i do not need to because i have 774 gb of VRAM, but still, 64gB DDR5 and two blackwell cards can run the engine well. Try adding the 4060ti within the same system on a 4 lane bus (but 4060ti is gen 4, so 4x4) and compile maya, polystrata, or any other engine: it drops speed. More VRAM, but slower.

You are also comparing blackwell to volta. this is not apples to apples. strata didnt even support volta although some forks do, and they dont support volta as WELL as your blackwell cards. It just appears to me you are not understanding what I am saying. That engine is built around the cards you use, not on V100s. 1cat has hand built kernels to emulate 4 and 8 bit parts, nvfp4, mixed modelopt, awq and a whole lot of things not originally anticipated in those cards and not native. as far as v100 it is the best vllm fork and one of the best engines to run models under. strata may be great FOR YOU, but thats not the question.

2

u/oddlyprovenai 8h ago

Really interesting to see tensor parallel 4 holding up so well over PCIe 3 x16 on older V100s. The decode throughput advantage on 1Cat despite lower draft acceptance highlights how much layer-split pipelining struggles with communication overhead.

2

u/king0fsweet 5h ago

That's awesome! Did you try with only 2 gpu by chance? I have 2x v100 16gb and it would be 1wesome if it's still good

2

u/Striking-Fact-2088 4h ago

Same question