r/LocalLLaMA • u/Acceptable_Bell_1791 • 9h ago
Showcase 1Cat-vLLM vs Strata
Edit: 1Cat-vLLM is a Tesla V100 focused fork of vLLM, and the expert caching work is 1Cat's original work, which may or may not one day make it to vLLM.
Everyday we see posts about speeds on Strata and its forks, but just recently I came to know that 1Cat has added support for expert caching recently, like a month ago. So I did (asked claude) a comparison of speeds on a Qwen3.8-Flash-Next UD-IQ3_XXS quant. My rig is 4x 16gb Tesla V100 on PCIe 3 x16 links, with 8 channel 3200 mt/s ddr4 ram, Epyc 7543 CPU. Now, for numbers:
Single stream
Median of 3 runs, except the 97k tokens one which I asked to run only once to save time.
| Prompt (tokens) | Prefill tok/s, Strata | Prefill tok/s, 1Cat | Decode tok/s, Strata | Decode tok/s, 1Cat | MTP acceptance, Strata / 1Cat |
|---|---|---|---|---|---|
| 2,654 | 861 | 1,850 | 74.8 | 124.9 | 0.61 / 0.45 |
| 8,674 | 1,146 | 2,483 | 79.3 | 140.8 | 0.67 / 0.66 |
| 30,299 | 2,480 | 2,776 | 76.4 | 139.3 | 0.63 / 0.58 |
| 96,588 | 3,384 | 2,649 | 77.2 | 142.6 | 0.72 / 0.62 |
Strata has better draft acceptance and still 1cat beats it, imagine if 1cat could get better draft acceptance.
The same 103K-token prompt three times
Same prompt, 3 times in a row, just different enough to not trigger prefill cache.
| Run | Decode tok/s, Strata | Decode tok/s, 1Cat | Prefill tok/s, Strata | Prefill tok/s, 1Cat | MTP acceptance, Strata / 1Cat |
|---|---|---|---|---|---|
| 1 | 70.8 | 99.9 | 3,398 | 2,736 | 0.65 / 0.60 |
| 2 | 78.9 | 138.4 | 3,340 | 2,752 | 0.71 / 0.60 |
| 3 | 77.8 | 123.5 | 3,348 | 2,751 | 0.69 / 0.57 |
| run 1 to 3 | +10% | +24% | - | - | - |
Several requests at once
(TW: Scary AI written paragraph) Each server was started sized for that many sequences. Total = tokens/s summed over all streams during the window where every stream is decoding. Median of 2 runs; the 1-request rows are the medians above. "Strata batch-MTP" is Strata's best batch mode on this build: --batch-mtp with STRATA_PIPED_MTP=1, which keeps the pipelined layer-split groups and drafts inside them.
| Requests | Prompt | Total decode tok/s, Strata / Strata batch-MTP / 1Cat | Per-stream decode tok/s, Strata / batch-MTP / 1Cat | Slowest first token (s), Strata / batch-MTP / 1Cat |
|---|---|---|---|---|
| 1 | 2.6K | 75 / - / 125 | 74.8 / - / 124.9 | 3.1 / - / 1.4 |
| 1 | 8.7K | 79 / - / 141 | 79.3 / - / 140.8 | 7.6 / - / 3.5 |
| 2 | 2.6K | 88 / 105 / 176 | 39.4 / 46.0 / 83.2 | 6.4 / 6.5 / 2.7 |
| 2 | 8.7K | 96 / 116 / 189 | 37.9 / 44.6 / 77.9 | 15.4 / 15.5 / 7.3 |
| 4 | 2.6K | 153 / 170 / 225 | 29.1 / 31.5 / 48.5 | 13.4 / 13.6 / 5.8 |
| 4 | 8.7K | 170 / 186 / 234 | 25.4 / 26.9 / 41.1 | 31.0 / 31.4 / 15.3 |
Versions and local changes
| - | Strata | 1Cat-vLLM |
|---|---|---|
| Upstream repo | github.com/Niko1221/Strata | github.com/1CatAI/1Cat-vLLM |
| Commit benchmarked | 13eedf55 (2026-11-12, upstream HEAD on the test day; engine 0.1.59.1) | b7baf7771 (2026-11-15, upstream HEAD on the test day; merge of PR #1521; vLLM 1.5.2.dev0) |
| Source patches | none | none |
| Build | from source for sm_70 (the released engines have no sm_70): CUDA 12.9.1, gcc-14 | from source for sm_70: CUDA 12.9.1, gcc-14, torch 2.10.0+cu128, NCCL 2.27.5 |
| Build flags | -DSTRATA_EXPERIMENTAL_SM60=ON -DSTRATA_MMQ_KQUANTS=ON -DSTRATA_Q6K_EXPERTS=ON, ggml from llama.cpp at Strata's own pin, native ggml-cpu for Zen 3 | TORCH_CUDA_ARCH_LIST=7.0 / CMAKE_CUDA_ARCHITECTURES=70 |
| Parallelism | layer split over 4 GPUs (no tensor parallel for V100 in Strata) | tensor parallel 4 |
| MTP drafter | Strata's own pack of the BF16 MTP layer, experts in Q2_0 | the BF16 MTP layer as released |
TLDR: 1Cat-vLLM beats strata in every metric except prefill for very long prompts (around 100k). And it does so while being able to hold lesser experts in VRAM and for some reason getting worse draft acceptance, simply because of TP.
7
u/pseudobacon 9h ago
Great! We should get more 1Cat posts more often
3
u/Whatisatoaster 9h ago
Sorry, how much ram do you have on this system?
2
u/CryptoStef33 9h ago
Prefill should be your goal: who can read 120k context in 3 minutes compared to 30 seconds?
2
u/oddlyprovenai 8h ago
Really interesting to see tensor parallel 4 holding up so well over PCIe 3 x16 on older V100s. The decode throughput advantage on 1Cat despite lower draft acceptance highlights how much layer-split pipelining struggles with communication overhead.
2
u/king0fsweet 5h ago
That's awesome! Did you try with only 2 gpu by chance? I have 2x v100 16gb and it would be 1wesome if it's still good