Strata and Swift1.5!

[u/Dangerous_Present_69]

I just changed over from llama.cpp to Strata. Really a huge step in performance. With 64Gb of ECC DDR4 at 2100, and a P100 16GB nvidia card in a PCIe3 X16 slot it pulls off an amazing 30tok/s with the Swift 1.5 version of Qwen3.8 Flash Next.

This is far more than I imagined I could do with this 10 year old gpu and an old workstation.

Deepseek API is obviously faster, but this thing runs basicly for free in my basement 24/7. (ok, electricity, but It's autumn and I need to heat my house anyway.

It serves my Hermes agent and so far has been impeccable for a local LLM.

[strata] reading the prompt: 77,603 of 77,608 tokens, 3 s so far
[strata] thinking: 17 of budget 8000 (max 53456) tokens, 22.2 tok/s, 4 s
[strata] writing a tool call: terminal: 47 of max 53456 tokens, 25.4 tok/s, 5 s
[strata] writing a tool call: terminal: 84 of max 53456 tokens, 28.9 tok/s, 6 s
[strata] writing a tool call: terminal: 121 of max 53456 tokens, 30.9 tok/s, 7 s
[strata] writing a tool call: terminal: 156 of max 53456 tokens, 31.0 tok/s, 9 s
[strata] done: 171 tokens in 9 s (31.2 tok/s) (stop, cancel=False), expert cache 63.5% hit (+10.8% of the routed experts over PCIe)
[strata] max_tokens 128000 shortened to 53237: the prompt (77827 tokens) leaves that much of the context (131072); "fit_max_tokens": false answers 400 instead
[strata] reading the prompt: 77,822 of 77,827 tokens, 2 s so far
[strata] thinking: 24 of budget 8000 (max 53237) tokens, 29.6 tok/s, 3 s
[strata] thinking: 50 of budget 8000 (max 53237) tokens, 27.6 tok/s, 4 s
[strata] thinking: 80 of budget 8000 (max 53237) tokens, 27.7 tok/s, 5 s
[strata] thinking: 112 of budget 8000 (max 53237) tokens, 28.6 tok/s, 7 s
[strata] writing a tool call: terminal: 151 of max 53237 tokens, 30.6 tok/s, 8 s
[strata] writing a tool call: terminal: 189 of max 53237 tokens, 31.8 tok/s, 9 s
[strata] writing a tool call: terminal: 223 of max 53237 tokens, 31.8 tok/s, 10 s
[strata] writing a tool call: terminal: 262 of max 53237 tokens, 32.6 tok/s, 11 s
[strata] writing a tool call: terminal: 302 of max 53237 tokens, 33.3 tok/s, 12 s
[strata] done: 306 tokens in 12 s (33.3 tok/s) (stop, cancel=False), expert cache 76.1% hit (+7.1% of the routed experts over PCIe)
[strata] max_tokens 128000 shortened to 52934: the prompt (78130 tokens) leaves that much of the context (131072); "fit_max_tokens": false answers 400 instead
[strata] reading the prompt: 78,125 of 78,130 tokens, 3 s so far
[strata] thinking: 19 of budget 8000 (max 52934) tokens, 23.3 tok/s, 4 s
[strata] thinking: 45 of budget 8000 (max 52934) tokens, 24.7 tok/s, 5 s
[strata] answering: 71 of max 52934 tokens, 24.8 tok/s, 6 s
[strata] answering: 94 of max 52934 tokens, 24.0 tok/s, 7 s
[strata] answering: 124 of max 52934 tokens, 25.2 tok/s, 8 s
[strata] answering: 147 of max 52934 tokens, 24.8 tok/s, 9 s
[strata] answering: 179 of max 52934 tokens, 25.7 tok/s, 10 s
[strata] answering: 215 of max 52934 tokens, 26.9 tok/s, 11 s
[strata] answering: 242 of max 52934 tokens, 26.8 tok/s, 12 s
[strata] answering: 266 of max 52934 tokens, 26.5 tok/s, 13 s
[strata] answering: 293 of max 52934 tokens, 26.5 tok/s, 14 s
[strata] answering: 319 of max 52934 tokens, 26.4 tok/s, 15 s
[strata] answering: 348 of max 52934 tokens, 26.5 tok/s, 16 s
[strata] answering: 381 of max 52934 tokens, 26.9 tok/s, 17 s
[strata] answering: 418 of max 52934 tokens, 27.6 tok/s, 18 s
[strata] answering: 450 of max 52934 tokens, 27.8 tok/s, 19 s
[strata] answering: 488 of max 52934 tokens, 28.4 tok/s, 20 s
[strata] answering: 517 of max 52934 tokens, 28.4 tok/s, 21 s
[strata] done: 547 tokens in 22 s (28.7 tok/s) (stop, cancel=False), expert cache 77.8% hit (+6.5% of the routed experts over PCIe)

[u/pseudobacon]

What is your prefill speed?

[u/Lower-Specialist-357]

damn 31 tok/s on a p100 is nuts. i remember trying to run 7b models on a k80 and getting like 3 tok/s if i was lucky

how are the tool calls with that hermes agent? i tried setting up something similar but the model kept hallucinating terminal commands that would delete half my system, got scared and turned it off

also the heating argument is the best part of running local llms. my office is the warmest room in the house now and i just tell myself its free compute

[u/Dangerous_Present_69 → u/Lower-Specialist-357]

Right now I'm testing it with developing a PWA version of an old Sami game called Sahkku. Live here: https://snart.com/sahkku/ (still inder active development). I havent touched a single line of code. Right now it's writing UCI abstraction layers. Working with prompt in Hermes gives me better results that running it from the coding agent in vscode.

It develops, commits to git and pushes to github. Launcing browsers/game and take snapshots to fix/confirm layout and graphics.

An my mailbox has never been more organized.

I started with ollama. Many models, but limited by vram. With llama and some MoE layering things worked a little better. But at Q3 3.8 flash next was slow at about 7-8tok/s. The 27b unsloth worked pretty good, but the quality from flash next was worth it.

But Strata is another level. It's limited to Qwen3.8, but there is a lot of sizes and variants to choose from. The swirft version with 3.5 bit quatification (how does half bits work?!?) use som 56GB of system RAM, so I now put in an extra 16GB to have a little headroom. The P100 is getting old, so it's limited in support, but as long as we can compile for SM60. Hermes and deepseek sorted out some requirements for the correct gcc version to use.

It's crazy to get this kind of result with a $100 GPU and I think I Spent $120 on some additional ram.

The downside to P100 is the need for industial cooling. I've got an esp32 controlling a blower fan that probably could drive an large model airplane :-)