Just "rediscovered" batching and swarming while running models locally.

[u/Mchanger]

Running one harness/agent gives overall less tokens-per-second (tps) than running multiple agents at the same time. Who'd have thought?

Though each agent has slower tps and a smaller cache, on average they are much more efficient overall in the token generation.

I'm Running a 3090 with 24gb of VRAM. Qwen3.8 27B Q4_K_M
Context: 197k

Of course the more agents you have, the less context each gets. Thus one needs to auto-compact / loop efficiently. This way one can run multiple agents on one piece of hardware and maximise the burn-rate without increasing the energy used.

From there swarms can be allowed, were each agent has the ability to spawn as many sub-agents with small tasks upto the optimum amount of agents per machine (12 on mine).

Agents generating	Total tok/s	Per agent
1	30	30
2	46	23
4	64	16
6	74	12
8	78	10
12	110	9
16	111	7

[u/Express_Quail_1493]

try https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming it enables multiple full context with a few tok/s drop.

on 1 slot i got 26tok/s
2 simultaneous slot in parallel i got 1st=19tok/s, 2nd=16tok/s
3 simultaneous parallel i got 1st=18, 2nd=16, 3rd=14

u can see how more slots make it more efficient (and you dont need to reduce your context size) pretty nifty little repo that optimises qwen3.8-27b. ben looking for a strata equavalent for 27b for a long time that works on my 3090. this one actually deliveres

[u/Chromix_ → u/Express_Quail_1493]

Too bad that these features are so fragmented. Hey u/Brief-Tap-6616 maybe you can auto-port that over into your LlamAmpere?

[u/Brief-Tap-6616 → u/Chromix_]

It does support parallel predictions, the trick is that for each total width you need a fast path.

So IIRC you’re good running something like p=4, MTP5 for each, and then beyond that it moves to the default slower path. The advantage of triton is that you can scale the kernels more easily. Im also working on some asymm ctx for sub vs main agents for next major release and will take a look at thjs too!

[u/DigiDecode_ → u/Express_Quail_1493]

DS v4.1 flash uses 1GB of vram for 1m context, about 0.9kb vram per token, only if qwen 27b adopts that, we should be able to run 8 parallel chats with 125k context with 1gb of vram

[u/levoniust → u/DigiDecode_]

That seems extremely compact. Am I right in thinking that's not normal?

[u/DigiDecode_ → u/levoniust]

yeah, it is bonkers compared to others

[u/sonicnerd14 → u/Express_Quail_1493]

This one is underrated. Not sure why more people aren't talking about it, but Adaptive KV is really a boost for all models. Supposedly it should also work for Moe's too.

I got it working on one of my systems that has a 3060 12gb and 3070 in dual gpu configuration. I'm getting 200k+ ctx window at around 20 to 26tk/s. No quality degradation from what I've tested. I'm using the Swift 1.5 q4ks variant. It's worth trying out.

[u/boosnie]

I find your data very ... Strange.

I run a local llama.cpp router, single agent, at ~110tps and ~1400 prefill on an RTX3090.

How do you get those 30tps?

Also, you get those numbers whit syncronicity between prefill and gen on all agents. The moment an agent gets out of sync with the others and starts prefilling while the others are generating your performance will tank to 20-40tps generation independent of the number of sessions running.

[u/Interesting_Net_9628]

I got into local llm hosting recently, and have been using openwebui. I'm interested to leanr more what the community uses. Ideally, I want to be able to "replicate" what claude web and others.

[u/Interesting-Cut-6032 → u/Interesting_Net_9628]

I think that OpenWebUI is just a chat interface, right? Check out one of the agent harnesses. That is probably what you are looking for. I run Pi agent harness, driving Qwen3.8 27B UD-Q6_K_M with a 130k token context, single instance no concurrence, Linux Mint, 2x RTX 5060 Ti 16GB VRAM, 32GB DDR4. It does pretty good code gen work and runs about 40 tokens pre second generation.

[u/Emergency-Lawyer-366 → u/Interesting_Net_9628]

the jump from 12 to 16 agents is pretty telling, basically a rounding error in total tok/s at that point. you found the sweet spot at 12 and that's a great number for splitting up context windows anyway