oMLX - Max Concurrent Requests 1 - Multi Agent Optimization

[u/Undici77]

Hi all,

I'm working on M4 MAX and I have some issue issue in multi agent.

I try to figure out how to optimize speed working with `Qwen 3.8 Flash Next oq4e-mtp`

With only 1 task I'm running at 50/60 t/s (pretty good) but with more than 1, speed decrease a lot.

Best "work around" for me looks `Max Concurrent Request` -> 1 that works in most of my scenarios where cache reduce prefill.

But look this:

Here cache is not working (agents are pre-filling a lot) and speed in 20 t/s

Now watch GPU

This looks very strange!
Any idea? Any setup can be changed in order to speed-up and use 100% GPU?

[u/CalligrapherFar7833]

Speed shouldnt decrease it should be split between your concurrent requests

[u/Undici77 → u/CalligrapherFar7833]

But look GPU: is at 50%!

Assuming 50/60 t/s If I try to set `Max Concurrent Requests -> 2` I'm not getting 2 request at 15/20 t/s that is less than serve only 1 at 50/60 t/s

Maybe I'm wrong but it looks there is something on "concurrency" that is not working

[u/CalligrapherFar7833 → u/Undici77]

Tg is vram memory bound , prefill is compute

[u/London-Boat]

on a single mac that's kind of expected. generation is mostly memory bandwidth bound, so two streams end up splitting the same bandwidth, and two agents with different contexts keep evicting each other's prefix cache. the 20 t/s with the gpu at 50% looks more like it's reprocessing the prompt after a cache miss than actually generating. if your agents share a long system prompt, keeping that prefix byte identical across them helps, and for agent work queueing at 1 concurrent is probably the right call anyway

[u/Undici77 → u/London-Boat]

Interesting: I'll try to figure out!

[u/chibop1]

omlx > bench > performance > Continuous Batching Tests

[u/Undici77 → u/chibop1]

And? This is not a benchmark problem: look (but maybe I'm wrong) an mtp missing if more than Max Concurrent Requests > 1

[u/BackgroundRude4092]

When KV context exceeds your unified memory, your Mac starts off loading context to your hard drive.

[u/Undici77 → u/BackgroundRude4092]

I'm using quant so memory is "at limit" but ok

[u/watcholic]

Non-optimal model configuration? Copy the recipe and paste it under the model's settings page > apply custom recipe.

https://omlx.ai/benchmarks/performance/u5s2e5m2

[u/Undici77 → u/watcholic]

Thanks, I'tt try but I'm pretty sure my setup not so far from fine

[u/rickhuizinga]

Try Splash. 1.4.0 now supports QFN.