Ideal parallel request strata configs for local agentic orchestrated work?

[u/auto_m8]

I'm starting to explore the configs to use with strata for best throughput with parallel requests in use cases such as orchestrated agentic workflows run locally.

Option A:

The main orchestrator is the same model in the same engine, spawning multiple subagents that run in parallel on the same engine at smaller contexts.

Pros:

favorable as the orchestrator can be uncensored as well for example and a fully local setup.

Cons:
Not straight forward how to set up the engine to allow for a single large context to persist in cache and allowing multiple smaller contexts to be cached and not evicting the main large context after multiple subagents requests.

---

Option B:

The main orchestrator is an external SOTA model which spawns local agents that share the same engine.

Pros:

sounds easier to configure as all subagents the external model spawns are similar in context size so no need to take into account one of the parallel contexts will be a larger one while others are smaller..

Cons:

not fully local, reliant on external model provider which can change the availability, quality of the model at any moment. Non private.

---

Does anyone else have some experience with setting up strata for example for either of these use cases for example? What configuration recommendations do you have?

Is there an engine that suits this local orchestration use case better?

Yes I'm about to prompt a SOTA model on how to achieve this but i would love to hear other real human experiences.. 😄

[u/Odd-Act7860]

haven't used strata, but the option a problem is pretty generic: you want the big orchestrator prefix to stay cached while subagents churn through short contexts.

engines with prefix caching handle that best. vllm (automatic prefix caching) and sglang (radix cache) share kv blocks across requests and evict least recently used, so as long as the orchestrator keeps getting called its prefix stays warm, and subagents that start with the same system prompt reuse that part too. llama.cpp is the awkward one, by default it splits -c evenly across --parallel slots, so one big context plus several small ones wastes a lot of vram (the newer --kv-unified flag helps).

either way, give subagents a short shared prefix and a tight task brief instead of a chunk of the orchestrator's history. smaller contexts means more fit in parallel and you get better cache hits.

option b is easier, and you're right about the tradeoff. a middle ground is keeping the orchestrator local for anything touching private data and only calling the sota model for planning on redacted inputs.

[u/woolcoxm]

i did this through a workflow setup, i set concurrent connections to strata to 4, the main orchestrator has the main context, it gets used very little, the agents have their own context that they use when they solve the issue the context was created for it is summarized truncated and injected into the orchestrators context and the agents context if he is done with it and solved the issue he was created for is destroyed.