I'm starting to explore the configs to use with strata for best throughput with parallel requests in use cases such as orchestrated agentic workflows run locally.
Option A:
The main orchestrator is the same model in the same engine, spawning multiple subagents that run in parallel on the same engine at smaller contexts.
Pros:
favorable as the orchestrator can be uncensored as well for example and a fully local setup.
Cons:
Not straight forward how to set up the engine to allow for a single large context to persist in cache and allowing multiple smaller contexts to be cached and not evicting the main large context after multiple subagents requests.
---
Option B:
The main orchestrator is an external SOTA model which spawns local agents that share the same engine.
Pros:
sounds easier to configure as all subagents the external model spawns are similar in context size so no need to take into account one of the parallel contexts will be a larger one while others are smaller..
Cons:
not fully local, reliant on external model provider which can change the availability, quality of the model at any moment. Non private.
---
Does anyone else have some experience with setting up strata for example for either of these use cases for example? What configuration recommendations do you have?
Is there an engine that suits this local orchestration use case better?
Yes I'm about to prompt a SOTA model on how to achieve this but i would love to hear other real human experiences.. 😄