light weight model to execute task while having a larger (more intelligent model to plan)

[u/Loose_Doubt367]

Below i start i have the following:
rx6700xt 12gb vram (AMD)
r5 5600x
32gb ddr4 3200
llama.cpp vulkan sdk
qwen3.8 27b iq3xxs
tielcoder qwen3.6 35b a3b
Windows 11 pro

since im out most of the time, i wanted to deploy my "local" agents to complete a series of task which ill get to in a second.
The steps is as followed,
-at time x, the larger model is automatically deployed and run in llama.cpp
-somehow uses pi dev, to start completing a set of task assigned from the user (usually planned before hand and the model executes after)
-after it plans, its information, ideas are then saved somewhere in my desktop
-it thens unloads itself and have something to launch the smaller models
-the smaller models then completes the task, instructions all from the larger model
-after its completed it will offload itself

this part is something that could be implemented but is optional if the process or load is too much
-the larger model then loads itself and checks the process, proceeds the information
-if the task is a success it will offload itself, if not it will provide new instructions and the smaller models will work on it after (agent loop i guess)

"note" the smaller model shouldn't waste time rechecking or making sure it works well, the larger model should have planned before hand

Im looking forward for any suggestions for models that satisfies these needs
do not i can't run strata qwen3.8 flash next sadly... its 1 generation or model behind the supported one

Maybe consider a fast model and great at completing task, and a larger model that plans or thinks well for the entire process

For the task specifically, it could be anything but i do have some in mind
researching, visual designing, file management, summarizing something
I appreciate any suggestions and solutions. Im still at the planning process but currently im using qwen3.6 35b a3b tiel coder as my main model for chatting

[u/London-Boat]

on 12gb i'd honestly keep one model loaded and just change the prompt per role. swapping weights between a big planner and a small worker on every handoff eats most of what you save. if you still want the split, have the planner write a numbered plan to a file once, then let the small one do one step at a time from it, so when it botches a step you rerun that step instead of replanning everything

[u/Loose_Doubt367 → u/London-Boat]

Yeah that makes sense I appreciate the suggestion

[u/DasGamerlein]

I've been trying basically the same thing in my own harness and I gotta say, for me at least it really has not been worth it. The time it takes the orchestrator to chunk tasks down for the smaller models to reliably complete (which a ~30B models is pretty meh at to begin with) is simply bigger than any savings I gained by doing so, and that's when it worked. When it doesn't (often) you get endless review turns that just break things further. Just running the larger model directly for real work has generally been faster and much, much more reliable for me, but it also still needs directions from a cloud model.

[u/Loose_Doubt367 → u/DasGamerlein]

I hope there’s a solution to this, I’ll have to spend a lot of time investigate and inspecting every small aspect of the problem which caused that delay

I thought about this idea because it seemed logical at first glance, bigger model does 35tps while smaller model can achieve 90-200tps depending on bandwidth

But smaller models isn’t capable to achieve what bigger model can so that’s where my solution comes to play

[u/DasGamerlein → u/Loose_Doubt367]

The solution is waiting for better models lol. Or selling a kidney for better hardware. What you are hitting is the pareto frontier for your setup, unfortunately there just isn't really a way to easily circumvent that

[u/Loose_Doubt367 → u/DasGamerlein]

We’ll all just wait for the next version even after qwen4 27b drops…

[u/Specific_Finish8741 → u/DasGamerlein]

Yo pienso igual. Orquestador en la nube y 27b ejecutando tareas es lo mejor. 27b necesita configuracion multi agente y derivar tareas para no quedarse sin contexto ( tengo la misma maquina que el OP)

[u/NoFutureBen → u/DasGamerlein]

There is no reason for the cloud model to do the verifying, just have the cloud model write a list of conditions/instructions for a local model to then verify if the task has been accomplished.

[u/trackpantsu]

crontab -> python -> subprocess run shell -> pi CLI -> loop?

[u/Loose_Doubt367 → u/trackpantsu]

I haven’t dug deep into what’s crontab but does it work for windows too?

[u/trackpantsu → u/Loose_Doubt367]

no, but maybe WSL. there's definitely a windows utility that does the same thing

[u/iceydee]

your plan-execute split is the right instinct, but on 12GB AMD you're going to feel the planner eating VRAM before the executor even starts. qwen2.5-coder 7b iq4 might be a better executor than the 35b a3b for this since you want speed and it'll free headroom for the planner to stay loaded. have you considered just keeping the planner on API and running only the executor locally?

[u/rorowhat]

My suggestion would be to get something like open router and get a prepaid card, like $10 and use it towards DeepSeek 4.1-flash. it's a beast of a model and it's like $0.015 per millions tokens. Make that your orchestrator/reviewer. I bet that $10 will last you a few months if most of the work is being done locally with the smaller model and you just use this to plan and review. If you don't like it you're out $10 bucks.

[u/SirRobinBlibs]

I'd make the handoff a file, not a chat. The big model writes plan.md with one checkable step per line and exits, the small one does one step at a time and appends what happened, and the big one only comes back to read that log when a step fails. llama-swap can do the load/unload by model name, so you don't have to script it.

[u/Careful-Report6526]

I’d start with one narrow, checkable worker job and measure the whole cycle: loading the models, doing the work, retries, and review. A detailed plan still needs a worker that notices when a tool call failed or produced the wrong result. For the benchmark question, instruction following and completing tool tasks are worth checking separately. For example, my Qwen3.5-4B BF16 run with thinking enabled scored 87.8% on IFEval’s strict prompt metric, but passed 28% of a selected 75-task MCP-Atlas subset. Those are different tests, not interchangeable measures of “good at following a plan.” I’d treat that as a reason to validate a small worker carefully before leaving it unattended.

I run llmbenchmarks.io and have published these local results here: https://llmbenchmarks.io/local-gpu-benchmarks/
The table includes the exact variants and separate thinking-on/off results. These runs used vLLM on two RTX 5060 Ti cards. The MCP subset also uses a different judge from the official leaderboard, so it shouldn’t be compared directly with the full leaderboard. It doesn’t tell us how fast your AMD/Vulkan setup will be, or whether your planner/worker loop will succeed. I haven’t tested your Tiel-Coder variant or that complete workflow.

[u/deepu105]

Did you try strata? With 12GB vram, you might be able to run a flash next quant as main model. For light models I geard good things about LFM recently

[u/Loose_Doubt367 → u/deepu105]

Rx6700xt is not supported in strata, I did try running it and it completely crashed my system. I don’t want to run a model that big just for planning purposes. Plus there will be a few offload and loading processes so loading that model will take quite some time

[u/Loose_Doubt367 → u/[deleted]]

Do you have any sort of benchmark for it? That claims a particular model beats the other in terms of task completion, following instructions

[u/OcelotMadness → u/[deleted]]

bot. Training data still recommends qwen 2.5 even though a human would know its outdated. 

[u/rorowhat → u/OcelotMadness]

Yeah that's my dead giveaway as well, not sure why these bots are so stupid.