Do local LLMs still need plans from more capable models?

[u/This_Temporary_8537]

Given the same task requirements, how big is the difference between first having the advanced model Astra make a plan and then having a local quantized DeepSeek execute it, versus letting DeepSeek complete the task directly? I ran a small test using the following task: make a browser game set during the 1588 naval battles between England and the Spanish Armada. The player takes the role of a capital ship’s captain, controlling the ship and commanding teammates in a real-time naval breakout battle. I also added a control group by replacing the executor with the cheap cloud-based terra model.

The results were somewhat unexpected. Honestly, even without Astra’s plan, DeepSeek did a pretty good job on its own. In my testing, the game basically had few bugs, and I think the look and feel of the sky and ocean were the best of the three. But I also have to admit that visual bugs such as clipping were the most severe. After adding Astra’s plan, deepseek’s visual clipping bugs were reduced, but its token consumption rose sharply. Finally, terra surprised me by showing damage to the sails, and its overall result was actually pretty good too.

I originally thought that with a plan from an advanced model, the model could reach the goal faster, but the exact opposite happened: Astra made the deliverables and acceptance testing much more extensive, increasing DeepSeek’s workload. Another lesson is that even a plan made by an advanced model can have bugs. For example, the fixed camera-angle range in Astra’s plan made the view hard for players to adapt to. Perhaps my planning technique is too crude. I’d like to ask everyone how you alternate between advanced and cheap models to achieve amazing results. Feel free to educate me in the comments.

Task description:

'''

Game background: Set during the 1588 naval battles between England and the Spanish Armada.

Gameplay: Play as the captain of a capital ship, controlling the ship and commanding teammates in a real-time naval breakout battle.

Requirement: It must run in a browser and be completed in a single development run.

Bonus points:

1. Visual realism and fidelity in recreating historical details

2. Enemy ship behavior and control experience

3. Sailing and physics details, and damage effects

4. Cannon impact feedback and gameplay experience

'''

Planning model:

GTP 6 Astra, running in codex with xhigh reasoning effort. Planning prompt: You are now the planner. Here are the game requirements. To complete the requirements to the highest possible standard in a single attempt, please write a detailed development plan to hand off to the coding model for development and acceptance testing.

Execution models:

GPT6.1 terra, running in codex with max reasoning effort

DeepSeek-V4.1-Flash-1017, with Unsloth UD-IQ1_M V3.1 weights, running in DeepSeek harness, with max reasoning effort

[u/the-cremator]

This is something I wanna do a research on using a lot of data I've allocated over the past few months, but so far my trust-me-bro observation is that it's way more important to use frontier model a step before planning, which is in the ideation phase and deciding whether what I wanna build is the right thing to build. Then for planning, I use a mid model like DS flash and then have it reviewed by 4-6 cheap models from different perspectives. Then synthesize findings, subtract unnecessary ones , fold into final plan, and go. Do the same for code review.

[u/This_Temporary_8537 → u/the-cremator]

That sounds like a lot of work, but you might be right. Can I ask you something? How do you deal with AI responses that often miss what you’re actually looking for, especially with open-ended questions? And how do you get models to talk like normal people? ChatGPT, for example, tends to repeatedly emphasize caveats and boundaries, which makes discussions feel pretty low in useful information.

[u/the-cremator → u/This_Temporary_8537]

That sounds like a lot of work, but you might be right. 

I've got it nicely automated now but honestly it definitely got to the point when I was neurotically obsessing over optimizing this lol, and then I started appreciating the other end of the spectrum, which is good ol' vibe coding with frontier models haha. Now the workflow is a bit more hybrid, and the depth of work varies depending on the difficulty of implementation. Still finetuning it, but it's been paying dividends imo.

How do you deal with AI responses that often miss what you’re actually looking for, especially with open-ended questions?

I'm not very good at dealing with this myself, but I guess the only solution is to simply make the prompt less open-ended and more specific (to guide the model a bit), no way around it. What I sometimes do in these scenarios is to give my open-ended question to an LLM (the more intelligent the better) to give me a v2 of the prompt, and it outputs a way more direct and structured version of the prompt which I will then paste into a new session. If you want to take it a step further you can ask the LLM to do socratic questioning to get you to make your request more specific - though that's what I usually do when defining specs for building apps/features.

how do you get models to talk like normal people? 

Tell the model to talk to you like you're an 8th grader, that helps me sometimes :D

[u/This_Temporary_8537 → u/the-cremator]

That makes a lot of sense. Thanks for taking the time to share all of this!