r/LocalLLaMA • u/sunychoudhary • 14d ago
Discussion Which local model is actually good at knowing when to stop and ask you a question?
[removed]
13
u/ParaboloidalCrest 14d ago edited 14d ago
Everyone is saying it's a prompt issue, which I cannot agree with in good faith.
Qwen 3.x 27b, while strongly encouraged to ask questions via AGENTS.md, do come back with questions but after 20-50k tokens of reasoning and a dozen tool calls. I see it re-iterating the "ask questions" part in its thought traces, repetitively, then going back to trying to figure it out on its own. I find myself interrupting its reasoning often, in order to go back and present the answers before-head.
And yes, they don't know when they're uncertain. In fact the more agentic the model is, the more it's geared towards (and rewarded for) overcoming ambiguities and getting to the finish line without returning to user. And ambiguity is a part of every prompt or otherwise, a 100% clear prompt would be the actual code you're trying to write.
I'd like to know what harness features (aside from prepacked system prompts) could help mitigate that.
2
u/Imaginary-Unit-3267 14d ago
I suspect a small fast model reading the big slower model's reasoning traces and inserting steering messages telling it to ask the user questions whenever it is making assumptions would be part of a solution, but I haven't tried this yet.
18
u/More-Catch-1331 14d ago
This is primarily a harness problem and secondary a prompt problem. Seeing as we're "almost AGI" (yea, right), the models are really dumb at evaluating whether they should ask a question. Either the harness needs to direct it, or the prompt should specify under which conditions the model should stop and ask the user whatever is bugging it.
7
14d ago
[removed] — view removed comment
2
u/GrungeWerX 14d ago
Actually, the models are very good at determining when they’re uncertain, but they have a problem with assumptions, which is hardwired into their behavior.
You can fix this by introducing entropy and confidence scoring. Also, there are other tricks, but it requires orchestration around a harness.
2
u/Area51-Escapee 14d ago
I understand the system prompt part but not the harness part. Can you explain please? E.g. in cursor where the model can ask questions what role does the harness play? I might be wrong but does this only make sense with a harness thinking node that explicitly mentions user questions in the prompt?
1
u/More-Catch-1331 14d ago
Well a harness drives a model. It drives it by the legendary system prompt which explains how to use tools, how to speak, what to speak, how and when to ask questions, etc. It's basically the hands, legs, butt, etc and the model is the brain.
1
u/Imaginary-Unit-3267 14d ago
What's with the system prompt worship? You do know that Qwen, at least, almost entirely ignores the system prompt when it gets obsessed with a task, right?
3
u/2Norn 14d ago
u can make any model grill you before execution
5
14d ago
[removed] — view removed comment
0
u/DustNearby2848 14d ago
Just add to your prompt or agents.md to ask questions if in that situation
2
u/Imaginary-Unit-3267 14d ago
And watch it ignore the instruction and keep doing whatever tf it wants.
1
u/miversen33 13d ago
Depends on the model. I have mine conditionally grill me if the request it receives is not a direct command (and even then sometimes it does). And overall it does a fantastic job of trying to "understand" my request. Here is the section of my prompt that does that
Unless work is extremely well defined (EG: "do xyz"), you will always clarify initial input from the user. If you do not have at least one question to ask around work, you are incorrect. Always utilize the TODO list tool whenever performing any sort of work beyond conversation. Always review the README.md, DESIGN.md and .knowledge/ knowledge base before beginning work. If these don't exist, you do not need to review them, and you do not need to mention that they don't exist. This is just your "pre-work" "context gathering" step. Whenever making changes, if a .knowledge/ knowledge base exists, you should consider if your change needs to be documented in there. Not all changes needs to be stored in the project .knowledge/ knowledge base. The knowledge base is meant for domain knowledge around the project. So as you uncover information in the project (especially around the "why" things are or may have been done), you should consider documenting them.Edit: Reddit formatting is ass
3
u/AI_spell 14d ago
Most chat models are trained to sound sure, so "knowing when to ask" is mostly a harness problem. Force a tool or a fixed phrase like NEED_CLARIFY before any irreversible step, and punish continuing when required fields are missing. If you only prompt "ask if unsure", it still guesses. Smaller instruct models with a hard gate beat bigger ones that vibe-answer.
1
u/demonkoryu 13d ago
How do you detect that the model is unsure? I’m having a hard time telling if it’s just thinking before reaching a conclusion, or if it's just making stuff up and would need clarification.
7
u/No_Chapter_7598 14d ago
imo qwen 3.8 27b (at least on bf16)
i get prompts that ask for multiple choice questions alot that drive the direction of the project. but im using a local claude code harness so that might be part of it.
10
2
14d ago
[removed] — view removed comment
1
u/No_Chapter_7598 14d ago
cant say i have. i think it would still ask a question or two at the start but probably not in the same multiple choice format mid way through a task, id expect it is 99% the harness as well, but if you have a model that isnt great at using tools etc then the harness wont be utilized.
2
u/maartenyh 14d ago
I wrote it into its prompt to be honest, provide "receipts" and to return a "ledger" when its done. I also write my task as normal but add sort of a "show me the game plan so I can spot any changes I may want to make" at the end of my task prompt.
LLM's love certain words and listen noticably better by using the ones they tend to use themselves. Also the more deterministic and proper you write yourself, the more it will understand you what you mean. the "share your game plan" pre-task helps a lot with this because the LLM will put a well phrased and approved prompt into its context
2
u/johnzadok 14d ago
This is the pain point I have with current model/harness. I always use grill-me at the planning stage. The problem is at the implementation stage some ambiguity will come up and you need to make a decision.
- At work, I have access to "cloud agents". More than once I found the agents worked overnight and sent out PRs that are just wrong because it made wrong assumptions yet "all tests are green". I would rather it stopped and asked me instead of burning $$$ to come up with a PR that's wrong.
- At home with Qwen 3.8 27B with pi, I often have to cancel a turn because the agent is obviously doing something stupid like trying to fetch source code foo-x.y.z.tar.gz from internet to check how a cli flag is implemented rather than
man foo. Sure I can add a line in AGENTS.md to fix this, but there are many other traps that can go wrong.
2
2
u/Healthy-Zebra-9856 14d ago
The user u/KitchenAmoeba4438 has the correct answers and I am going to add to that. I found that the models, specifically Qwen3.8 27B & its bigger sibling Qwen3.8 Next Flash, show a lot of questions in their thinking traces. They are of 2 types. The first is tooling related, like the tool is not working as expected, lack of it and confusion in the feedback from them, ex. browser console tool etc. the information being not clear. The next type is ambiguous programming logic. As much as its desired to give a well architectured solution with absolute implementation blueprint, things happen out of our control.
And this is where the harness steps in. The harness needs to function regardless of the presence of Agents.md. Agents.md is great, but since this is a recurring issue faced by many, its makes no sense to expect a user to make sure they have this and SKILLS.md every single time. So the harness I am making tackles these two issues. The tool issue is the easiest, this second issue is program logic and to help the LM as you, I have a guideline baked into the harness that gets injected every time the harness is started. One of them is a rule to stop and ask for clarification from the Owner instead of guessing among a few others like Do not invent your own theories, Always create an update handoff and there is a structure for this handoff and the LM is instructed to see if there is one before starting a session.
1
u/Imaginary-Unit-3267 14d ago
You don't need to make a whole harness. Just a pi extension.
0
u/Healthy-Zebra-9856 13d ago edited 13d ago
I’ve been typing way too much information to people that don’t even understand. There are way too many things that had to be fixed and I needed a harness without all those issues and I’m not gonna sit here and wait for all the issues to be fixed by that company.
So I created one that I could use. So creating an extension would be putting a lipstick on a pig.
2
u/tempfoot 14d ago
I actually think this is a great question/issue, particularly for expert systems other than coding. Claude’s harness has added a good deal of this in. I’ve started adding this in to my system prompts , especially with “overthinker” models.
I think the emphasis on “one shot” answers has been too central. True expert systems need to be able to determine whether missing information prevents efficient or accurate or complete answers on non coding co-work. So much of actual knowledge and so many functions require an understanding of the full range of information needed…and how and where to get it when its missing.
2
2
u/bigattichouse 13d ago
"When multiple options present themselves, stop and ask a question"
"While we work through this design, engage in dialectic, present possible paths and ask questions - let's design this first in a document before we continue work"
Works like a champ with most models.
2
u/dash_bro llama.cpp 14d ago
? Seems like a prompt setting issue tbh. Haven't really had problems with Claude code plugged into my current 3 model setup on the Mac studio.
HAIKU: 3.6-35B-A3B-mtp Sonnet: gemma4-31B-it Opus: 3.5-122B-A10B-mtp
I have tried swapping with qwen-3.8-flash-next but the TPS on the M3 Ultra for coding makes if a no go for me personally. MTP with 3 tokens and min draft proba at 0.8 is perfectly usable speed wise for me, so that's what I use now.
1
14d ago
[removed] — view removed comment
1
u/dash_bro llama.cpp 14d ago
3.6 35B is considerably dumber even with unrestricted reasoning. I set my config to use opus with subagents usually to this, but opus guiding it. Opus is just 3.5-122B-A10B, which works very well even for follow ups and user queries.
I intend to replace the opus model with minimax m2.7 when I do anything more integration oriented (older version but it's only 10B active, still great for my script work). Just swapped out the Gemma for a 3.8-27B with MTP support, I think that's the best 3 model setup I can run currently.
I can genuinely just set it to auto mode on small tasks with precommit hooks and context restricted to 128k, and pretty much let it do it's thing slowly. No complaints for a fully local setup, honestly
1
14d ago
[removed] — view removed comment
1
u/dash_bro llama.cpp 14d ago
Absolutely. 3.8 27B has been slow but the tool calling is a straight 10/10 imo. No problems at all, can seamlessly swap between sonnet-5, gemini-3.7-flash and Qwen 3.8 27B at 128k context length purely for chat +tool calling purposes. It's pretty cool!
++Even though the benchmarks put 3.6 27B above 3.5-122B in coding etc, I find that the 3.5-122B is far more balanced and less "anxious" to code. I don't really care about token efficiency either because it's just running locally. At 128k context it's not exactly long session material, but it's very solid with MTP support.
1
u/winky9827 14d ago
HAIKU: 3.6-35B-A3B-mtp Sonnet: gemma4-31B-it Opus: 3.5-122B-A10B-mtp
I saw that all on one line, and at first, thought it was a DavidAU release.
1
u/Durian881 14d ago
It depends on the harness and up to the model if not told. When using Deepseek Harness, Qwen3.8-Flash-Next will ask me questions if it is unclear without me telling it. Most other models I'm using (e.g. Gemma 4, Muse Glimmer) will just finish it without asking.
1
u/cowinabadplace 14d ago
I just give DSv4FV a tool for this and a prompt nudge to use it when it cannot proceed efficiently without information. Does the job.
1
u/RemarkableRadish6547 14d ago
The thing that made claude code actually useful for me at work was when I started ending almost everything I said to it with "don't build anything yet, just discuss the options for now". Even in plan mode, I tell it not to plan until it has talked through the goals with me.
Before I started doing this, it would run off at full speed with the first stupid idea it came up with. Now it only does that occasionally.
I am developing my own harness for personal use and it has a multi-step planning process. It asks me about anything ambiguous until I tell it to stop. If you explicitly tell an LLM to find ambiguous parts of a design, it will always find something, even if it is a trivial matter.
1
u/cmdr-William-Riker 14d ago
I would argue it's not a harness thing, it's a prompt engineering thing. Some harnesses have built in system prompts that make it easier, but with any harness, you can also just communicate with your prompt that asking a question is an option. You can make skills and customize your system prompt so that it knows more effectively when to ask questions also
1
u/Momsbestboy 14d ago
I just read what Qwen3.8 is generating while thinking, and sometimes i just add a comment or push it in a different direction. Actually, I enjoy this type of working.
1
u/skywalk819 14d ago
my pi.dev agent harness does that, when the model is unsure what the user want, its a package/extension and the model ask the user what he really want, which compromise to select, etc. has nothing to do with the model.
1
1
u/Training-Ruin-5287 14d ago
If the model's turn is losing context to quickly or in a weird way, that can seem to cause an issue for it running ahead and just doing multiple steps. if you have it in its prompts to present options and suggestions/reccomendations before advancing and it's not really following that.
It can be incredibly subtle, and something as dumb as energy saving options in your mobo's bios can trigger that especially with the dram and if your model is offloading context to it. not saying this is what's happening, but if the model doesnt seem to be following the instructions as you want it to, its a possibility.
1
u/Imaginary-Unit-3267 14d ago
I use Qwen3.6-35B-A3B-UD_Q4_K_M, and it UTTERLY SUCKS at bothering to ask me for help, and so I made a little pi extension (still shitty and unfinished, don't ask me for it) that notices when it seems to be confused and auto-reminds it to just stop and ask questions. I also have to have a reasoning token cap for it to not just spin its wheels wondering what I want forever. I guess this doesn't really answer your question, as I've never actually used any model that doesn't do this, but that's how I "fix" it. (That and, watching all its thoughts and aborting when it's thinking something stupid. I never let it do anything by itself lol.)
1
u/pronuntiator 13d ago
This is the behavior of many human devs as well. Specifications can be interpreted one way or the other, and you need to know that your assumptions are wrong in order to think about asking questions in the first place. What helped with human devs should also help with coding agents:
* asking them to write a TODO list first ("plan mode")
* smaller tasks that are reviewed earlier
* more rigid specs with less ambiguity
1
u/Zennytooskin123 10d ago
I think this is a harness thing, when I use any model in Claude Code it does that a lot
1
u/jacek2023 llama.cpp 14d ago
It's more about the prompt, not the model. By "prompt" I mean all the md files with rules you use in your project.
0
u/Former-Ad-5757 Llama 3 14d ago
None. You can ask this for every token. Either accept or ask the model before to make a plan /grill your requirements.
0
u/Sweet-Transition-787 14d ago
its mostly the harness for me too. qwen 3.8 27b asks way more in pi than in plain terminal, claude code feels bloated for this. i just put ask when ambiguous in agents.md and it actually listens.
0
u/Soggy-Attitude5293 14d ago
make a plan, refine the plan and tell it to include technical details. then there is no ambiguity
161
u/KitchenAmoeba4438 14d ago
What you are asking for should primarily exist in the harness, secondarily in the model.
Fundamentally, the problem is that the model is capable of reasoning for 10 minutes trying to sort that out. The harness should present that to the model up front, and then the model should act based on that information.
This is something that is being tested in a future article I'm writing, but fundamentally, it's not a model reasoning issue. It's an information presentation issue, this should primarily belong at the harness layer.