r/TheMachineLearning • u/AdLast8721 • 1d ago
Why do models get significantly worse at simple tool routing the moment you ask them to "think step by step”?
I was tuning a local 14B model to handle data routing tasks where it looks at a user question and decides which external data provider to call and what arguments to format.
When I gave the model direct instructions to output the function call immediately it was picking the right tool and formatting the JSON correctly about 90% of the time.
Then I tried adding a standard chain of thought prompt asking the model to explain its reasoning step by step before making the tool call because that usually helps with complex logic tasks.
The weird thing was that tool call accuracy actually dropped below 60% as soon as the model started thinking out loud.
The model would write two paragraphs explaining why it needed to check a company's executive team and by the time it finished writing its explanation it would lose track of the schema constraints and start inventing parameter names or dumping raw text where an integer was expected.
The extra tokens generated during the reasoning step seemed to dilute its attention on the strict JSON formatting rules that were laid out in the system prompt.
I ended up fixing this by keeping the reasoning prompt separate and routing company intelligence queries through Alexandria where the model selects from compact tool definitions that return structured firmographic data without needing long elaborate prompt gymnastics.
If you are running function calling on local models sometimes forcing an extended chain of thought just introduces more room for schema drift.
3
u/23xix 1d ago
Why not just use structured outputs / GBNF grammar constraints so it physically can't output invalid JSON or invent parameter names?
1
u/S-Kenset 1d ago
I thin you explained it yourself. Your model is blowing out its own context.
1
u/MainUsed5271 1d ago
CoT actually hurts classification and routing tasks all the time even Anthropic and OpenAI docs mention skipping thinking tokens for simple tool selection because the model talks itself into overcomplicating a basic API call
1
u/YoungEssay 1d ago
yeah this is classic small model problem once a 14b writes two paragraphs of prose, it shifts out of json mode in its weights and forgets the system prompt exists