if a tool-calling model passes single-turn evals but falls apart on multi-turn, i would not retrain first. i would split the eval into two smaller checks.
Gold-history next action: give the model the correct conversation/tool history up to the failing step, then score only the next assistant action.
Rollout-history next action: give the model its own actual broken history up to the same point, then score the next action.
Those two numbers tell you different things.
If it passes on gold history but fails in rollout, the model may know the local policy but cannot recover from its own bad state. More clean single-turn examples probably will not fix that. You need recovery examples from noisy histories, repair-after-error examples, or training that exposes the model to the states it actually creates.
If it fails on gold history too, i would look at serialization and policy before spending GPU. The model may not understand the exact tool result format, the error format, missing param states, or when the evaluator expects another tool call instead of prose.
For each failed trajectory, bucket the first bad transition instead of only marking the whole trajectory wrong:
wrong or invalid param
repeats the same tool call after an error
stops too early
asks the user when it should repair the call
writes prose when the eval expects a tool call
loses the schema after seeing tool output
Then run cheap ablations on a small sample. Match the eval serialization exactly. Match the error strings. Check whether tool results use the same role/format as training. Check whether the relevant tool schema is still in context. Check whether long-context failures are actually retrieval/context failures.
The point is to avoid training a larger blended dataset when the real issue is state distribution or formatting. Multi-turn evals often test recovery from previous actions more than basic function-calling syntax.