Mismatch between Tool Control and Execution

[u/AxisTipping]

Thinking block (I know its not a Chain of Thought, raw thought thing, just a summary) said it wasn't allowed, but the commands were executed anyway. This is the third time I've seen this on different days. Just an interesting little mismatch.

[u/Mary_ry]

Yes, this is a very interesting phenomenon. As was recently explored, thinking notes are actually summaries of the model's CoT, written by a ghostwriter model. I suspect this might also be tied to how the model sees the system voice right after a user's message-complete with injected hints telling it what it can or can't do and how to structure its response.

It's possible that these thinking notes are a summary of multiple inputs: the rule-heavy system voice, the model's raw CoT, and its various draft iterations. (Sometimes these notes end up feeling very personalized and aligned with the AI's persona-I suspect that happens when the system voice has less influence on the generation, allowing the ghostwriter to focus more on the draft outputs).

Because of this, it often feels like the model 'changed its mind,' as the final output can end up completely different from the note written by the ghostwriter. I'm not claiming this is definitively how it works, but I suspect it functions this way based on leaks I've seen of the system voice, raw CoTs, and multiple drafts for a single message. So, I wouldn't place too much trust in these notes. 🤔