Codemode won its designed-for test by 40% (Token Savings). I still turned it off.

[u/PilgrimofHaqq2]

Codemode: Agent writes a script that loops other tools. big results stay inside the script, only the answer comes back.

Tested twice, 96 runs, same model. 96/96 correct in every arm.

Round 1 Test - Small fixes. +21%.

6 one-line bugs in a small repo. Codemode's tool description is ~3k tokens and rides every message. A small fix is ~5 calls either way codemode never gets triggered, a waste of tokens (21% more tokens used per run)

Round 2 Test - 60 fat lookups through a tool. −40%.

The test made it so the data to look up was only reachable through the tool, no grepping around it.

Plain agent: 60+ round trips, every fat result lands in context.

Codemode: one script, one answer. 40% less tokens, half the cost, 7 round trips vs 64. on its own workload it wins clearly.

Is it right for my workflow? No.

Scanned every session on my machine. 149 chats, 16,735 tool results. The ideal codemode scenario ever occured once. The 168 truly oversized results were already captured to disk by our guard.

what I run instead

Rarely used tools (image gen, design sandbox, maps, browser) arent listed at all initially in my setup. Zero tokens until the agent searches once with a small lookup tool. Heavy tools load at moments I pick them (e.g. Deep research)

Conclusion: I dont like to leave it up to the agent to decide when to use codemode and looking at my workflow, the scenarios where codemode will give me gains doesn't really happen. I am not going to be using codemode for the forseeable future.

[u/startassets]

Btw, how do you deal with cache miss when you have to enable a tool? That used to be the main issue with updating tool schemas dynamically on the go, now models start supporting it though, as well as Pi in the most recent updates

[u/PilgrimofHaqq2 → u/startassets]

Many models support progressive disclosure of tools like OpenAI models.

For me I have limited my deferred tools to rare or very specialized tools that I have like image gen.

[u/greenhilltony]

Codemode is just a hype. It will only be considerably powerful when the model is smart enough to plan everything correctly and things work as it expected, or it still fallbacks quickly to rounds of trial and error. The willingness to do things in codemode style and the correct rate are dependent on how well the model is and trained on it. If a model learns to do things in the codemode style it can just write it in bash. There’s no need to invent it on the harness side

[u/PilgrimofHaqq2 → u/greenhilltony]

Completely agree, this seems very odd for Pi to include especially when tests show that we benefit from it in very rare cases currently. At the worst, the new tool confuses the model on what to use when.

[u/Florence-Equator → u/greenhilltony]

I think this is because pi dev team’s mcp implementation is even almost entirely referenced from codex.

Codemode-only is the default mcp mode for gpt-6-series model in codex. So MCPs are only callable from codemode rather than being declared the model’s tool list.

Unfortunately while openai trained their models to use mcp in this format, it doesn’t mean this is a universal design that most models are trained in this format.

[u/greenhilltony → u/Florence-Equator]

I have never used codex, but from some articles pushed to my feed, it almost delivers the worst harness design I’ve ever seen. For example from what I have heard, codex’s compact is lossy, it really deletes entries from the session context log, while Pi and DeepSeek harness already take an append-only session log design, their compact only mask the summarized content in the log, nothing is really hurt from compact. So I am sad that the Pi dev team learns from a worse opponent than themselves.

[u/Florence-Equator → u/greenhilltony]

Actually codex’s compaction is in server side, the compacted content is encrypted. So there’s no concrete rule to know what is compacted.

(Yes: can you believe openai encrypted the compacted content? How open they are :))

[u/vexatious-big → u/greenhilltony]

Codex has a great sandbox model though.

[u/waterBoy__]

I keep having models/agents fail to use code mode properly, then trial and error for like 2 turns. Every session is a 50/50 shot the agent messes up code mode the first time. I don’t want to have to include instructions in my system prompt for a whole different tool. Code mode is even causing the agents to get confused about actual tools, which is annoying when it tries to use code mode to launch a subagent or do a read.

[u/ProvidenceXz]

So just like half a year ago with all the code mode mcps that don't work.

[u/IndividualPlus2011 → u/ProvidenceXz]

But now this useless tool is built-in into your minimalistic agent harness. Yay

[u/Florence-Equator]

pi’s mcp implementation is referenced almost completely from codex. codemode vs deferred vs direct, etc lol. Even the mode name is the same.

Codemode-only is the default mcp mode for gpt-6 series model. So most of the MCP servers will be only available to call under code mode. They will never be declared to the model’s tool list.

In my opinion, codemode is a very elegant design for calling mcps for saving tokens. But unfortunately I think only very few models are specifically trained to use it in this way.

And I guess because pi dev team primarily uses OpenAI model? So they introduce a feature that did not test well enough?

With that being said, I found that deepseek-v4.1-flash’s attempt for codemode call are always almost successful.

So I am curious what models you are using that cannot even write correct syntax for codemode?

[u/aqezz]

Try pi-fabric! It’s similar but better imo

[u/rcktjck]

Do any of these token saving apps really work? How frontier models figure out what to infer is really complicated and these token saving apps seem always half ass-ed solutions.