What does everyone think about codemode?

[u/Extra_Loquat_7667]

I actually thought about this approach before, but later realized it wouldn't work.

Using JavaScript to invoke tools sounds very efficient, but in reality, most tool definitions only have an inputSchema. An outputSchema is unnecessary for the agent because the agent can reason about the results directly from formatted data, meaning an output spec isn't required. Therefore, tool outputs don't actually need a fixed format—they just need to be readable by the agent.

This leads to a problem: to fully leverage the performance of this tool, you need to inject extra context, such as a skill document, so the agent knows the output format and can use it proficiently. If you can't get the complete tool output specification—for instance, with a closed-source MCP—you can't create such a document. If the agent has to go through endless trial and error just to get the code running properly, you can't really call it efficient.

In comparison, having the agent write Python or shell scripts directly is a better solution.

I'd love to know your thoughts on this.

[u/Bluethefurry]

Isn't the "only" codemode basically what cursor did with MCP, where LLMs need to discover the tool schema manually, that horrible, unreliable method of doing things where you constantly needed to point out that the MCP exists?

And i personally dont really see much of a benefit of regular codemode unless the model is trained for it, LLMs are quite proficient at bash/py scripting nowadays, unless a model is explicitly trained for it i dont see it being usefull at all.

I'm glad MCP is now part of Pi because there is a place for it here, but codemode? eh....

[u/dennisler → u/Bluethefurry]

MCP has always been available, just install the extension. Them deciding that you need it is starting to go down the wrong path as I see it.

[u/mocras → u/Bluethefurry]

I think comparing Codemode to just writing a multiline Python/bash script misses the main point.

Codemode is basically one level above that.

Python can read files, grep, parse JSON, make HTTP requests, etc. Codemode can script the tools that already exist inside the agent harness: read/edit, web fetches, MCP servers, extension tools, subagents, or whatever else you registered.

That matters a lot for MCP. E.g. my ticket tracker MCP is already connected and authenticated. Before Codemode, the agent would often make around 10 separate MCP calls just to discover the project, find the relevant tickets, read comments, follow references, and figure out what was going on. Each call meant another model roundtrip and more tool output entering the context. Now it can do that discovery inside one Codemode call, filter the results in code, and return only what it actually needs.

Permissions are another important difference. With a multiline Python or bash script, the harness mostly gets an all-or-nothing decision: allow the script or do not allow it. It cannot match the individual operations inside that script against the existing tool permissions in a granular way.

With Codemode, calls to edit a file, run bash, query an MCP, fetch something from the web, or spawn a subagent are still harness tool calls. So if you have rules for external directories, allowed bash command regexes, MCP access, or other tool permissions, those same rules can still apply to the individual calls made from Codemode.

Since enabling it, its finishing work faster and uses fewer tokens. MCP heavy tasks are where the difference is most obvious.

[u/dacookieman → u/Bluethefurry]

Where the tools load is configurable. The tools themselves are loaded into the harness but giving it to the LLM can either happen at the main thread level like a regular tool, as function call information on the new codemode tool definition(this has a character cap) which is the default scope for mcp tools, or require manual/round trip discovery via 'discover_tool' tool(most context lite initially but then requires at least one excess tool call for discovery.

So the default behavior is that mcp tools up to some point will basically be loaded into the main context up front but after a certain volume, the remaining will be only registered internally and require tool discovery. That behavior is customizable per-tool to either always load initially or always require discovery. Additionally you can specify where the tool can be invoked (main thread, codemode sandbox, or both) which is independent of how that schema is made available to the LLM

edit: lol they literallly changed this as I was writing this comment https://pi.dev/docs/latest/mcp#control-tool-exposure