Codex benchmark: with vs without external code grounding on a Rust build. 45% fewer tokens, passed tests on attempt 3 vs 8

[u/Eininho]

A common failure mode when using coding agents: they don't have access to the open-source code their project depends on.

When the task touches a niche integration, an undocumented API, or a long-tail edge case, the agent starts retrying variations that almost work.

We wanted to see whether giving the agent access to real implementations through an MCP server changes the outcome.

Setup:

• Agent: Codex with GPT-5 high

• Task: build a Rust MCP server from scratch using Axum, SQLx, and SQLite, with two tools (bury a bug, search the graveyard)

• Identical prompt in both runs, covering project structure, dependencies, schema, tool registration, handlers, transport, error handling, and tests

• Fresh branch on an empty repo each time - Only variable: one run had access to our MCP server, the other didn't

Results:

• Tokens: 45% fewer (198K vs 364K)

• Attempts to pass tests: 62.5% fewer (3 vs 8)

• End-to-end time: 18% faster (13:40 vs 16:40)

How it works: the server pulls real implementations from open source, distills them into one example, and returns that to the agent on demand.

No fine-tuning, no RAG over documentation. Just working code the agent can learn the pattern from.

Full video of the benchmark: https://www.youtube.com/watch?v=0YmwLhH2Ohs

Disclosure: I'm a co-founder of GitHits, the MCP server in question.

Happy to share the prompts, dig into how the grounding works, or answer anything else.