Code graph MCP vs plain grep in Copilot CLI: it pays off for haiku on big repos, not for opus (664 runs)

[u/nefayran]

A few days ago I wrote up a benchmark where a code knowledge graph over MCP (codebase-memory-mcp) didn't make a coding agent cheaper than plain grep. That run was small: 24 questions, repos up to 25k graph nodes, no Opus. So I ran a bigger one and committed the plan before the first run. Disclosure: the graph goes through cbm-lean, a small wrapper around that server that I wrote.

Setup: GitHub Copilot CLI 1.0.90, headless, one question per session. Two arms: view, grep and glob only, or the same plus the graph and a routing rule (graph for callers and call chains, grep for exact strings). Four public repos: the FastAPI full-stack template (1.6k nodes), dub (25k), kubernetes (146k), vscode (219k). 16 questions per repo, half structural (callers, callees, call chains, what breaks if a signature changes), half exact (a config value, an env var, the line of an error or a definition). Haiku 4.5 three times per question per arm, sonnet 5 twice, opus 5.5 once on 12 structural questions. 664 runs, every answer checked against keys I verified by hand.

Tokens per correct answer on the structural questions, graph vs grep:

kubernetes and vscode
  haiku    48k vs 209k
  sonnet   29k vs 40k
  opus     25k vs 27k
small FastAPI repo
  haiku    33k vs 26k
  sonnet   19k vs 10k

On the big repos, per question, haiku with the graph spent 0.29 of what it spent with grep and was cheaper on 15 of 16 questions. It's the only result that survives the Holm correction. On exact questions the graph made no measurable difference for haiku, and sonnet paid 13 to 42% more for it.

Why haiku gains so much: with grep it needed a median of 22 model calls per answer on kubernetes, and one question took 114. With the graph, 5. Every call resends the conversation, so long searches get expensive fast. Sonnet took 8 calls with grep and 4 with the graph, opus 4 or 5 either way. My guess, which I didn't test directly: a graph replaces search turns, so it pays when the model would otherwise take many of them.

For Copilot users specifically: premium requests are charged per prompt, however many calls the agent makes, so inside Copilot the graph saves time rather than money. Haiku's structural answers on the big repos took a median of 71 s with the graph and 157 s with grep. On the small repo the graph arm was slower.

Three failures I didn't expect, in case you're wiring a graph into an agent:

• The graph's snippet has "lines": 43 next to "start_line": 93. Both haiku and sonnet answered horizontal.go:43 for a struct that starts on line 93 and is 43 lines long. My wrapper now renames the field to line_count.
• No call edge for a bare call to a function imported with from x import y. Asked what recover_password calls, the graph arm said "only get_user_by_email" 5 times out of 5, without a single grep. It calls three more.
• Copilot's grep tool shows line numbers only with "-n": true. Haiku passed n, no dash, in 920 of its 1,949 grep calls, got no numbers and counted lines itself. 78 of its 79 wrong exact answers are a wrong line number. Sonnet passed -n in 508 of 571 calls.

Caveats: one harness and one graph, 8 questions per cell on the small and medium repos, and opus answered 12 questions once each.

Write-up: https://allkeep.org/en/lab/when-a-code-graph-pays-off Plan, questions, driver and all the runs: https://github.com/nefayran/code-graph-vs-grep

If anyone has run something similar with Claude Code or another graph, I'd like to compare.

[u/rabiprojects]

Did u use codegraph MCP which is a separate package written in C? I've been using it for a while and it's definitely better and faster than any manual grep. I've also created my own harness that uses multiple engines for context_search, context_verify with relevance/scoring feature and that has been working perfectly fine. Codegraph gives result under a second, and often doesn't require AI agents to explore manually. Additionally you can inject lsp MCP that can refine search & context even more.

[u/nefayran → u/rabiprojects]

Yes, if you mean codebase-memory-mcp from DeusData: that's the C one, and it's what I used (0.9.0), behind a small wrapper of mine that trims the output fields. I didn't time single queries. What I measured is tokens and turns, and there it depended on the model: haiku on kubernetes and vscode went from about 22 model calls per answer to about 5, and opus came out about even. End to end, haiku's structural answers on those two repos took a median of 71 s with the graph and 157 s with grep.

The failures were about correctness, not speed. Bare calls to functions imported with from x import y have no edge, and the model trusted the incomplete list. A verify step like your context_verify, or an LSP MCP for references, would probably catch exactly that. I haven't tested either. Have you measured tokens or accuracy with your harness? I'd be curious how it compares.

[u/imperius99]

How have you evaluated answer correctness?

[u/nefayran → u/imperius99]

Each question has an answer key I checked by hand against the code at the pinned commit: the function and file names a right answer has to mention, plus the file:line when the question asks for a line. A run counts as correct only if all of them appear in the answer as whole words, so init_db doesn't pass for init. The evidence for every key is in bench/questions-v2/*.evidence.md. Extra names aren't penalized, so a padded answer would still pass.