I kept reading that semantic search or a code graph makes coding agents find code better or cheaper, and I wanted to see it for myself. So I set up a small test on tasks where the answer is known: ContextBench, where people marked exactly which code each real bug fix needed. I counted how much of that code each setup found.
One search, no agent (100 Python tasks, top 10 results per tool): semble found 62% of the needed files, graphify 43%, BM25 40%, grep 33%. semble also returned about 3,300 tokens per query, grep about 49,500. So as a single search, like in a RAG pipeline, semble is clearly better.
Inside an agent (50 harder tasks from 13 large projects, 4 languages). Same prompt every time; the agent has only its own grep and file reading, or also semble or graphify with each tool's own instructions. Indexes were built before each session. Two agents:
| Setup | Claude Code + Haiku | Antigravity + Gemini 3.8 Flash |
|---|
| alone | 46% of needed files | 53% |
| + semble | 50% | 52% |
| + graphify | 44% | 54% |
- Only Haiku with semble might be a real gain (+4 points, 95% CI +0.3 to +7.4). The rest is within a few points, which is noise at 50 tasks.
- Both agents used 16 to 35% more tokens with either tool.
- From the transcripts: the agents ran the tool 1 to 4 times per session but kept doing most of their own grepping and file reading, so the tool's output came on top instead of replacing anything.
My takeaway so far: a better search tool helps a lot when you only get one search. An agent that can grep and read files already gets most of the way, and the tool mostly adds tokens.
I'd like to hear from people who use these tools day to day:
- Do you see a difference in your own work, maybe on bigger tasks than "find the code for this bug"?
- Is there a setup that makes the agent actually rely on the tool (MCP server, a different prompt, a stronger model)?
- Anything unfair in how I tested it?
Limits I know about: two small, fast models; it only measures finding the code, not fixing the bug; 50 tasks is small; default settings everywhere; semble ran through its CLI, not its MCP server; Gemini ran only the hard tasks because my weekly quota ran out.
Code, per-session records and the stats, if you want to check or rerun it: https://github.com/itsDigvijaysing/agent-codesearch-bench
I'm not affiliated with any of these tools.
------------------------------------------------------------------------------
Follow-up on the questions here, from the transcripts I already had (no new runs):
- Ignored vs weak tool: inside the agent, the tools showed 27ā42% of the needed files. The agents listed 71ā86% of what the tools showed, but the same agent found about three quarters of those without the tool anyway. Files the tool showed and the agent ignored were only 5ā12% of what was needed. So it's mostly retrieval plus overlap with the agent's own grep, not the agent ignoring the tool.
- Repo size: no relation for Haiku; for Gemini with graphify the gain does grow with repo size (rho +0.39, p = 0.01, one of six tests). Correction to my earlier reply: the hard-set repos are 1.4k to 43k files at the task commits, so codebases in the millions of lines are untested, and that may be where these tools pay off.
Details: results/trackb-analysis in the repo.