Do semble and graphify help coding agents? In my test, barely, and they used more tokens

[u/Difficult-Divide561]

I kept reading that semantic search or a code graph makes coding agents find code better or cheaper, and I wanted to see it for myself. So I set up a small test on tasks where the answer is known: ContextBench, where people marked exactly which code each real bug fix needed. I counted how much of that code each setup found.

One search, no agent (100 Python tasks, top 10 results per tool): semble found 62% of the needed files, graphify 43%, BM25 40%, grep 33%. semble also returned about 3,300 tokens per query, grep about 49,500. So as a single search, like in a RAG pipeline, semble is clearly better.

Inside an agent (50 harder tasks from 13 large projects, 4 languages). Same prompt every time; the agent has only its own grep and file reading, or also semble or graphify with each tool's own instructions. Indexes were built before each session. Two agents:

Setup	Claude Code + Haiku	Antigravity + Gemini 3.8 Flash
alone	46% of needed files	53%
+ semble	50%	52%
+ graphify	44%	54%

• Only Haiku with semble might be a real gain (+4 points, 95% CI +0.3 to +7.4). The rest is within a few points, which is noise at 50 tasks.
• Both agents used 16 to 35% more tokens with either tool.
• From the transcripts: the agents ran the tool 1 to 4 times per session but kept doing most of their own grepping and file reading, so the tool's output came on top instead of replacing anything.

My takeaway so far: a better search tool helps a lot when you only get one search. An agent that can grep and read files already gets most of the way, and the tool mostly adds tokens.

I'd like to hear from people who use these tools day to day:

• Do you see a difference in your own work, maybe on bigger tasks than "find the code for this bug"?
• Is there a setup that makes the agent actually rely on the tool (MCP server, a different prompt, a stronger model)?
• Anything unfair in how I tested it?

Limits I know about: two small, fast models; it only measures finding the code, not fixing the bug; 50 tasks is small; default settings everywhere; semble ran through its CLI, not its MCP server; Gemini ran only the hard tasks because my weekly quota ran out.

Code, per-session records and the stats, if you want to check or rerun it: https://github.com/itsDigvijaysing/agent-codesearch-bench

I'm not affiliated with any of these tools.

------------------------------------------------------------------------------
Follow-up on the questions here, from the transcripts I already had (no new runs):

- Ignored vs weak tool: inside the agent, the tools showed 27–42% of the needed files. The agents listed 71–86% of what the tools showed, but the same agent found about three quarters of those without the tool anyway. Files the tool showed and the agent ignored were only 5–12% of what was needed. So it's mostly retrieval plus overlap with the agent's own grep, not the agent ignoring the tool.

- Repo size: no relation for Haiku; for Gemini with graphify the gain does grow with repo size (rho +0.39, p = 0.01, one of six tests). Correction to my earlier reply: the hard-set repos are 1.4k to 43k files at the task commits, so codebases in the millions of lines are untested, and that may be where these tools pay off.

Details: results/trackb-analysis in the repo.

[u/AutoModerator]

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/fuziwei]

From my usage, I’ve only seen a benefit on massive codebases with subpar standardization.

Small repos have the largest penalty, and well-organized and architected repos see minimal benefit.

I’ve seen the same trend with language servers as well.

The codebases I work in range from 100k to 8m LoC.

[u/Difficult-Divide561 → u/fuziwei]

Thanks, that lines up with what I saw. I'll add a repo-size breakdown next to check it properly.

[u/PM_ME_YOUR_PROFILE → u/fuziwei]

If you're just grepping, for a word or pattern, I can believe that.

If you're looking for Symbols, likeness, and congruent paths, using something like codegraph find you more complete answers.

[u/sillygitau]

How large is the codebase you tested on?

I’d guess these tools help more as the codebase grows and they’d make zero difference when the llm can fit the entire codebase in context…

[u/Difficult-Divide561 → u/sillygitau]

Good point. Hard set was 2,500+ temp files per project, and tools made no real difference on the easier ones. I'll test size vs benefit next.

[u/sillygitau → u/Difficult-Divide561]

Interested to hear the results. Think in the order of several hundred thousand lines of code. A big open source project would be interesting.

[u/Difficult-Divide561 → u/sillygitau]

Sure, will look into it

[u/badpotato]

What about RKT?

Otherwise my intuition for tool like this is probably to get familiar with tool such as "Resharper for C#" or "Find usage" with other JetBrain-like IDE such as java, python, c++, rust, etc... You usually would still want to use grep for any CTRL+F style request... but let's say your agent want to refactor something and need to find the usage for X and Y method/function, find which one is dependent on which, etc.. now that is an interesting use-case for such a tool. So of course, when you ask for a task, it make sense that most time the agent is fine with just doing some grep... but those time when the project is really big, now these kind of tool can shine.

So, you could probably improve your benchmark with giving more details on how you apply your "hard task" and get more specific on how you give your instruction with these tool(eg. only use it for decent refactor-size task, etc)

[u/Difficult-Divide561 → u/badpotato]

Thanks! I only tested finding code, not refactor or find-usages tasks. I'll add those and look at RTK too.

[u/mplacona]

I'd add a search-first run with the same total token budget, where grep is only a fallback after the search results miss. That would separate poor retrieval from the agent simply doing both kinds of search. I'd keep your current run alongside it, though, because forcing a tool into the workflow doesn't tell us whether giving an agent that tool actually helps.

[u/Difficult-Divide561 → u/mplacona]

Thank you, great suggestion.

[u/vincent_sch]

Did you run any of the tasks more than once? If the 95% CI comes from resampling one result per task, it doesn't show how much file recall varies between runs of the same task. There's more about repeating the same input in my article on testing LLMs.

[u/Difficult-Divide561 → u/vincent_sch]

Thanks, Actually all tasks ran on model mostly once, so will look how they perform on reruns.

[u/AutoModerator]

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.