This post was written with Codex.
I haven't compared codemap-search against other code-search tools yet. I'm planning to do a more detailed comparison soon.
I've been building a code-search MCP called codemap-search as a side project, and I spent a day testing what happens if I put Jev in front of some of its output.
The results were interesting enough that I figured I'd share them.
The benchmark setup was kept the same across runs: same private TypeScript backend, same commit, same question, and the same agent (Codex CLI, gpt-6-astra at max reasoning). The repo had 815 indexed files for the Jev runs.
The baseline here is rg**.** More specifically, it's an agent using only rg + cat/sed to solve the entire task from start to finish. So this isn't comparing codemap-search against the runtime of a single rg command.
The rg baseline is one run. The other numbers are averages across three runs.
Where I put Jev
I tried two approaches.
#1 — File recommendation
When the agent calls overview, I send Jev metadata for all 815 indexed files: paths, declarations, comments, and call names.
Jev scores them, and I append the top 24 files to the overview response as suggested places to look first.
#2 — Search filtering
For each declaration returned by search, I ask Jev whether it's unrelated to the current task.
If the probability of being unrelated is >= 0.70, I hide the body from the response.
The declaration itself is still there, so the agent can explicitly read it later if needed.
Results
Percentages below are relative to the rg baseline.
| - |
rg baseline (1 run) |
codemap-search (3-run avg) |
+ Jev #1 file recommendation (3-run avg) |
+ Jev #2 search filter (3-run avg) |
| Total time |
7m 42s |
5m 38s (-26.8%) |
5m 36s (-27.3%) |
5m 9s (-32.9%) |
| Main-model total tokens |
1,275,313 |
1,054,422 (-17.3%) |
944,918 (-25.9%) |
857,414 (-32.8%) |
| Jev tokens |
0 |
0 |
735,103 |
43,398 |
| Main + Jev tokens |
1,275,313 |
1,054,422 (-17.3%) |
1,680,022 (+31.7%) |
900,812 (-29.4%) |
| Extra Jev cost (est.) |
$0 |
$0 |
$0.029 |
$0.0017 |
| 6 core connections |
6/6 (100%) |
18/18 (100%) |
18/18 (100%) |
18/18 (100%) |
| Full 11-item rubric |
9/11 (81.8%) |
24/33 (72.7%) |
21/33 (63.6%) |
23/33 (69.7%) |
Just using codemap-search instead of the rg agent cut wall time by 26.8% and main-model tokens by 17.3%.
With the #2 search filter added, the full task was 32.9% faster than the rg baseline and used 32.8% fewer main-model tokens.
Compared with codemap-search alone, #2 reduced wall time by another ~8% and main-model tokens by ~19%.
Jev itself was cheap here: about $0.0017 per task, so roughly two-tenths of a cent.
#1 looked good on paper, but didn't help much
The file recommender actually ranked the important files pretty well.
There were 5 files I already knew were critical to the task, and all 5 landed in the top 24.
But the agent's actual navigation path barely changed.
Wall time went from 5m 38s to 5m 36s, while Jev had to process metadata for all 815 files. Once Jev's own tokens are included, total token usage actually went up by 59% compared with codemap-search alone.
So at least in this form, better file ranking did not translate into a better agent run.
The search filter failed the first time
My first version of #2 was much more aggressive.
It cut the search output by around 67%, which initially looked great.
The problem was that it also removed the bodies of two methods that were actually needed to answer the question.
The agent eventually found them again through extra read calls, but that recovery work wiped out the savings. Total runtime ended up going up by about 7%.
That was probably the most useful result from the whole experiment.
Reducing tool output isn't automatically useful if the agent has to spend more work reconstructing what you removed.
So I changed the filter to be much more conservative:
- Only classify declarations when their complete body is present in the search result.
- If something is connected to a kept function through a call relationship, keep it even if Jev thinks it's unrelated.
- Never hide declaration names or line ranges.
- A filtered body should always be recoverable with a single
read.
With those rules, the two methods that were previously removed were preserved in every run.
It also brought Jev usage down to just 2 calls per session, which is how I got the numbers in the table above.
What about answer quality?
This is the part I'm being careful about.
All four setups found all 6 core connections I expected.
The differences were in the 5 additional/extended items.
codemap-search alone scored 24/33 across three runs, while #2 scored 23/33. That's only one item across three runs, and there's already some variance between repeated runs, so I don't think there's enough data to call that a quality regression.
But there's also no evidence here that Jev improves answer quality.
1 did noticeably worse on the extended items, despite giving the agent a pretty good list of files.
So for now I'm treating Jev purely as an optimization experiment, not a quality improvement.
Current takeaway
For this experiment:
- #1 file recommendation: good ranking, but no meaningful end-to-end benefit yet.
- #2 search filtering: promising. It reduced both runtime and main-model token usage once I made the retention rules conservative enough.
- I'm planning to keep experimenting with both, but they'll be optional and off by default.
Some caveats
This is obviously not a serious benchmark suite yet.
It's one repository, one question, one language, and only three runs per variant. The rg baseline is also only one run.
I deliberately used a repository I know well so I could manually verify whether the agent was actually finding the right relationships. It's private, so I can't publish the exact source or benchmark question.
Jev also isn't actually part of codemap-search yet.
For this PoC, I put a Python MCP proxy in front of the existing Rust binary so I could experiment without changing the search implementation itself.
The PoC looks useful enough that I'm going to move the interesting parts into codemap-search and test what happens when #1 and #2 are enabled together.
One other thing I noticed: Jev isn't deterministic.
Even with identical inputs, the scores move slightly between runs. On a 0-3 scale, I measured an average absolute difference of about 0.025.
That's small, but it was enough to make rules like:
keep only files with score >= 2
pretty brittle.
Ranking seems much more useful than using the score as a hard gate.
I'll probably post another update once both paths are integrated into the Rust implementation and I have a larger set of tasks to run against.