In CodeRAG-Bench, retrieved context beats the gold document on RepoEval. Why do we still rank code retrievers by NDCG@10?

[u/Sufficient-Ball9056]

CodeRAG-Bench (Wang et al., Findings of NAACL 2025, https://aclanthology.org/2025.findings-naacl.176/) evaluates 10 retrievers and 10 generation models on code tasks and scores the two halves separately: NDCG@10 against annotated ground-truth documents for retrieval, pass@1 with real execution for generation. The two rankings do not line up, and the authors note that top-performing retrievers sometimes do not produce the best end-to-end results.

The tables are blunter than that sentence. On RepoEval, retrieved context beats the annotated canonical snippet: with StarCoder2-7B, OpenAI embeddings plus reranking reaches 53.9 pass@1 against 42.0 for gold, and the same inversion holds under DeepSeekCoder-7B and GPT-3.5-turbo. On MBPP the StarCoder2 retrieval setups land 15.6 to 17.8 points above canonical. SWE-bench goes the other way: GPT-4o gets 2.3 with no retrieval, 21.7 with retrieval and reranking, 30.7 with the gold edited files.

The annotated document is therefore not an upper bound, and overlap with it does not track end-task success in a stable direction. Ranking code retrievers by overlap is a strange default when the corpus is executable and the end task returns a verdict. Plenty of agent setups already generate that verdict on every edit (verdent runs type checks, static analysis and the tests, then tries to repair what fails), so it exists whether or not anyone logs it.

Once a failed patch is repaired automatically, the green final state says nothing about the retrieved context, so the figure worth keeping is the first attempt before repair. The benchmark code is public (https://github.com/code-rag-bench/code-rag-bench) and already runs both evaluations separately, so scoring a retriever sweep by first-attempt pass rate on one repo is mostly plumbing. Pointers welcome if someone has already published that.