Rank and budget disagree
Qwen3-Embedding-4B leads MRR, while RepoMap packs the most gold context into 8K tokens.
Repository context retrieval for coding agents
Agent Retrieval Bench
A benchmark for finding the files an agent needs next, not merely the code that looks most similar to a query.
The benchmark
Agentic relevance is defined by workflow need.
Queries come from real coding signals: implementation changes, review comments, failure traces, and anchored edits. Gold files are frozen at the base commit and identify the context needed for the next step in the workflow.
427total samples
345positive retrieval cases
82no-gold selective cases
25open-source repositories
Five subsets
Different workflow signals expose different retrieval failures.
Four subsets evaluate positive retrieval. A fifth asks whether a system should return any context at all.
code2testRetrieve regression tests implied by an implementation change.
106comment2contextFind missing context beyond the file named in a review comment.
80trace2codeMove from visible failure frames to the implementation that matters.
101edit2rippleRetrieve additional files affected by an anchored edit.
58selectiveAbstain on natural evidence-backed and wrong-repository no-gold cases.
82Positive retrieval leaderboard
No retrieval family dominates every objective.
Results are sample-weighted over all 345 positive cases. BCY@8k measures gold-file coverage after ranked files are packed into an 8K-token context budget.
| Retriever | MRR | Recall@20 | BCY@8k |
|---|---|---|---|
| Qwen3-Embedding-4B | 0.2379 | 0.6306 | 0.3409 |
| Qwen3-Embedding-8B | 0.2336 | 0.7029 | 0.3732 |
| pplx-embed-v1-4b * | 0.2267 | 0.6072 | 0.3549 |
| RepoMap | 0.2158 | 0.6333 | 0.3788 |
| nomic-embed-code | 0.1986 | 0.5244 | 0.2781 |
| jina-code-embeddings-0.5b | 0.1914 | 0.4823 | 0.2783 |
| Lexical | 0.1574 | 0.4940 | 0.2650 |
| BM25 | 0.1520 | 0.4452 | 0.2051 |
* The pplx context-budget result is provisional because its run used a different tokenizer path. Rank-based metrics are directly comparable.
Task winners change
Embeddings, lexical retrieval, and repository structure have complementary strengths across workflow signals.
Abstention remains open
On 50 natural no-gold cases, simple top-score thresholds do not improve selective success over always returning.
Download
Each subset ships with its matching frozen corpus.
Inspect the release catalog, download one subset or all five current tracks, then merge their corpus manifests when running selective evaluation.
v2_code2test106v2_comment2context80v2_trace2code101v2_edit2ripple58v2_abstention82
# Install and inspect the evaluation CLI pip install -e . arb releases # Download all five benchmark subsets and their corpora arb download-benchmark \ --all \ --local-dir data \ --force # Deduplicate corpus snapshots for selective evaluation arb merge-corpus-manifests --local-dir data