Skill Retrieval on BigCodeBench
73.2nDCG@1Qwen3-32B
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Qwen3-32BRetrieval Stage=Reranker, Candidate selection=top-50 candidates retrieved by BM252026.04 | 73.2 | 74.1 | |
| Mistral3.1-24BRetrieval Stage=Reranker, Candidate selection=top-50 candidates retrieved by BM252026.04 | 72 | 72.9 | |
| Llama-3.3-70BRetrieval Stage=Reranker, Candidate selection=top-50 candidates retrieved by BM252026.04 | 71.1 | 72.1 | |
| Qwen3-235BRetrieval Stage=Reranker, Candidate selection=top-50 candidates retrieved by BM252026.04 | 70.7 | 73.1 | |
| Qwen3-4BRetrieval Stage=Reranker, Candidate selection=top-50 candidates retrieved by BM252026.04 | 68.4 | 66.6 | |
| BM25Retrieval Stage=First-stage2026.04 | 61.7 | 55.4 | |
| HybridRetrieval Stage=First-stage2026.04 | 61.7 | 59.9 | |
| Llama-3.1-8BRetrieval Stage=Reranker, Candidate selection=top-50 candidates retrieved by BM252026.04 | 59.2 | 61.6 | |
| TF-IDFRetrieval Stage=First-stage2026.04 | 55.2 | 52 | |
| BGERetrieval Stage=First-stage2026.04 | 54 | 53.6 | |
| ContrieverRetrieval Stage=First-stage2026.04 | 49.2 | 47.9 |