Skill Retrieval on BigCodeBench (Recall@1, Recall@10)
28.2Recall@1Qwen3-32B
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Qwen3-32BRetrieval Stage=LLM-based reranker, Candidate Pool=top-50 candidates retrieved by BM252026.04 | 28.2 | 79.7 | |
| Mistral3.1-24BRetrieval Stage=LLM-based reranker, Candidate Pool=top-50 candidates retrieved by BM252026.04 | 27.7 | 79.4 | |
| Llama-3.3-70BRetrieval Stage=LLM-based reranker, Candidate Pool=top-50 candidates retrieved by BM252026.04 | 27.4 | 78.6 | |
| Qwen3-235BRetrieval Stage=LLM-based reranker, Candidate Pool=top-50 candidates retrieved by BM252026.04 | 27.2 | 80 | |
| Qwen3-4BRetrieval Stage=LLM-based reranker, Candidate Pool=top-50 candidates retrieved by BM252026.04 | 26.3 | 72.5 | |
| BM25Retrieval Stage=First-stage2026.04 | 23.6 | 61.1 | |
| HybridRetrieval Stage=First-stage2026.04 | 23.6 | 68.4 | |
| Llama-3.1-8BRetrieval Stage=LLM-based reranker, Candidate Pool=top-50 candidates retrieved by BM252026.04 | 22.9 | 68.7 | |
| TF-IDFRetrieval Stage=First-stage2026.04 | 20.9 | 60.2 | |
| BGERetrieval Stage=First-stage2026.04 | 20.7 | 62.1 | |
| ContrieverRetrieval Stage=First-stage2026.04 | 19 | 54.1 |