Retrieval on Semantic Benchmarks
100Success RateQwen3C-30B-Inst-Q3_K_M
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3C-30B-Inst-Q3_K_MType=Agent2026.04 | 100 | 80 | 62.7 | 58 | 66 | 74 | 78 | 80 | 22.2 | 1.42 | |
| qwen3-coder:30bType=Agent2026.04 | 100 | 80 | 51.8 | 46 | 52 | 72 | 76 | 80 | 21.1 | 1.34 | |
| BM25Type=Baseline2026.04 | 100 | 78 | 54.1 | 48 | 60 | 66 | 78 | 78 | — | — | |
| deepseek-coder-v3:16bType=Agent2026.04 | 100 | 66 | 39.6 | 32 | 46 | 52 | 60 | 66 | 47.9 | 1.54 | |
| SemanticType=Baseline2026.04 | 100 | 54 | 27.9 | 22 | 32 | 38 | 52 | 54 | — | — | |
| Simple (bag-of-words)Type=Baseline2026.04 | 100 | 54 | 27.9 | 22 | 32 | 38 | 52 | 54 | — | — | |
| qwen2.5-coder:7bType=Agent2026.04 | 100 | 54 | 31.1 | 26 | 36 | 40 | 52 | 54 | 59.3 | 0.84 | |
| Qwen3C-30B-Inst-Q3_K_MType=Agent2026.04 | 100 | 42 | 34.8 | 32 | 38 | 38 | 40 | 42 | 22.7 | 1.4 | |
| deepseek-coder:33bType=Agent2026.04 | 100 | 12 | 8.7 | 8 | 8 | 12 | 12 | 12 | 180.4 | 0.14 | |
| granite-code:34bType=Agent2026.04 | 100 | 2 | 1 | 0 | 2 | 2 | 2 | 2 | 111.3 | 0.04 | |
| Hybrid (BM25+semantic)Type=Baseline2026.04 | 100 | 2 | 0.1 | 0 | 0 | 0 | 0 | 2 | — | — | |
| qwen2.5-coder:1.5bType=Agent2026.04 | 100 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 63.7 | 0 | |
| qwen2.5-coder:3bType=Agent2026.04 | 94 | 60 | 36.6 | 28 | 45 | 53 | 55 | 57 | 210.4 | 1.51 | |
| qwen2.5-coder:14bType=Agent2026.04 | 82 | 56 | 46.1 | 41 | 51 | 51 | 56 | 56 | 73.4 | 1.05 | |
| microcoder-oss-20bType=Agent2026.04 | 54 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 47.6 | 0 | |
| microcoder-deepseekr1-14.8Type=Agent2026.04 | 52 | 73 | 45.3 | 38 | 46 | 65 | 69 | 73 | 107.4 | 4.15 | |
| qwen3vl-4b-orlexType=Agent2026.04 | 12 | 8 | 8 | 8 | 8 | 8 | 8 | 8 | 37.9 | 0.14 |