Document Retrieval on OrgForge (Evaluation Questions)
54.17MRR@10BM25
Evaluation results
| Method | Links | ||
|---|---|---|---|
| BM25Question Type=CAUSAL, N=202026.03 | 54.17 | 55 | |
| BM25Question Type=GAP DETECTION, N=32026.03 | 50 | 88.89 | |
| Dense retrieval (stella_en_1.5b_v5)Model=Losspost/stella_en_1.5b_v5, Parameters=1.5B, Framework=Ollama, Similarity=Cosine similarity, Top-K=10, N (Sample Size)=282026.03 | 39.4 | 60.71 | |
| BM25Question Type=RETRIEVAL, N=282026.03 | 35.48 | 64.29 | |
| BM25Question Type=Overall, N=832026.03 | 28.3 | 46.59 | |
| BM25Question Type=ROUTING, N=42026.03 | 21.9 | 100 | |
| Dense retrieval (stella_en_1.5b_v5)Model=Losspost/stella_en_1.5b_v5, Parameters=1.5B, Framework=Ollama, Similarity=Cosine similarity, Top-K=10, N (Sample Size)=202026.03 | 21.62 | 15 | |
| BM25Question Type=TEMPORAL, N=72026.03 | 4.9 | 42.86 | |
| BM25Question Type=PLAN, N=152026.03 | 0 | 0 | |
| BM25Question Type=ESCALATION, N=62026.03 | 0 | 0 |