Relevance Assessment Label Alignment on Robust 2004
0.56Cohen's Kappa (κ)TREC Topic (Baseline)
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| TREC Topic (Baseline)Model=TREC Human Judgments2026.04 | 0.56 | 0.4 | 43 | 33 | 24 | |
| query-contrastiveModel=gpt-oss-120b, q=1, d+=3, d-=32026.04 | 0.44 | 0.49 | 50 | 36 | 14 | |
| query-contrastiveModel=gpt-oss-120b, q=1, d+=4, d-=42026.04 | 0.44 | 0.49 | 51 | 36 | 13 | |
| query-contrastiveModel=gpt-oss-120b, q=1, d+=5, d-=52026.04 | 0.44 | 0.49 | 51 | 36 | 13 | |
| query-contrastiveModel=gpt-oss-120b, q=1, d+=2, d-=22026.04 | 0.43 | 0.51 | 51 | 36 | 13 | |
| query-contrastiveModel=gpt-oss-120b, q=1, d+=1, d-=12026.04 | 0.42 | 0.51 | 53 | 36 | 12 | |
| query-docs-posModel=gpt-oss-120b, q=1, d+=52026.04 | 0.38 | 0.55 | 54 | 39 | 7 | |
| query-docs-posModel=gpt-oss-120b, q=1, d+=12026.04 | 0.36 | 0.57 | 57 | 35 | 7 | |
| query-docs-negModel=gpt-oss-120b, q=1, d-=12026.04 | 0.33 | 0.6 | 61 | 33 | 6 | |
| query-docs-negModel=gpt-oss-120b, q=1, d-=52026.04 | 0.32 | 0.62 | 63 | 32 | 6 | |
| queryModel=gpt-oss-120b, q=12026.04 | 0.31 | 0.61 | 60 | 36 | 3 |