Relevance Assessment Label Alignment on DL 2019
0.46Cohen's Kappa (κ)query-contrastive
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| query-contrastiveModel=gpt-oss-120b, q=1, d+=1, d-=12026.04 | 0.46 | 0.54 | 38 | 45 | 11 | 7 | |
| query-contrastiveModel=gpt-oss-120b, q=1, d+=5, d-=52026.04 | 0.44 | 0.54 | 39 | 45 | 12 | 5 | |
| TREC Topic (Baseline)Model=TREC Human Judgments2026.04 | 0.41 | 0.67 | 31 | 40 | 13 | 16 | |
| queryModel=gpt-oss-120b, q=12026.04 | 0.37 | 0.58 | 40 | 49 | 8 | 3 |