Aggregate Quality Recovery on UltraFeedback (kappa_max = 10 Aggregate)
0.8467Q-hatEnhanced (global sentiment)
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Enhanced (global sentiment)kappa_max=102026.05 | 0.8467 | 0.832 | 0.222 | — | |
| Ground truth Q*kappa_max=102026.05 | 0.6249 | — | 0 | — | |
| Hierarchical-Informedkappa_max=102026.05 | 0.5484 | 0.376 | 0.077 | — | |
| IPWkappa_max=102026.05 | 0.379 | — | 0.246 | — | |
| Corrected-Global (3-param)kappa_max=102026.05 | 0.3705 | 0.187 | 0.254 | — | |
| Basic (two-stage)kappa_max=102026.05 | 0.3638 | 0.331 | 0.261 | — | |
| Hierarchical-Sentimentkappa_max=102026.05 | 0.3004 | 0.14 | 0.325 | — | |
| Naive meankappa_max=102026.05 | 0.2617 | — | 0.363 | — |