# Usefulness Prediction on SIGIR 16 Source: [Usefulness Prediction on SIGIR 16]() Collection index: [llms.txt]() Data updated: 2026-06-16T09:12:55.849Z Evaluation of Usefulness Prediction performance on the SIGIR 16 dataset, using metrics such as S-rho, C-kappa, Precision, Recall, F1, and MAE. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Usefulness Prediction | | Dataset (including any stated split) | SIGIR 16 | | Metric | S-rho | | Unit | SCALAR | | Direction | Higher is better | | Results loaded | 26 of 26 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [S-rho](): Higher is better; unit: SCALAR. - [C-kappa](): Higher is better; unit: SCALAR. - [Precision](): Higher is better; unit: PERCENT. - [Recall](): Higher is better; unit: PERCENT. - [F1 Score](): Higher is better; unit: PERCENT. - [MAE](): Lower is better; unit: SCALAR. ## Recorded results | Method | S-rho | Result date | Report paper | | --- | --- | --- | --- | | TRUERubric+Session | 0.53 | Not recorded | 68fa787a1e209131cea78911 | | TRUESession | 0.5 | Not recorded | 68fa787a1e209131cea78911 | | TRUERubric+Session | 0.5 | Not recorded | 68fa787a1e209131cea78911 | | TRUESession | 0.43 | Not recorded | 68fa787a1e209131cea78911 | | GBDT | 0.42 | Not recorded | 68fa787a1e209131cea78911 | | TRUESession | 0.42 | Not recorded | 68fa787a1e209131cea78911 | | TRUERubric+Session | 0.42 | Not recorded | 68fa787a1e209131cea78911 | | TRUERubric+Session | 0.41 | Not recorded | 68fa787a1e209131cea78911 | | TRUERubric+Baseline | 0.4 | Not recorded | 68fa787a1e209131cea78911 | | CLUE | 0.38 | Not recorded | 68fa787a1e209131cea78911 | | TRUEBaseline | 0.38 | Not recorded | 68fa787a1e209131cea78911 | | CLUE | 0.37 | Not recorded | 68fa787a1e209131cea78911 | | TRUESession | 0.36 | Not recorded | 68fa787a1e209131cea78911 | | U\_a (Human–Human) | 0.35 | Not recorded | 68fa787a1e209131cea78911 | | TRUEBaseline | 0.34 | Not recorded | 68fa787a1e209131cea78911 | | TRUERubric+Baseline | 0.32 | Not recorded | 68fa787a1e209131cea78911 | | GBDT | 0.31 | Not recorded | 68fa787a1e209131cea78911 | | Linear Regression | 0.3 | Not recorded | 68fa787a1e209131cea78911 | | TRUEBaseline | 0.29 | Not recorded | 68fa787a1e209131cea78911 | | TRUERubric+Baseline | 0.28 | Not recorded | 68fa787a1e209131cea78911 | | TRUEBaseline | 0.27 | Not recorded | 68fa787a1e209131cea78911 | | TRUERubric+Session | 0.27 | Not recorded | 68fa787a1e209131cea78911 | | TRUERubric+Baseline | 0.26 | Not recorded | 68fa787a1e209131cea78911 | | TRUESession | 0.23 | Not recorded | 68fa787a1e209131cea78911 | | TRUEBaseline | 0.12 | Not recorded | 68fa787a1e209131cea78911 | | TRUERubric+Baseline | 0.1 | Not recorded | 68fa787a1e209131cea78911 | ### TRUERubric+Session S-rho: 0.53. Result ID: 6a311385fbc373592c2da1d8. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"70B"}. This record does not assert independent reproduction. ### TRUESession S-rho: 0.5. Result ID: 6a311385fbc373592c2da1ce. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"70B"}. This record does not assert independent reproduction. ### TRUERubric+Session S-rho: 0.5. Result ID: 6a311385fbc373592c2da1d9. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"GPT-3.5"}. This record does not assert independent reproduction. ### TRUESession S-rho: 0.43. Result ID: 6a311385fbc373592c2da1cf. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"GPT-3.5"}. This record does not assert independent reproduction. ### GBDT S-rho: 0.42. Result ID: 6a311385fbc373592c2da1c3. [Reporting paper]() Recorded evaluation configuration: {"Evaluation Protocol":"iid"}. This record does not assert independent reproduction. ### TRUESession S-rho: 0.42. Result ID: 6a311385fbc373592c2da1d0. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"GPT-4o"}. This record does not assert independent reproduction. ### TRUERubric+Session S-rho: 0.42. Result ID: 6a311385fbc373592c2da1d7. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"8B"}. This record does not assert independent reproduction. ### TRUERubric+Session S-rho: 0.41. Result ID: 6a311385fbc373592c2da1da. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"GPT-4o"}. This record does not assert independent reproduction. ### TRUERubric+Baseline S-rho: 0.4. Result ID: 6a311385fbc373592c2da1d5. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"GPT-4o"}. This record does not assert independent reproduction. ### CLUE S-rho: 0.38. Result ID: 6a311385fbc373592c2da1c6. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"GPT-4o"}. This record does not assert independent reproduction. ### TRUEBaseline S-rho: 0.38. Result ID: 6a311385fbc373592c2da1cb. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"GPT-4o"}. This record does not assert independent reproduction. ### CLUE S-rho: 0.37. Result ID: 6a311385fbc373592c2da1c5. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"GPT-3.5"}. This record does not assert independent reproduction. ### TRUESession S-rho: 0.36. Result ID: 6a311385fbc373592c2da1cd. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"8B"}. This record does not assert independent reproduction. ### U\_a (Human–Human) S-rho: 0.35. Result ID: 6a311385fbc373592c2da1c1. [Reporting paper]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. ### TRUEBaseline S-rho: 0.34. Result ID: 6a311385fbc373592c2da1c9. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"70B"}. This record does not assert independent reproduction. ### TRUERubric+Baseline S-rho: 0.32. Result ID: 6a311385fbc373592c2da1d4. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"GPT-3.5"}. This record does not assert independent reproduction. ### GBDT S-rho: 0.31. Result ID: 6a311385fbc373592c2da1c4. [Reporting paper]() Recorded evaluation configuration: {"Evaluation Protocol":"ood"}. This record does not assert independent reproduction. ### Linear Regression S-rho: 0.3. Result ID: 6a311385fbc373592c2da1c2. [Reporting paper]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. ### TRUEBaseline S-rho: 0.29. Result ID: 6a311385fbc373592c2da1c8. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"8B"}. This record does not assert independent reproduction. ### TRUERubric+Baseline S-rho: 0.28. Result ID: 6a311385fbc373592c2da1d2. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"8B"}. This record does not assert independent reproduction. ### TRUEBaseline S-rho: 0.27. Result ID: 6a311385fbc373592c2da1ca. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"GPT-3.5"}. This record does not assert independent reproduction. ### TRUERubric+Session S-rho: 0.27. Result ID: 6a311385fbc373592c2da1d6. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"3B"}. This record does not assert independent reproduction. ### TRUERubric+Baseline S-rho: 0.26. Result ID: 6a311385fbc373592c2da1d3. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"70B"}. This record does not assert independent reproduction. ### TRUESession S-rho: 0.23. Result ID: 6a311385fbc373592c2da1cc. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"3B"}. This record does not assert independent reproduction. ### TRUEBaseline S-rho: 0.12. Result ID: 6a311385fbc373592c2da1c7. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"3B"}. This record does not assert independent reproduction. ### TRUERubric+Baseline S-rho: 0.1. Result ID: 6a311385fbc373592c2da1d1. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"3B"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.