# Probability Calibration on ChaosNLI combined SNLI+MNLI K=3 Source: [Probability Calibration on ChaosNLI combined SNLI+MNLI K=3]() Collection index: [llms.txt]() Data updated: 2026-03-25T13:47:34.897Z Evaluation of model probability calibration performance on the ChaosNLI dataset, which combines samples from SNLI and MNLI (K=3 subset), measured using metrics like Expected Calibration Error (ECE), Brier Score (Br), and Negative Log Likelihood (NLL). A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Probability Calibration | | Dataset (including any stated split) | ChaosNLI combined SNLI+MNLI K=3 | | Metric | ECE | | Unit | SCALAR | | Direction | Lower is better | | Results loaded | 26 of 26 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [ECE](): Lower is better; unit: SCALAR. - [aECE](): Lower is better; unit: SCALAR. - [cwECE](): Lower is better; unit: SCALAR. - [Br](): Lower is better; unit: SCALAR. - [NLL](): Lower is better; unit: SCALAR. ## Recorded results | Method | ECE | Result date | Report paper | | --- | --- | --- | --- | | IR-Soft | 2.15 | Not recorded | 69c36c3d0636e5bf4afc664b | | Dirichlet-Soft | 2.57 | Not recorded | 69c36c3d0636e5bf4afc664b | | IR-Soft | 2.65 | Not recorded | 69c36c3d0636e5bf4afc664b | | LS-TS | 2.65 | Not recorded | 69c36c3d0636e5bf4afc664b | | SoftPlatt | 2.66 | Not recorded | 69c36c3d0636e5bf4afc664b | | MCTS S=1 | 2.82 | Not recorded | 69c36c3d0636e5bf4afc664b | | SoftPlatt | 3.17 | Not recorded | 69c36c3d0636e5bf4afc664b | | Dirichlet-Soft | 3.19 | Not recorded | 69c36c3d0636e5bf4afc664b | | MCTS S=1 | 3.2 | Not recorded | 69c36c3d0636e5bf4afc664b | | SLTS | 3.22 | Not recorded | 69c36c3d0636e5bf4afc664b | | SLTS | 3.45 | Not recorded | 69c36c3d0636e5bf4afc664b | | VS | 3.46 | Not recorded | 69c36c3d0636e5bf4afc664b | | Oracle TS | 3.48 | Not recorded | 69c36c3d0636e5bf4afc664b | | Oracle TS | 3.66 | Not recorded | 69c36c3d0636e5bf4afc664b | | VS | 3.91 | Not recorded | 69c36c3d0636e5bf4afc664b | | LS-TS | 4.16 | Not recorded | 69c36c3d0636e5bf4afc664b | | TS | 10.55 | Not recorded | 69c36c3d0636e5bf4afc664b | | Platt | 11.16 | Not recorded | 69c36c3d0636e5bf4afc664b | | ATS | 11.36 | Not recorded | 69c36c3d0636e5bf4afc664b | | Dirichlet-Hard | 11.55 | Not recorded | 69c36c3d0636e5bf4afc664b | | TS | 11.63 | Not recorded | 69c36c3d0636e5bf4afc664b | | Platt | 12.43 | Not recorded | 69c36c3d0636e5bf4afc664b | | Dirichlet-Hard | 12.69 | Not recorded | 69c36c3d0636e5bf4afc664b | | ATS | 13.09 | Not recorded | 69c36c3d0636e5bf4afc664b | | Uncalibrated | 27.79 | Not recorded | 69c36c3d0636e5bf4afc664b | | Uncalibrated | 35.97 | Not recorded | 69c36c3d0636e5bf4afc664b | ### IR-Soft ECE: 2.15. Result ID: 69c3e77347f34ce924ad8805. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeBERTa-v3"}. This record does not assert independent reproduction. ### Dirichlet-Soft ECE: 2.57. Result ID: 69c3e77347f34ce924ad87f9. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"RoBERTa-Large"}. This record does not assert independent reproduction. ### IR-Soft ECE: 2.65. Result ID: 69c3e77347f34ce924ad87f8. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"RoBERTa-Large"}. This record does not assert independent reproduction. ### LS-TS ECE: 2.65. Result ID: 69c3e77347f34ce924ad8807. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeBERTa-v3","Temperature (T)":"6.15"}. This record does not assert independent reproduction. ### SoftPlatt ECE: 2.66. Result ID: 69c3e77347f34ce924ad87f6. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"RoBERTa-Large"}. This record does not assert independent reproduction. ### MCTS S=1 ECE: 2.82. Result ID: 69c3e77347f34ce924ad87f5. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"RoBERTa-Large","Temperature (T)":"3.49"}. This record does not assert independent reproduction. ### SoftPlatt ECE: 3.17. Result ID: 69c3e77347f34ce924ad8803. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeBERTa-v3"}. This record does not assert independent reproduction. ### Dirichlet-Soft ECE: 3.19. Result ID: 69c3e77347f34ce924ad8806. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeBERTa-v3"}. This record does not assert independent reproduction. ### MCTS S=1 ECE: 3.2. Result ID: 69c3e77347f34ce924ad8802. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeBERTa-v3","Temperature (T)":"5.45"}. This record does not assert independent reproduction. ### SLTS ECE: 3.22. Result ID: 69c3e77347f34ce924ad87f4. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"RoBERTa-Large","Temperature (T)":"3.41"}. This record does not assert independent reproduction. ### SLTS ECE: 3.45. Result ID: 69c3e77347f34ce924ad8801. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeBERTa-v3","Temperature (T)":"5.28"}. This record does not assert independent reproduction. ### VS ECE: 3.46. Result ID: 69c3e77347f34ce924ad87f7. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"RoBERTa-Large"}. This record does not assert independent reproduction. ### Oracle TS ECE: 3.48. Result ID: 69c3e77347f34ce924ad87fb. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"RoBERTa-Large","Temperature (T)":"3.34"}. This record does not assert independent reproduction. ### Oracle TS ECE: 3.66. Result ID: 69c3e77347f34ce924ad8808. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"DeBERTa-v3","Temperature (T)":"5.21"}. This record does not assert independent reproduction. ### VS ECE: 3.91. Result ID: 69c3e77347f34ce924ad8804. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeBERTa-v3"}. This record does not assert independent reproduction. ### LS-TS ECE: 4.16. Result ID: 69c3e77347f34ce924ad87fa. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"RoBERTa-Large","Temperature (T)":"4.40"}. This record does not assert independent reproduction. ### TS ECE: 10.55. Result ID: 69c3e77347f34ce924ad87f0. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"RoBERTa-Large","Temperature (T)":"2.42"}. This record does not assert independent reproduction. ### Platt ECE: 11.16. Result ID: 69c3e77347f34ce924ad87f2. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"RoBERTa-Large"}. This record does not assert independent reproduction. ### ATS ECE: 11.36. Result ID: 69c3e77347f34ce924ad87f1. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"RoBERTa-Large","Temperature (T)":"2.28"}. This record does not assert independent reproduction. ### Dirichlet-Hard ECE: 11.55. Result ID: 69c3e77347f34ce924ad87f3. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"RoBERTa-Large"}. This record does not assert independent reproduction. ### TS ECE: 11.63. Result ID: 69c3e77347f34ce924ad87fd. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeBERTa-v3","Temperature (T)":"3.81"}. This record does not assert independent reproduction. ### Platt ECE: 12.43. Result ID: 69c3e77347f34ce924ad87ff. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"DeBERTa-v3"}. This record does not assert independent reproduction. ### Dirichlet-Hard ECE: 12.69. Result ID: 69c3e77347f34ce924ad8800. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeBERTa-v3"}. This record does not assert independent reproduction. ### ATS ECE: 13.09. Result ID: 69c3e77347f34ce924ad87fe. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"DeBERTa-v3","Temperature (T)":"3.57"}. This record does not assert independent reproduction. ### Uncalibrated ECE: 27.79. Result ID: 69c3e77347f34ce924ad87ef. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"RoBERTa-Large"}. This record does not assert independent reproduction. ### Uncalibrated ECE: 35.97. Result ID: 69c3e77347f34ce924ad87fc. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"DeBERTa-v3"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.