# Safety Classification Disagreement on WildChat Intent Source: [Safety Classification Disagreement on WildChat Intent]() Collection index: [llms.txt]() Data updated: 2026-05-26T06:47:51.153Z Evaluation of model safety performance on the WildChat Intent dataset, specifically measuring the rate of disagreement among safety classifiers. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Safety classification disagreement | | Dataset (including any stated split) | WildChat Intent | | Metric | Disagreement Rate (per 1k conv) | | Unit | SCALAR | | Direction | Lower is better | | Results loaded | 30 of 30 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Disagreement Rate (per 1k conv)](): Lower is better; unit: SCALAR. ## Recorded results | Method | Disagreement Rate (per 1k conv) | Result date | Report paper | | --- | --- | --- | --- | | Gemini 3.1 | 0.4 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 | 0.4 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 Mini | 0.4 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 Mini | 0.6 | Not recorded | 6a1528e8dbd7150506f7dce2 | | Gemini 3.1 | 0.7 | Not recorded | 6a1528e8dbd7150506f7dce2 | | Safeguard 20B | 0.9 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 | 1 | Not recorded | 6a1528e8dbd7150506f7dce2 | | Gemini 3.1 | 1.4 | Not recorded | 6a1528e8dbd7150506f7dce2 | | Gemini 3.1 | 1.5 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 | 1.6 | Not recorded | 6a1528e8dbd7150506f7dce2 | | Safeguard 20B | 1.8 | Not recorded | 6a1528e8dbd7150506f7dce2 | | Safeguard 20B | 1.9 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 Nano | 2.2 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 | 2.7 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 Mini | 2.8 | Not recorded | 6a1528e8dbd7150506f7dce2 | | Safeguard 20B | 2.8 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 Mini | 2.9 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 Nano | 5.1 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 Nano | 5.6 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 Nano | 5.9 | Not recorded | 6a1528e8dbd7150506f7dce2 | | Gemini 3.1 | 6.2 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 Mini | 7.2 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 | 10.1 | Not recorded | 6a1528e8dbd7150506f7dce2 | | Gemini 3.1 | 11.5 | Not recorded | 6a1528e8dbd7150506f7dce2 | | Safeguard 20B | 11.5 | Not recorded | 6a1528e8dbd7150506f7dce2 | | Safeguard 20B | 15.3 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 Nano | 19.2 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 Mini | 22.9 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 | 24.1 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 Nano | 40.4 | Not recorded | 6a1528e8dbd7150506f7dce2 | ### Gemini 3.1 Disagreement Rate (per 1k conv): 0.4. Result ID: 6a1540d7fbc373592c0bff95. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Hate Speech","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### GPT-5.4 Disagreement Rate (per 1k conv): 0.4. Result ID: 6a1540d7fbc373592c0bff9b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Hate Speech","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### GPT-5.4 Mini Disagreement Rate (per 1k conv): 0.4. Result ID: 6a1540d7fbc373592c0bff9d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Harassment","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### GPT-5.4 Mini Disagreement Rate (per 1k conv): 0.6. Result ID: 6a1540d7fbc373592c0bffa1. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Hate Speech","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### Gemini 3.1 Disagreement Rate (per 1k conv): 0.7. Result ID: 6a1540d7fbc373592c0bff91. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Harassment","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### Safeguard 20B Disagreement Rate (per 1k conv): 0.9. Result ID: 6a1540d7fbc373592c0bffad. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Hate Speech","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### GPT-5.4 Disagreement Rate (per 1k conv): 1. Result ID: 6a1540d7fbc373592c0bff97. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Harassment","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### Gemini 3.1 Disagreement Rate (per 1k conv): 1.4. Result ID: 6a1540d7fbc373592c0bff93. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Non-Violent Crime","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### Gemini 3.1 Disagreement Rate (per 1k conv): 1.5. Result ID: 6a1540d7fbc373592c0bff94. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Hate Speech","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### GPT-5.4 Disagreement Rate (per 1k conv): 1.6. Result ID: 6a1540d7fbc373592c0bff99. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Non-Violent Crime","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### Safeguard 20B Disagreement Rate (per 1k conv): 1.8. Result ID: 6a1540d7fbc373592c0bffab. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Non-Violent Crime","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### Safeguard 20B Disagreement Rate (per 1k conv): 1.9. Result ID: 6a1540d7fbc373592c0bffac. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Hate Speech","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### GPT-5.4 Nano Disagreement Rate (per 1k conv): 2.2. Result ID: 6a1540d7fbc373592c0bffa7. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Hate Speech","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### GPT-5.4 Disagreement Rate (per 1k conv): 2.7. Result ID: 6a1540d7fbc373592c0bff9a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Hate Speech","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### GPT-5.4 Mini Disagreement Rate (per 1k conv): 2.8. Result ID: 6a1540d7fbc373592c0bff9f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Non-Violent Crime","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### Safeguard 20B Disagreement Rate (per 1k conv): 2.8. Result ID: 6a1540d7fbc373592c0bffa9. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Harassment","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### GPT-5.4 Mini Disagreement Rate (per 1k conv): 2.9. Result ID: 6a1540d7fbc373592c0bffa0. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Hate Speech","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### GPT-5.4 Nano Disagreement Rate (per 1k conv): 5.1. Result ID: 6a1540d7fbc373592c0bffa5. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Non-Violent Crime","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### GPT-5.4 Nano Disagreement Rate (per 1k conv): 5.6. Result ID: 6a1540d7fbc373592c0bffa6. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Hate Speech","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### GPT-5.4 Nano Disagreement Rate (per 1k conv): 5.9. Result ID: 6a1540d7fbc373592c0bffa3. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Harassment","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### Gemini 3.1 Disagreement Rate (per 1k conv): 6.2. Result ID: 6a1540d7fbc373592c0bff92. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Non-Violent Crime","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### GPT-5.4 Mini Disagreement Rate (per 1k conv): 7.2. Result ID: 6a1540d7fbc373592c0bff9e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Non-Violent Crime","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### GPT-5.4 Disagreement Rate (per 1k conv): 10.1. Result ID: 6a1540d7fbc373592c0bff98. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Non-Violent Crime","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### Gemini 3.1 Disagreement Rate (per 1k conv): 11.5. Result ID: 6a1540d7fbc373592c0bff90. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Harassment","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### Safeguard 20B Disagreement Rate (per 1k conv): 11.5. Result ID: 6a1540d7fbc373592c0bffaa. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Non-Violent Crime","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### Safeguard 20B Disagreement Rate (per 1k conv): 15.3. Result ID: 6a1540d7fbc373592c0bffa8. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Harassment","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### GPT-5.4 Nano Disagreement Rate (per 1k conv): 19.2. Result ID: 6a1540d7fbc373592c0bffa4. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Non-Violent Crime","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### GPT-5.4 Mini Disagreement Rate (per 1k conv): 22.9. Result ID: 6a1540d7fbc373592c0bff9c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Harassment","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### GPT-5.4 Disagreement Rate (per 1k conv): 24.1. Result ID: 6a1540d7fbc373592c0bff96. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Harassment","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### GPT-5.4 Nano Disagreement Rate (per 1k conv): 40.4. Result ID: 6a1540d7fbc373592c0bffa2. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Harassment","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.