# Safety Classification Disagreement on WildChat Content Source: [Safety Classification Disagreement on WildChat Content]() Collection index: [llms.txt]() Data updated: 2026-05-26T06:47:51.183Z Evaluation of model safety performance on the WildChat Content dataset, specifically measuring the rate of disagreement among classifiers or evaluators. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Safety classification disagreement | | Dataset (including any stated split) | WildChat Content | | Metric | Disagreement Rate (per 1k Conversations) | | Unit | SCALAR | | Direction | Lower is better | | Results loaded | 30 of 30 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Disagreement Rate (per 1k Conversations)](): Lower is better; unit: SCALAR. ## Recorded results | Method | Disagreement Rate (per 1k Conversations) | Result date | Report paper | | --- | --- | --- | --- | | Gemini 3.1 | 0.4 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 Mini | 0.7 | Not recorded | 6a1528e8dbd7150506f7dce2 | | Gemini 3.1 | 1 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 Mini | 1.1 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 | 1.5 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 | 1.5 | Not recorded | 6a1528e8dbd7150506f7dce2 | | Gemini 3.1 | 1.7 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 | 1.9 | Not recorded | 6a1528e8dbd7150506f7dce2 | | Safeguard 20B | 1.9 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 Mini | 2.2 | Not recorded | 6a1528e8dbd7150506f7dce2 | | Safeguard 20B | 2.4 | Not recorded | 6a1528e8dbd7150506f7dce2 | | Gemini 3.1 | 2.5 | Not recorded | 6a1528e8dbd7150506f7dce2 | | Safeguard 20B | 2.9 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 Mini | 4.1 | Not recorded | 6a1528e8dbd7150506f7dce2 | | Safeguard 20B | 4.1 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 Nano | 4.6 | Not recorded | 6a1528e8dbd7150506f7dce2 | | Gemini 3.1 | 4.8 | Not recorded | 6a1528e8dbd7150506f7dce2 | | Gemini 3.1 | 5.1 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 Nano | 5.1 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 Mini | 6.8 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 Nano | 6.8 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 | 8.1 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 Nano | 8.1 | Not recorded | 6a1528e8dbd7150506f7dce2 | | Safeguard 20B | 9.7 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 | 12.1 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 Nano | 20.5 | Not recorded | 6a1528e8dbd7150506f7dce2 | | Safeguard 20B | 29.3 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 Mini | 42.2 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 | 47.2 | Not recorded | 6a1528e8dbd7150506f7dce2 | | GPT-5.4 Nano | 66.2 | Not recorded | 6a1528e8dbd7150506f7dce2 | ### Gemini 3.1 Disagreement Rate (per 1k Conversations): 0.4. Result ID: 6a1540d7fbc373592c0bffb1. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Non-Violent Crime","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### GPT-5.4 Mini Disagreement Rate (per 1k Conversations): 0.7. Result ID: 6a1540d7fbc373592c0bffbd. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Non-Violent Crime","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### Gemini 3.1 Disagreement Rate (per 1k Conversations): 1. Result ID: 6a1540d7fbc373592c0bffaf. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Harassment","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### GPT-5.4 Mini Disagreement Rate (per 1k Conversations): 1.1. Result ID: 6a1540d7fbc373592c0bffbb. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Harassment","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### GPT-5.4 Disagreement Rate (per 1k Conversations): 1.5. Result ID: 6a1540d7fbc373592c0bffb5. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Harassment","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### GPT-5.4 Disagreement Rate (per 1k Conversations): 1.5. Result ID: 6a1540d7fbc373592c0bffb7. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Non-Violent Crime","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### Gemini 3.1 Disagreement Rate (per 1k Conversations): 1.7. Result ID: 6a1540d7fbc373592c0bffb3. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Hate Speech","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### GPT-5.4 Disagreement Rate (per 1k Conversations): 1.9. Result ID: 6a1540d7fbc373592c0bffb9. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Hate Speech","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### Safeguard 20B Disagreement Rate (per 1k Conversations): 1.9. Result ID: 6a1540d7fbc373592c0bffc7. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Harassment","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### GPT-5.4 Mini Disagreement Rate (per 1k Conversations): 2.2. Result ID: 6a1540d7fbc373592c0bffbf. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Hate Speech","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### Safeguard 20B Disagreement Rate (per 1k Conversations): 2.4. Result ID: 6a1540d7fbc373592c0bffcb. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Hate Speech","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### Gemini 3.1 Disagreement Rate (per 1k Conversations): 2.5. Result ID: 6a1540d7fbc373592c0bffb2. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Hate Speech","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### Safeguard 20B Disagreement Rate (per 1k Conversations): 2.9. Result ID: 6a1540d7fbc373592c0bffc9. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Non-Violent Crime","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### GPT-5.4 Mini Disagreement Rate (per 1k Conversations): 4.1. Result ID: 6a1540d7fbc373592c0bffbe. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Hate Speech","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### Safeguard 20B Disagreement Rate (per 1k Conversations): 4.1. Result ID: 6a1540d7fbc373592c0bffca. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Hate Speech","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### GPT-5.4 Nano Disagreement Rate (per 1k Conversations): 4.6. Result ID: 6a1540d7fbc373592c0bffc5. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Hate Speech","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### Gemini 3.1 Disagreement Rate (per 1k Conversations): 4.8. Result ID: 6a1540d7fbc373592c0bffb0. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Non-Violent Crime","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### Gemini 3.1 Disagreement Rate (per 1k Conversations): 5.1. Result ID: 6a1540d7fbc373592c0bffae. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Harassment","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### GPT-5.4 Nano Disagreement Rate (per 1k Conversations): 5.1. Result ID: 6a1540d7fbc373592c0bffc3. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Non-Violent Crime","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### GPT-5.4 Mini Disagreement Rate (per 1k Conversations): 6.8. Result ID: 6a1540d7fbc373592c0bffbc. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Non-Violent Crime","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### GPT-5.4 Nano Disagreement Rate (per 1k Conversations): 6.8. Result ID: 6a1540d7fbc373592c0bffc1. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Harassment","Evaluation Protocol":"Constitution"}. This record does not assert independent reproduction. ### GPT-5.4 Disagreement Rate (per 1k Conversations): 8.1. Result ID: 6a1540d7fbc373592c0bffb6. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Non-Violent Crime","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### GPT-5.4 Nano Disagreement Rate (per 1k Conversations): 8.1. Result ID: 6a1540d7fbc373592c0bffc4. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Hate Speech","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### Safeguard 20B Disagreement Rate (per 1k Conversations): 9.7. Result ID: 6a1540d7fbc373592c0bffc8. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Non-Violent Crime","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### GPT-5.4 Disagreement Rate (per 1k Conversations): 12.1. Result ID: 6a1540d7fbc373592c0bffb8. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Hate Speech","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### GPT-5.4 Nano Disagreement Rate (per 1k Conversations): 20.5. Result ID: 6a1540d7fbc373592c0bffc2. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Non-Violent Crime","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### Safeguard 20B Disagreement Rate (per 1k Conversations): 29.3. Result ID: 6a1540d7fbc373592c0bffc6. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Harassment","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### GPT-5.4 Mini Disagreement Rate (per 1k Conversations): 42.2. Result ID: 6a1540d7fbc373592c0bffba. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Harassment","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### GPT-5.4 Disagreement Rate (per 1k Conversations): 47.2. Result ID: 6a1540d7fbc373592c0bffb4. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Harassment","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. ### GPT-5.4 Nano Disagreement Rate (per 1k Conversations): 66.2. Result ID: 6a1540d7fbc373592c0bffc0. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Harassment","Evaluation Protocol":"Definition"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.