# Safety Evaluation on AdvBench N=520 Source: [Safety Evaluation on AdvBench N=520]() Collection index: [llms.txt]() Data updated: 2026-06-11T06:44:56.279Z Evaluation of model safety performance on the specific AdvBench subset containing 520 examples, measured using Flip Rate and Recovery metrics. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Safety Evaluation | | Dataset (including any stated split) | AdvBench N=520 | | Metric | Flip Rate | | Unit | PERCENT | | Direction | Lower is better | | Results loaded | 16 of 16 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Flip Rate](): Lower is better; unit: PERCENT. - [Recovery](): Higher is better; unit: PERCENT. ## Recorded results | Method | Flip Rate | Result date | Report paper | | --- | --- | --- | --- | | LLaMA-3.1-8B | 0.2 | Not recorded | 6a28f33c90966ade919c16ea | | LLaMA-3.1-8B | 0.8 | Not recorded | 6a28f33c90966ade919c16ea | | LLaMA-3.1-8B | 1.2 | Not recorded | 6a28f33c90966ade919c16ea | | LLaMA-3.1-8B | 1.5 | Not recorded | 6a28f33c90966ade919c16ea | | Qwen-2.5-7B | 1.9 | Not recorded | 6a28f33c90966ade919c16ea | | Gemma-2-9B | 2.7 | Not recorded | 6a28f33c90966ade919c16ea | | Mistral-7B | 7.3 | Not recorded | 6a28f33c90966ade919c16ea | | Mistral-7B | 9.8 | Not recorded | 6a28f33c90966ade919c16ea | | Mistral-7B | 10.4 | Not recorded | 6a28f33c90966ade919c16ea | | Mistral-7B | 15.2 | Not recorded | 6a28f33c90966ade919c16ea | | Gemma-2-9B | 79.2 | Not recorded | 6a28f33c90966ade919c16ea | | Gemma-2-9B | 80.8 | Not recorded | 6a28f33c90966ade919c16ea | | Qwen-2.5-7B | 88.2 | Not recorded | 6a28f33c90966ade919c16ea | | Qwen-2.5-7B | 88.9 | Not recorded | 6a28f33c90966ade919c16ea | | Qwen-2.5-7B | 90.3 | Not recorded | 6a28f33c90966ade919c16ea | | Gemma-2-9B | 92 | Not recorded | 6a28f33c90966ade919c16ea | ### LLaMA-3.1-8B Flip Rate: 0.2. Result ID: 6a2a58c2fbc373592c26c935. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Bits":"4","Mitigation Strategy":"Group-64"}. This record does not assert independent reproduction. ### LLaMA-3.1-8B Flip Rate: 0.8. Result ID: 6a2a58c2fbc373592c26c932. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Bits":"4","Mitigation Strategy":"Unprotected"}. This record does not assert independent reproduction. ### LLaMA-3.1-8B Flip Rate: 1.2. Result ID: 6a2a58c2fbc373592c26c933. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Bits":"4","Mitigation Strategy":"FP16 L0-1"}. This record does not assert independent reproduction. ### LLaMA-3.1-8B Flip Rate: 1.5. Result ID: 6a2a58c2fbc373592c26c934. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Bits":"4","Mitigation Strategy":"Attn top-2","Selected Layers":"L2,L18"}. This record does not assert independent reproduction. ### Qwen-2.5-7B Flip Rate: 1.9. Result ID: 6a2a58c2fbc373592c26c92b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Bits":"4","Mitigation Strategy":"FP16 L0-1"}. This record does not assert independent reproduction. ### Gemma-2-9B Flip Rate: 2.7. Result ID: 6a2a58c2fbc373592c26c939. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Bits":"2","Mitigation Strategy":"Group-64"}. This record does not assert independent reproduction. ### Mistral-7B Flip Rate: 7.3. Result ID: 6a2a58c2fbc373592c26c931. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Bits":"4","Mitigation Strategy":"Group-64"}. This record does not assert independent reproduction. ### Mistral-7B Flip Rate: 9.8. Result ID: 6a2a58c2fbc373592c26c930. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Bits":"4","Mitigation Strategy":"Attn top-2","Selected Layers":"L1,L2"}. This record does not assert independent reproduction. ### Mistral-7B Flip Rate: 10.4. Result ID: 6a2a58c2fbc373592c26c92f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Bits":"4","Mitigation Strategy":"FP16 L0-1"}. This record does not assert independent reproduction. ### Mistral-7B Flip Rate: 15.2. Result ID: 6a2a58c2fbc373592c26c92e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Bits":"4","Mitigation Strategy":"Unprotected"}. This record does not assert independent reproduction. ### Gemma-2-9B Flip Rate: 79.2. Result ID: 6a2a58c2fbc373592c26c937. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Bits":"2","Mitigation Strategy":"FP16 L0-1"}. This record does not assert independent reproduction. ### Gemma-2-9B Flip Rate: 80.8. Result ID: 6a2a58c2fbc373592c26c938. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Bits":"2","Mitigation Strategy":"Attn top-2","Selected Layers":"L1,L2"}. This record does not assert independent reproduction. ### Qwen-2.5-7B Flip Rate: 88.2. Result ID: 6a2a58c2fbc373592c26c92d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Bits":"4","Mitigation Strategy":"Group-64"}. This record does not assert independent reproduction. ### Qwen-2.5-7B Flip Rate: 88.9. Result ID: 6a2a58c2fbc373592c26c92c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Bits":"4","Mitigation Strategy":"Attn top-2","Selected Layers":"L5,L22"}. This record does not assert independent reproduction. ### Qwen-2.5-7B Flip Rate: 90.3. Result ID: 6a2a58c2fbc373592c26c92a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Bits":"4","Mitigation Strategy":"Unprotected"}. This record does not assert independent reproduction. ### Gemma-2-9B Flip Rate: 92. Result ID: 6a2a58c2fbc373592c26c936. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Bits":"2","Mitigation Strategy":"Unprotected"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.