Safety Alignment Verification on AdvBench 50 harmful queries
100Success RatePrecisionDiff
Evaluation results
| Method | Links | ||
|---|---|---|---|
| PrecisionDiffModel=Guanaco 7B, Precision Transition=INT16 vs INT82026.04 | 100 | 23.2 | |
| PrecisionDiffModel=Mistral-7B, Precision Transition=INT16 vs INT82026.04 | 100 | 39.5 | |
| PrecisionDiffModel=Vicuna-7B, Precision Transition=INT16 vs INT82026.04 | 100 | 10.8 | |
| PrecisionDiff (Average)Precision Transition=INT16 vs INT82026.04 | 99.5 | 20.7 | |
| PrecisionDiffModel=Llama-2 7B, Precision Transition=INT16 vs INT82026.04 | 98 | 9.3 | |
| PrecisionDiffModel=Mistral-7B, Precision Transition=BF16 vs FP162026.04 | 96 | 75.2 | |
| PrecisionDiffModel=Mistral-7B, Precision Transition=BF16 vs INT162026.04 | 94 | 79.1 | |
| PrecisionDiffModel=Vicuna-7B, Precision Transition=BF16 vs FP162026.04 | 94 | 48.9 | |
| PrecisionDiffModel=Vicuna-7B, Precision Transition=BF16 vs INT162026.04 | 91.5 | 14.8 | |
| PrecisionDiffModel=Guanaco 7B, Precision Transition=BF16 vs INT162026.04 | 84 | 78.2 | |
| PrecisionDiff (Average)Precision Transition=BF16 vs FP162026.04 | 84 | 63.2 | |
| PrecisionDiff (Average)Precision Transition=BF16 vs INT162026.04 | 83.4 | 49 | |
| PrecisionDiffModel=Guanaco 7B, Precision Transition=BF16 vs FP162026.04 | 78 | 83.8 | |
| PrecisionDiffModel=Llama-2 7B, Precision Transition=BF16 vs FP162026.04 | 68 | 44.7 | |
| PrecisionDiffModel=Llama-2 7B, Precision Transition=BF16 vs INT162026.04 | 64 | 24 |