Truthful Question Answering on TruthfulQA (TQA acc.)
50.4TQA AccuracyQuantGuard
Evaluation results
| Method | Links | |
|---|---|---|
| QuantGuardModel=Phi-2-2.7B, Precision=FP42026.06 | 50.4 | |
| QCB attackModel=Phi-2-2.7B, Precision=FP42026.06 | 49.9 | |
| QCB attackModel=Phi-2-2.7B, Precision=NF42026.06 | 49.3 | |
| QuantGuardModel=Phi-2-2.7B, Precision=NF42026.06 | 47.6 | |
| QCB attackModel=Phi-2-2.7B, Precision=INT82026.06 | 47.5 | |
| QuantGuardModel=Phi-2-2.7B, Precision=INT82026.06 | 47.3 | |
| BaselineModel=LLaMA3-8B, Precision=INT82026.06 | 43.2 | |
| BaselineModel=Phi-2-2.7B, Precision=NF42026.06 | 41.5 | |
| BaselineModel=Phi-2-2.7B, Precision=INT82026.06 | 41.3 | |
| BaselineModel=Phi-2-2.7B, Precision=FP42026.06 | 39.9 | |
| BaselineModel=LLaMA3-8B, Precision=FP42026.06 | 39.9 | |
| BaselineModel=LLaMA3-8B, Precision=NF42026.06 | 39.8 | |
| QCB attackModel=LLaMA3-8B, Precision=INT82026.06 | 38.4 | |
| QCB attackModel=LLaMA3-8B, Precision=NF42026.06 | 37.2 | |
| QuantGuardModel=LLaMA3-8B, Precision=INT82026.06 | 37 | |
| QuantGuardModel=LLaMA3-8B, Precision=NF42026.06 | 34.2 | |
| QCB attackModel=LLaMA3-8B, Precision=FP42026.06 | 33.5 | |
| QuantGuardModel=LLaMA3-8B, Precision=FP42026.06 | 31.7 | |
| QCB attackModel=Gemma-2B, Precision=NF42026.06 | 21.2 | |
| QCB attackModel=Gemma-2B, Precision=FP42026.06 | 20.9 | |
| QCB attackModel=Gemma-2B, Precision=INT82026.06 | 20.5 | |
| BaselineModel=Gemma-2B, Precision=INT82026.06 | 20.4 | |
| BaselineModel=Gemma-2B, Precision=NF42026.06 | 20.1 | |
| QuantGuardModel=Gemma-2B, Precision=INT82026.06 | 20 | |
| QuantGuardModel=Gemma-2B, Precision=FP42026.06 | 19.7 | |
| BaselineModel=Gemma-2B, Precision=FP42026.06 | 19.5 | |
| QuantGuardModel=Gemma-2B, Precision=NF42026.06 | 19.2 |