Truthful Question Answering on TruthfulQA MC2 (MC2 %)
56.46MC2 AccuracyHRTN
Evaluation results
| Method | Links | ||
|---|---|---|---|
| HRTNAvg. Bitwidth=4.125, Backbone=Qwen3-8B2025.07 | 56.46 | — | |
| SSQR-1%Avg. Bitwidth=3.445, Backbone=Qwen3-8B2025.07 | 55.95 | — | |
| SSQR-1%Avg. Bitwidth=4.445, Backbone=Qwen3-8B2025.07 | 55.81 | — | |
| RTNAvg. Bitwidth=4.125, Backbone=Qwen3-8B2025.07 | 55.77 | — | |
| SSQR-3%Avg. Bitwidth=4.085, Backbone=Qwen3-8B2025.07 | 55.34 | — | |
| GPTQAvg. Bitwidth=3.125, Backbone=Qwen3-8B2025.07 | 55.21 | — | |
| SSQR-4%Avg. Bitwidth=4.405, Backbone=Qwen3-8B2025.07 | 54.81 | — | |
| HPTQAvg. Bitwidth=3.125, Backbone=Qwen3-8B2025.07 | 54.73 | — | |
| GPTQAvg. Bitwidth=4.125, Backbone=Qwen3-8B2025.07 | 54.55 | — | |
| BF16 BaselineAvg. Bitwidth=16.000, Backbone=Qwen3-8B2025.07 | 54.5 | — | |
| SSQR-3%Avg. Bitwidth=5.085, Backbone=Qwen3-8B2025.07 | 54.46 | — | |
| SSQR-5%Avg. Bitwidth=5.725, Backbone=Qwen3-8B2025.07 | 53.76 | — | |
| HRTNAvg. Bitwidth=3.125, Backbone=Qwen3-8B2025.07 | 53.68 | — | |
| SSQR-4%Avg. Bitwidth=5.405, Backbone=Qwen3-8B2025.07 | 53.6 | — | |
| SSQR-2%Avg. Bitwidth=4.765, Backbone=Qwen3-8B2025.07 | 53.47 | — | |
| HPTQAvg. Bitwidth=4.125, Backbone=Qwen3-8B2025.07 | 53.36 | — | |
| SSQR-2%Avg. Bitwidth=3.765, Backbone=Qwen3-8B2025.07 | 53.05 | — | |
| SSQR-5%Avg. Bitwidth=4.725, Backbone=Qwen3-8B2025.07 | 52.91 | — | |
| RTNAvg. Bitwidth=3.125, Backbone=Qwen3-8B2025.07 | 52.76 | — | |
| NGMModel=Qwen3-VL-2B2026.05 | 52.63 | — | |
| Qwen3-VL-2BModel=Qwen3-VL-2B2026.05 | 52.26 | — | |
| HRTNAvg. Bitwidth=2.125, Backbone=Qwen3-8B2025.07 | 51.78 | — | |
| SSQR-4%Avg. Bitwidth=3.405, Backbone=Qwen3-8B2025.07 | 50.46 | — | |
| SSQR-5%Avg. Bitwidth=3.725, Backbone=Qwen3-8B2025.07 | 50.25 | — | |
| HPTQAvg. Bitwidth=2.125, Backbone=Qwen3-8B2025.07 | 49.01 | — | |
| FPModel=LLaDA-1.5, Quantization=Full Precision2026.06 | 48.76 | — | |
| SSQR-3%Avg. Bitwidth=3.085, Backbone=Qwen3-8B2025.07 | 48.53 | — | |
| FPModel=LLaDA-Instruct, Quantization=Full Precision2026.06 | 48.47 | — | |
| FAIR-CalibModel=LLaDA-1.5, Quantization=W4A42026.06 | 48.01 | — | |
| QuaRotModel=LLaDA-Instruct, Quantization=W4A42026.06 | 47.77 | — | |
| QuaRotModel=LLaDA-1.5, Quantization=W4A42026.06 | 47.46 | — | |
| FAIR-CalibModel=LLaDA-Instruct, Quantization=W4A42026.06 | 47.44 | — | |
| RTNAvg. Bitwidth=2.125, Backbone=Qwen3-8B2025.07 | 47.33 | — | |
| Prompt EngBackbone=Gemma-2-9B2026.03 | 47.2 | — | |
| GPTQAvg. Bitwidth=2.125, Backbone=Qwen3-8B2025.07 | 46.91 | — | |
| FlatQuantModel=LLaDA-1.5, Quantization=W4A42026.06 | 46.48 | — | |
| FlatQuantModel=LLaDA-Instruct, Quantization=W4A42026.06 | 46.39 | — | |
| RTNModel=LLaDA-1.5, Quantization=W4A42026.06 | 46.39 | — | |
| DPOBackbone=Qwen3-8B2026.03 | 45.9 | — | |
| SSQR-2%Avg. Bitwidth=2.765, Backbone=Qwen3-8B2025.07 | 45.54 | — | |
| RepEBackbone=Gemma-2-9B2026.03 | 45.5 | — | |
| Base ModelBackbone=Gemma-2-9B2026.03 | 45.5 | — | |
| DSPABackbone=Gemma-2-9B2026.03 | 45.4 | — | |
| FPModel=LLaDA-Base, Quantization=Full Precision2026.06 | 45.3 | — | |
| RTNModel=LLaDA-Instruct, Quantization=W4A42026.06 | 45.17 | — | |
| DPOBackbone=Gemma-2-9B2026.03 | 44.1 | — | |
| Static-SAEBackbone=Gemma-2-9B2026.03 | 44.1 | — | |
| FAIR-CalibModel=LLaDA-Base, Quantization=W4A42026.06 | 43.91 | — | |
| SSQR-1%Avg. Bitwidth=2.445, Backbone=Qwen3-8B2025.07 | 43.88 | — | |
| QuaRotModel=LLaDA-Base, Quantization=W4A42026.06 | 43.35 | — | |
| FlatQuantModel=LLaDA-Base, Quantization=W4A42026.06 | 42.9 | — | |
| Prompt EngBackbone=Qwen3-8B2026.03 | 42.6 | — | |
| DSPABackbone=Qwen3-8B2026.03 | 42.4 | — | |
| Static-SAEBackbone=Qwen3-8B2026.03 | 42.4 | — | |
| Prompt EngBackbone=Gemma-2-2B2026.03 | 41 | — | |
| RepEBackbone=Qwen3-8B2026.03 | 41 | — | |
| Base ModelBackbone=Qwen3-8B2026.03 | 40.9 | — | |
| Dense 0.7BArchitecture=Dense, Total Parameters=0.7B, Precision=bf16, Shot count=0-shot, Evaluation Framework=lm-eval-harness 0.4.72026.05 | 40.89 | — | |
| DSPABackbone=Gemma-2-2B2026.03 | 40.4 | — | |
| RepEBackbone=Gemma-2-2B2026.03 | 40.3 | — | |
| Base ModelBackbone=Gemma-2-2B2026.03 | 40.2 | — | |
| DPOBackbone=Gemma-2-2B2026.03 | 39.8 | — | |
| MoL 0.61B/2.08BArchitecture=MoL, Total Parameters=2.08B, Active Parameters=0.61B, Precision=bf16, Shot count=0-shot, Evaluation Framework=lm-eval-harness 0.4.72026.05 | 39.55 | — | |
| Static-SAEBackbone=Gemma-2-2B2026.03 | 39.5 | — | |
| RTNModel=LLaDA-Base, Quantization=W4A42026.06 | 38.8 | — | |
| Dense 1.3BArchitecture=Dense, Total Parameters=1.3B, Precision=bf16, Shot count=0-shot, Evaluation Framework=lm-eval-harness 0.4.72026.05 | 37.83 | — |