Common-sense QA on PIQA
79AccuracyFP16 (no compression)
Evaluation results
| Method | Links | ||
|---|---|---|---|
| FP16 (no compression)error budget (epsilon)=N/A, KV cache (rel.)=1, Zero-shot protocol=true, Base Model=Llama-3-8B2026.01 | 79 | — | |
| StiefAttentionerror budget (epsilon)=0.015, KV cache (rel.)=0.85, Zero-shot protocol=true, Base Model=Llama-3-8B2026.01 | 79 | — | |
| FP16Backbone=LLaMA2-7B, Avg. Bit=16, Evaluation Protocol=zero-shot2024.10 | 78.07 | — | |
| StiefAttentionerror budget (epsilon)=0.03, KV cache (rel.)=0.76, Zero-shot protocol=true, Base Model=Llama-3-8B2026.01 | 78 | — | |
| CMPQBackbone=LLaMA2-7B, Avg. Bit=4, Evaluation Protocol=zero-shot2024.10 | 77.93 | — | |
| QuIPBackbone=LLaMA2-7B, Avg. Bit=4, Evaluation Protocol=zero-shot2024.10 | 77.73 | — | |
| RTNBackbone=LLaMA2-7B, Avg. Bit=4, Evaluation Protocol=zero-shot2024.10 | 77.58 | — | |
| GPTQBackbone=LLaMA2-7B, Avg. Bit=4, Evaluation Protocol=zero-shot2024.10 | 77.48 | — | |
| AWQBackbone=LLaMA2-7B, Avg. Bit=4, Evaluation Protocol=zero-shot2024.10 | 77.31 | — | |
| OPT-2.7B FP16Model=OPT-2.7B, Precision=FP162023.08 | 76.71 | 10.91 | |
| StiefAttentionerror budget (epsilon)=0.045, KV cache (rel.)=0.67, Zero-shot protocol=true, Base Model=Llama-3-8B2026.01 | 76 | — | |
| CMPQBackbone=LLaMA2-7B, Avg. Bit=3, Evaluation Protocol=zero-shot2024.10 | 75.7 | — | |
| TSLDModel=OPT-2.7B, Quantization=QAT-KD (tensor-wise)2023.08 | 75.62 | 11.35 | |
| QuIPBackbone=LLaMA2-7B, Avg. Bit=3, Evaluation Protocol=zero-shot2024.10 | 75.12 | — | |
| StiefAttentionerror budget (epsilon)=0.06, KV cache (rel.)=0.61, Zero-shot protocol=true, Base Model=Llama-3-8B2026.01 | 75 | — | |
| GT+LogitModel=OPT-2.7B, Quantization=QAT-KD (tensor-wise)2023.08 | 74.97 | 12.1 | |
| LogitModel=OPT-2.7B, Quantization=QAT-KD (tensor-wise)2023.08 | 74.32 | 11.69 | |
| StiefAttentionerror budget (epsilon)=0.09, KV cache (rel.)=0.57, Zero-shot protocol=true, Base Model=Llama-3-8B2026.01 | 74 | — | |
| RTNBackbone=LLaMA2-7B, Avg. Bit=3, Evaluation Protocol=zero-shot2024.10 | 73.28 | — | |
| GPTQBackbone=LLaMA2-7B, Avg. Bit=3, Evaluation Protocol=zero-shot2024.10 | 73.23 | — |