Zero-shot Reasoning on ARC-e, ARC-c, BoolQ, Hellaswag, WinoGrande, and MMLU
77.1ARC-e Accuracy (Zero-shot)16-bit baseline
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| 16-bit baselineModel=Llama-3-8B, Precision=16-bit, zero-shot=true2026.06 | 77.1 | 53.2 | 81.1 | 79.2 | 73.4 | 56.7 | 70.1 | |
| TOGA-qModel=Llama-3-8B, Quantization=Mixed-precision (12.5% INT8, remainder INT4), KV cache=INT4, zero-shot=true2026.06 | 76.3 | 47.1 | 76.9 | 76.8 | 70.1 | 56.3 | 67.2 | |
| AtomModel=Llama-3-8B, Quantization=Mixed-precision (12.5% INT8, remainder INT4), KV cache=INT4, zero-shot=true2026.06 | 75.9 | 47.4 | 73.5 | 75.5 | 68.6 | 54 | 65.8 | |
| SpinQuantModel=Llama-3-8B, Quantization=Uniform W4A4, KV cache=INT4, zero-shot=true2026.06 | 75.4 | 48 | 75.8 | 75.4 | 69.2 | 51.2 | 65.8 | |
| ResQModel=Llama-3-8B, Quantization=Mixed-precision (12.5% INT8, remainder INT4), KV cache=INT4, zero-shot=true2026.06 | 75 | 49.2 | 72.5 | 76.5 | 71 | 56.6 | 66.8 | |
| 16-bit baselineModel=Llama-2-7B, Precision=16-bit, zero-shot=true2026.06 | 74.6 | 46.3 | 77.8 | 75.9 | 69.1 | 39.5 | 63.9 | |
| TOGA-qModel=Llama-2-7B, Quantization=Mixed-precision (12.5% INT8, remainder INT4), KV cache=INT4, zero-shot=true2026.06 | 74.5 | 43.5 | 75.6 | 74.3 | 69.3 | 37.6 | 62.5 | |
| AtomModel=Llama-2-7B, Quantization=Mixed-precision (12.5% INT8, remainder INT4), KV cache=INT4, zero-shot=true2026.06 | 74 | 42.7 | 74.9 | 73.5 | 66.9 | 34.8 | 61.1 | |
| ResQModel=Llama-2-7B, Quantization=Mixed-precision (12.5% INT8, remainder INT4), KV cache=INT4, zero-shot=true2026.06 | 72 | 43.3 | 75.9 | 73.9 | 66.8 | 37.3 | 61.5 | |
| 16-bit baselineModel=Llama-3.2-3B, Precision=16-bit, zero-shot=true2026.06 | 71.7 | 46.2 | 73.1 | 73.7 | 69.1 | 49.5 | 63.9 | |
| SpinQuantModel=Llama-2-7B, Quantization=Uniform W4A4, KV cache=INT4, zero-shot=true2026.06 | 71.3 | 43.6 | 73.8 | 73.2 | 65.4 | 33.5 | 60.1 | |
| TOGA-qModel=Llama-3.2-3B, Quantization=Mixed-precision (12.5% INT8, remainder INT4), KV cache=INT4, zero-shot=true2026.06 | 70.5 | 41.1 | 72.6 | 70.2 | 63.2 | 46.5 | 60.7 | |
| AtomModel=Llama-3.2-3B, Quantization=Mixed-precision (12.5% INT8, remainder INT4), KV cache=INT4, zero-shot=true2026.06 | 69 | 40 | 64.2 | 69.1 | 62.4 | 44 | 58.1 | |
| ResQModel=Llama-3.2-3B, Quantization=Mixed-precision (12.5% INT8, remainder INT4), KV cache=INT4, zero-shot=true2026.06 | 65.6 | 43.1 | 68.8 | 70.5 | 64.8 | 47.8 | 60.1 | |
| SpinQuantModel=Llama-3.2-3B, Quantization=Uniform W4A4, KV cache=INT4, zero-shot=true2026.06 | 64.8 | 38.9 | 68 | 69.1 | 62.9 | 37.2 | 56.8 | |
| 16-bit baselineModel=Llama-3.2-1B, Precision=16-bit, zero-shot=true2026.06 | 60.6 | 36.5 | 63.4 | 63.6 | 60.1 | 32.8 | 52.8 | |
| TOGA-qModel=Llama-3.2-1B, Quantization=Mixed-precision (12.5% INT8, remainder INT4), KV cache=INT4, zero-shot=true2026.06 | 59.7 | 33.8 | 59.9 | 57.7 | 57.4 | 26.8 | 49.2 | |
| AtomModel=Llama-3.2-1B, Quantization=Mixed-precision (12.5% INT8, remainder INT4), KV cache=INT4, zero-shot=true2026.06 | 58.8 | 33.7 | 58.9 | 56.7 | 53 | 28.6 | 48.3 | |
| ResQModel=Llama-3.2-1B, Quantization=Mixed-precision (12.5% INT8, remainder INT4), KV cache=INT4, zero-shot=true2026.06 | 56.6 | 33.6 | 58.9 | 58.2 | 55.3 | 26.6 | 48.2 | |
| SpinQuantModel=Llama-3.2-1B, Quantization=Uniform W4A4, KV cache=INT4, zero-shot=true2026.06 | 51.8 | 32.3 | 59.3 | 55.4 | 54.7 | 24.9 | 46.4 |