Multiple Choice Question Answering on ARC Challenge (Non-generative Acc)
30Non-generative AccuracyNoPE
Evaluation Results
| Method | Links | |
|---|---|---|
| NoPEModel Architecture=GLA, Parameters=1.3B, Training Tokens=26B2025.11 | 30 | |
| Selective RoPEModel Architecture=GLA, Parameters=1.3B, Training Tokens=26B2025.11 | 28.8 | |
| RoPEModel Architecture=GLA, Parameters=1.3B, Training Tokens=26B2025.11 | 28.5 | |
| Selective RoPEModel Architecture=GLA, Parameters=370M, Training Tokens=10B2025.11 | 26.6 | |
| Selective RoPEModel Architecture=FoX, Parameters=370M, Training Tokens=10B, Output-norm setting=OFF2025.11 | 25.7 | |
| Selective RoPEModel Architecture=Gated DeltaNet, Parameters=~400M, Training Tokens=10B, Variant=stable_selective_rope2025.11 | 25.5 | |
| NoPEModel Architecture=FoX, Parameters=370M, Training Tokens=10B, Output-norm setting=OFF2025.11 | 25 | |
| NoPEModel Architecture=GLA, Parameters=370M, Training Tokens=10B2025.11 | 24.9 | |
| RoPEModel Architecture=FoX, Parameters=370M, Training Tokens=10B, Output-norm setting=OFF2025.11 | 24.6 | |
| RoPEModel Architecture=GLA, Parameters=370M, Training Tokens=10B2025.11 | 24.5 | |
| NoPEModel Architecture=Gated DeltaNet, Parameters=~400M, Training Tokens=10B2025.11 | 24.2 | |
| RoPEModel Architecture=Gated DeltaNet, Parameters=~400M, Training Tokens=10B2025.11 | 24.1 | |
| BF16Model=Llama-3 70B, Setting=W16A162025.12 | 0.6451 | |
| SpQRModel=Llama-3 70B, Setting=W4A162025.12 | 0.6399 | |
| SQ-formatModel=Llama-3 70B, Setting=W4A(SQ6)2025.12 | 0.628 | |
| GPTQModel=Llama-3 70B, Setting=W4A82025.12 | 0.6254 | |
| SpinQuantModel=Llama-3 70B, Setting=W4A82025.12 | 0.6203 | |
| SQ-formatModel=Llama-3 70B, Setting=W4A(SQ5)2025.12 | 0.6195 | |
| SQ-formatModel=Llama-3 70B, Setting=W(SQ6)A42025.12 | 0.6101 | |
| SQ-formatModel=Llama-3 70B, Setting=W(SQ5)A42025.12 | 0.5998 | |
| GPTQModel=Llama-3 70B, Setting=W4A42025.12 | 0.5989 | |
| SQ-formatModel=Llama-3 70B, Setting=W(SQ4.5)A42025.12 | 0.5964 | |
| BF16Model=Qwen-3 30B-A3B, Setting=W16A162025.12 | 0.558 | |
| SQ-formatModel=Qwen-3 30B-A3B, Setting=W4A(SQ5)2025.12 | 0.5546 | |
| SQ-formatModel=Qwen-3 30B-A3B, Setting=W4A(SQ6)2025.12 | 0.5538 | |
| SQ-formatModel=Qwen-3 30B-A3B, Setting=W(SQ6)A42025.12 | 0.5375 | |
| BF16Model=Llama-3 8B, Setting=W16A162025.12 | 0.5341 | |
| GPTQModel=Llama-3 8B, Setting=W4A82025.12 | 0.5315 | |
| GPTQModel=Qwen-3 30B-A3B, Setting=W4A82025.12 | 0.5273 | |
| SpQRModel=Llama-3 8B, Setting=W4A162025.12 | 0.5213 | |
| SQ-formatModel=Llama-3 8B, Setting=W4A(SQ6)2025.12 | 0.5213 | |
| SQ-formatModel=Llama-3 8B, Setting=W4A(SQ5)2025.12 | 0.5196 | |
| SpinQuantModel=Llama-3 8B, Setting=W4A82025.12 | 0.5178 | |
| GPTQModel=Qwen-3 30B-A3B, Setting=W4A42025.12 | 0.4948 | |
| SQ-formatModel=Llama-3 8B, Setting=W(SQ6)A42025.12 | 0.494 | |
| SparseGPTModel=Llama-3 70B, Setting=W16A162025.12 | 0.4872 | |
| SpinQuantModel=Llama-3 8B, Setting=W4A42025.12 | 0.4795 | |
| SQ-formatModel=Llama-3 8B, Setting=W(SQ4.5)A42025.12 | 0.4761 | |
| SQ-formatModel=Llama-3 8B, Setting=W(SQ5)A42025.12 | 0.4744 | |
| GPTQModel=Llama-3 8B, Setting=W4A42025.12 | 0.4692 | |
| SmoothQuantModel=Qwen-3 30B-A3B, Setting=W4A82025.12 | 0.4035 | |
| SpinQuantModel=Llama-3 70B, Setting=W4A42025.12 | 0.3797 | |
| SmoothQuantModel=Llama-3 8B, Setting=W4A82025.12 | 0.372 | |
| SparseGPTModel=Llama-3 8B, Setting=W16A162025.12 | 0.3174 | |
| SmoothQuantModel=Llama-3 70B, Setting=W4A42025.12 | 0.2521 | |
| SmoothQuantModel=Llama-3 70B, Setting=W4A82025.12 | 0.244 | |
| SmoothQuantModel=Llama-3 8B, Setting=W4A42025.12 | 0.2423 | |
| SmoothQuantModel=Qwen-3 30B-A3B, Setting=W4A42025.12 | 0.232 |