Reading Comprehension on BoolQ (Accuracy)
88.47Accuracy (BoolQ)FP
Evaluation results
| Method | Links | |
|---|---|---|
| FPModel=LLaDA-1.5, Quantization=Full Precision2026.06 | 88.47 | |
| FPModel=LLaDA-Instruct, Quantization=Full Precision2026.06 | 88.38 | |
| FAIR-CalibModel=LLaDA-Instruct, Quantization=W4A42026.06 | 88.29 | |
| BEAMSparsity Level=Mid Sparsity, beta=0.012026.05 | 88.07 | |
| QuaRotModel=LLaDA-Instruct, Quantization=W4A42026.06 | 87.98 | |
| FlatQuantModel=LLaDA-Instruct, Quantization=W4A42026.06 | 87.98 | |
| FAIR-CalibModel=LLaDA-1.5, Quantization=W4A42026.06 | 87.89 | |
| Top-K ReducedSparsity Level=Mid Sparsity, K=42026.05 | 87.68 | |
| MoE-DynamicSparsity Level=Mid Sparsity, phi=0.32026.05 | 87.22 | |
| QuaRotModel=LLaDA-1.5, Quantization=W4A42026.06 | 87.07 | |
| FlatQuantModel=LLaDA-1.5, Quantization=W4A42026.06 | 86.92 | |
| Qwen3-30B-A3BSparsity Level=Base, K=82026.05 | 86.76 | |
| Full-precisionModel=Mixtral 8x22B, Avg. bits/exp.=16 (FP), Memory (GB)=281.2, Evaluation protocol=zero-shot2026.04 | 86.23 | |
| HRM-Text 1BArchitecture=Recurrent, FLOPs ($10^{21}$)=1, Tokens (T)=0.062026.05 | 86.2 | |
| UniformModel=Mixtral 8x7B, Avg. bits/exp.=3, Memory (GB)=19.3, Evaluation protocol=zero-shot2026.04 | 86.09 | |
| RTNModel=LLaDA-1.5, Quantization=W4A42026.06 | 85.99 | |
| BF16Model=Mixtral-8x7B, Precision=BF16, Evaluation Protocol=Zero-shot2026.05 | 85.9 | |
| BEAMSparsity Level=High Sparsity, beta=0.12026.05 | 85.75 | |
| Router norm + Max varModel=Mixtral 8x7B, Avg. bits/exp.=2.625, Memory (GB)=16.9, Evaluation protocol=zero-shot2026.04 | 85.6 | |
| Router norm + Max varModel=Mixtral 8x7B, Avg. bits/exp.=2.75, Memory (GB)=17.7, Evaluation protocol=zero-shot2026.04 | 85.57 | |
| Olmo3 7BArchitecture=Dense, FLOPs ($10^{21}$)=252, Tokens (T)=62026.05 | 85.4 | |
| Router norm + Max varModel=Mixtral 8x7B, Avg. bits/exp.=2.5, Memory (GB)=16.1, Evaluation protocol=zero-shot2026.04 | 85.14 | |
| Full-precisionModel=Mixtral 8x7B, Avg. bits/exp.=16 (FP), Memory (GB)=96.8, Evaluation protocol=zero-shot2026.04 | 85.05 | |
| Router norm + Max varModel=Mixtral 8x7B, Avg. bits/exp.=2.375, Memory (GB)=15.3, Evaluation protocol=zero-shot2026.04 | 84.92 | |
| PMQModel=Mixtral 8x7B, Avg. bits/exp.=2.75, Memory (GB)=17.7, Evaluation protocol=zero-shot2026.04 | 84.8 | |
| AdaMoESparsity Level=Mid Sparsity, Null=1282026.05 | 84.71 | |
| RTNModel=LLaDA-Instruct, Quantization=W4A42026.06 | 84.68 | |
| Router norm + Max varModel=Mixtral 8x7B, Avg. bits/exp.=2.25, Memory (GB)=14.5, Evaluation protocol=zero-shot2026.04 | 84.04 | |
| Mistral-7BRatio=0%, Evaluation Protocol=zero-shot2026.03 | 83.98 | |
| Top-K ReducedSparsity Level=High Sparsity, K=22026.05 | 83.85 | |
| PMQModel=Mixtral 8x7B, Avg. bits/exp.=2.625, Memory (GB)=16.9, Evaluation protocol=zero-shot2026.04 | 83.67 | |
| Ouro 1.4BArchitecture=Recurrent, FLOPs ($10^{21}$)=259, Tokens (T)=72026.05 | 83.6 | |
| PMQModel=Mixtral 8x7B, Avg. bits/exp.=2.5, Memory (GB)=16.1, Evaluation protocol=zero-shot2026.04 | 83.15 | |
| PMQModel=Mixtral 8x7B, Avg. bits/exp.=2.25, Memory (GB)=14.5, Evaluation protocol=zero-shot2026.04 | 83.15 | |
| SARQC-GBS(Saliency)Model=Mixtral-8x7B, Precision=W3A16, Evaluation Protocol=Zero-shot2026.05 | 82.6 | |
| SARQC-GBS(Identity)Model=Mixtral-8x7B, Precision=W3A16, Evaluation Protocol=Zero-shot2026.05 | 82.35 | |
| FP16Model=LLaMA2-13B, Precision=FP16, Evaluation Protocol=Zero-shot2026.05 | 82.11 | |
| GPTQModel=Mixtral-8x7B, Precision=W3A16, Evaluation Protocol=Zero-shot2026.05 | 82.11 | |
| OriginalBase Model=LLaMA3-8B2026.05 | 82.02 | |
| OriginalBackbone=LLaMA3-8B, Pruning Ratio=0%2026.05 | 82.02 | |
| PMQModel=Mixtral 8x7B, Avg. bits/exp.=2.375, Memory (GB)=15.3, Evaluation protocol=zero-shot2026.04 | 81.93 | |
| SARQC-GBS(Saliency)Model=Qwen3-MoE-30B, Precision=W3A16, Evaluation Protocol=Zero-shot2026.05 | 81.9 | |
| GPTQModel=Qwen3-MoE-30B, Precision=W3A16, Evaluation Protocol=Zero-shot2026.05 | 81.25 | |
| HessianModel=Mixtral 8x7B, Avg. bits/exp.=2.5, Memory (GB)=17.0, Evaluation protocol=zero-shot2026.04 | 81.11 | |
| MoE-DynamicSparsity Level=High Sparsity, phi=0.12026.05 | 81.07 | |
| Top-K PruningSparsity Level=Mid Sparsity, K=42026.05 | 81.04 | |
| Router norm + Max varModel=Mixtral 8x7B, Avg. bits/exp.=2.0, Memory (GB)=13.1, Evaluation protocol=zero-shot2026.04 | 80.61 | |
| BF16Model=Qwen3-MoE-30B, Precision=BF16, Evaluation Protocol=Zero-shot2026.05 | 80.61 | |
| Qwen3.5 2BArchitecture=Dense, FLOPs ($10^{21}$)=432, Tokens (T)=362026.05 | 80.5 | |
| Router norm + Max varModel=Mixtral 8x7B, Avg. bits/exp.=2.125, Memory (GB)=13.8, Evaluation protocol=zero-shot2026.04 | 80.12 | |
| SARQC-GBS(Identity)Model=Qwen3-MoE-30B, Precision=W3A16, Evaluation Protocol=Zero-shot2026.05 | 80.12 | |
| UnprunedBackbone=Llama-3.1-8B2026.04 | 80 | |
| PMQModel=Mixtral 8x7B, Avg. bits/exp.=1.75, Memory (GB)=11.7, Evaluation protocol=zero-shot2026.04 | 79.63 | |
| FP16Model=LLaMA2-7B, Precision=FP16, Evaluation Protocol=Zero-shot2026.05 | 79.27 | |
| HessianModel=Mixtral 8x7B, Avg. bits/exp.=2.25, Memory (GB)=15.3, Evaluation protocol=zero-shot2026.04 | 79.15 | |
| CoA-LoRAModel=Llama-2-7B, Average Bit Budget=3-bit, Zero-shot=true2025.09 | 78.91 | |
| PMQModel=Mixtral 8x7B, Avg. bits/exp.=2.0, Memory (GB)=13.1, Evaluation protocol=zero-shot2026.04 | 78.65 | |
| UniformModel=Mixtral 8x22B, Avg. bits/exp.=3, Memory (GB)=57.5, Evaluation protocol=zero-shot2026.04 | 78.53 | |
| BEAMSparsity Level=Extreme Sparsity, beta=1.02026.05 | 78.47 | |
| Accuracy (Ours)Backbone=LLaMA3-8B, Pruning Ratio=25%2026.05 | 78.38 | |
| Shared-LoRAModel=Llama-2-7B, Average Bit Budget=3-bit, Zero-shot=true2025.09 | 78.12 | |
| Slice-GPTBackbone=LLaMA3-8B, Pruning Ratio=25%2026.05 | 77.49 | |
| Cosine SimilarityPruning Method=Cosine Similarity, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance2026.05 | 76.7 | |
| BEAMSparsity Level=Mid Sparsity, beta=0.01, Avg. K=2.612026.05 | 76.57 | |
| LQ-LoRAModel=Llama-2-7B, Average Bit Budget=3-bit, Zero-shot=true2025.09 | 76.56 | |
| Top-K ReducedSparsity Level=Mid Sparsity, K=4, Avg. K=4.002026.05 | 76.39 | |
| Llama3.2 3BArchitecture=Dense, FLOPs ($10^{21}$)=162, Tokens (T)=92026.05 | 76.2 | |
| SARQC-GBS(Saliency)Model=LLaMA2-13B, Precision=W3A16, Evaluation Protocol=Zero-shot2026.05 | 76.02 | |
| GPTQModel=LLaMA2-13B, Precision=W3A16, Evaluation Protocol=Zero-shot2026.05 | 75.84 | |
| SARQC-GBS(Identity)Model=LLaMA2-13B, Precision=W3A16, Evaluation Protocol=Zero-shot2026.05 | 75.78 | |
| GPTAQModel=LLaMA2-13B, Precision=W3A16, Evaluation Protocol=Zero-shot2026.05 | 75.72 | |
| Cosine SimilarityBackbone=LLaMA3-8B, Pruning Ratio=25%2026.05 | 75.53 | |
| DeepSeekV2-LiteK=6, Avg. K=6.002026.05 | 75.2 | |
| QLoRAModel=Llama-2-7B, Average Bit Budget=3-bit, Zero-shot=true2025.09 | 75 | |
| OLMoE-1B-7B-0924Model Type=MoE2026.04 | 74.5 | |
| PMQModel=Mixtral 8x22B, Avg. bits/exp.=2.5, Memory (GB)=46.7, Evaluation protocol=zero-shot2026.04 | 74.28 | |
| BF16Model=DeepSeek-MoE-16B, Precision=BF16, Evaluation Protocol=Zero-shot2026.05 | 74.07 | |
| XPERT-OLMoE#Params=570M, #Size=dim=2048 16 Layers, Pre-training Tokens=5B, Fine-tuning Epochs=32026.05 | 73.91 | |
| XPERT-OLMoE#Params=480M, #Size=dim=2048 12 Layers, Pre-training Tokens=5B, Fine-tuning Epochs=32026.05 | 73.76 | |
| Router norm + Max varModel=Mixtral 8x7B, Avg. bits/exp.=1.75, Memory (GB)=11.7, Evaluation protocol=zero-shot2026.04 | 73.64 | |
| Router norm + Max varModel=Mixtral 8x22B, Avg. bits/exp.=2.5, Memory (GB)=46.7, Evaluation protocol=zero-shot2026.04 | 73.49 | |
| XPERT-DeepSeek#Params=270M, #Size=dim=1024 16 Layers, Pre-training Tokens=5B, Fine-tuning Epochs=32026.05 | 73.24 | |
| XPERT-OLMoE#Params=270M, #Size=dim=1024 16 Layers, Pre-training Tokens=5B, Fine-tuning Epochs=32026.05 | 73.21 | |
| Mula-7B-A1BModel Type=MoE2026.04 | 73.1 | |
| SARQC-GBS(Identity)Model=LLaMA2-7B, Precision=W3A16, Evaluation Protocol=Zero-shot2026.05 | 72.94 | |
| SARQC-GBS(Saliency)Model=DeepSeek-MoE-16B, Precision=W3A16, Evaluation Protocol=Zero-shot2026.05 | 72.66 | |
| Distillation#Params=270M, #Size=dim=1024 16 Layers, Pre-training Tokens=5B, Fine-tuning Epochs=32026.05 | 72.66 | |
| XPERT-OLMoE#Params=391M, #Size=dim=2048 8 Layers, Pre-training Tokens=5B, Fine-tuning Epochs=32026.05 | 72.63 | |
| UniformModel=Mixtral 8x7B, Avg. bits/exp.=2, Memory (GB)=13.1, Evaluation protocol=zero-shot2026.04 | 72.51 | |
| XPERT-DeepSeek#Params=480M, #Size=dim=2048 12 Layers, Pre-training Tokens=5B, Fine-tuning Epochs=32026.05 | 72.35 | |
| Gemma3 4BArchitecture=Dense, FLOPs ($10^{21}$)=96, Tokens (T)=42026.05 | 72.3 | |
| PMQModel=Mixtral 8x7B, Avg. bits/exp.=2.125, Memory (GB)=13.8, Evaluation protocol=zero-shot2026.04 | 72.26 | |
| Router norm + Max varModel=Mixtral 8x22B, Avg. bits/exp.=2.25, Memory (GB)=43.0, Evaluation protocol=zero-shot2026.04 | 72.23 | |
| SARQC-GBS(Identity)Model=DeepSeek-MoE-16B, Precision=W3A16, Evaluation Protocol=Zero-shot2026.05 | 72.23 | |
| XPERT-DeepSeek#Params=570M, #Size=dim=2048 16 Layers, Pre-training Tokens=5B, Fine-tuning Epochs=32026.05 | 72.14 | |
| Out. Cosine-SimPruning Method=Out. Cosine-Sim, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance2026.05 | 71.99 | |
| Distillation#Params=570M, #Size=dim=2048 16 Layers, Pre-training Tokens=5B, Fine-tuning Epochs=32026.05 | 71.77 | |
| Distillation#Params=480M, #Size=dim=2048 12 Layers, Pre-training Tokens=5B, Fine-tuning Epochs=32026.05 | 71.62 | |
| Scratch#Params=270M, #Size=dim=1024 16 Layers, Pre-training Tokens=5B, Fine-tuning Epochs=32026.05 | 71.56 | |
| XPERT-DeepSeek#Params=391M, #Size=dim=2048 8 Layers, Pre-training Tokens=5B, Fine-tuning Epochs=32026.05 | 71.56 |