Question Answering on OpenBookQA (Accuracy and Performance Gain)
84.4AccuracyQwen3 8B Base
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Qwen3 8B BaseClassifier=Majority-labeled2026.01 | 84.4 | -0.6 | |
| Qwen3 8B BaseClassifier=Self-labeled2026.01 | 84.2 | -0.8 | |
| Kimi-AudioFull Duplex=false2026.07 | 83.5 | — | |
| TS (C)Contextual=true2026.02 | 79.3 | 5.8 | |
| Llama 3.1 8BClassifier=Self-labeled2026.01 | 76.6 | 1 | |
| Llama 3.1 8BClassifier=Majority-labeled2026.01 | 76.6 | 1 | |
| DuplexPOFull Duplex=true2026.07 | 73.7 | — | |
| Mistral Nemo Base 2407Classifier=Majority-labeled2026.01 | 73.4 | 0.2 | |
| Mistral Nemo Base 2407Classifier=Self-labeled2026.01 | 73 | -0.2 | |
| SFT BaselineFull Duplex=true2026.07 | 72.2 | — | |
| Baichuan-AudioFull Duplex=false2026.07 | 71.7 | — | |
| Mistral 7B v0.3Classifier=Self-labeled2026.01 | 68.8 | 0.6 | |
| Mistral 7B v0.3Classifier=Majority-labeled2026.01 | 68.4 | 0.2 | |
| GLM-4-VoiceFull Duplex=false2026.07 | 53.4 | — | |
| Qwen3 0.6BClassifier=Self-labeled2026.01 | 53 | 0.8 | |
| Qwen3 0.6BClassifier=Majority-labeled2026.01 | 52.2 | 0 | |
| Qwen2-AudioFull Duplex=false2026.07 | 49.5 | — | |
| Llama 3.2 1BClassifier=Self-labeled2026.01 | 47.4 | 3.8 | |
| MoonlightPruning ratio=0%, Base Model=Moonlight, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.07 | 45.6 | — | |
| Qwen3-30B-A3BPruning ratio=0%, Base Model=Qwen3-30B-A3B, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.07 | 44.8 | — | |
| OriginalBase Model=LLaMA3-8B2026.05 | 44.8 | — | |
| OriginalBackbone=LLaMA3-8B2026.05 | 44.8 | — | |
| OLMoEPruning ratio=0%, Base Model=OLMoE, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.07 | 44.4 | — | |
| Qwen2-57B-A14BPruning ratio=0%, Base Model=Qwen2-57B-A14B, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.07 | 44.2 | — | |
| Llama 3.2 1BClassifier=Majority-labeled2026.01 | 44 | 0.4 | |
| CleanBackbone=LLaMA2-13B-Chat2026.03 | 44 | — | |
| Fine-PruningBackbone=LLaMA2-13B-Chat, Parameter Setting=LoRA Adapter2026.03 | 43.8 | — | |
| BD-VAXBackbone=LLaMA2-13B-Chat, Parameter Setting=LoRA Adapter2026.03 | 43.8 | — | |
| Accuracy (Ours)Backbone=LLaMA3-8B, Pruning Ratio=25%2026.05 | 43.8 | — | |
| Deepseek-V2-LitePruning ratio=0%, Base Model=Deepseek-V2-Lite, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.07 | 43.6 | — | |
| CleanBackbone=LLaMA2-7B-Chat2026.03 | 43.6 | — | |
| FTBackbone=LLaMA2-13B-Chat, Parameter Setting=Full Params2026.03 | 43.6 | — | |
| QuantizationBackbone=LLaMA2-13B-Chat, Parameter Setting=Full Params2026.03 | 43.6 | — | |
| CROWBackbone=LLaMA2-13B-Chat, Parameter Setting=Full Params2026.03 | 43.6 | — | |
| Accuracy (Arc-E)Backbone=LLaMA3-8B, Pruning Ratio=25%2026.05 | 43.6 | — | |
| Fine-PruningBackbone=LLaMA2-13B-Chat, Parameter Setting=Full Params2026.03 | 43.4 | — | |
| Fine-PruningBackbone=LLaMA2-7B-Chat, Parameter Setting=Full Params2026.03 | 43 | — | |
| BD-VAXBackbone=LLaMA2-13B-Chat, Parameter Setting=Full Params2026.03 | 43 | — | |
| Baichuan-7bRatio=0%, Zero-shot=true2024.03 | 42.8 | — | |
| RSPruning ratio=50%, Base Model=Qwen3-30B-A3B, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.07 | 42.4 | — | |
| Fine-PruningBackbone=LLaMA2-7B-Chat, Parameter Setting=LoRA Adapter2026.03 | 42.4 | — | |
| BD-VAXBackbone=LLaMA2-7B-Chat, Parameter Setting=LoRA Adapter2026.03 | 42.4 | — | |
| FTBackbone=LLaMA2-7B-Chat, Parameter Setting=Full Params2026.03 | 42.2 | — | |
| PruningBackbone=LLaMA2-7B-Chat, Parameter Setting=LoRA Adapter2026.03 | 42.2 | — | |
| PruningBackbone=LLaMA2-13B-Chat, Parameter Setting=LoRA Adapter2026.03 | 42.2 | — | |
| FTBackbone=LLaMA2-13B-Chat, Parameter Setting=LoRA Adapter2026.03 | 42.16 | — | |
| MoNEPruning ratio=50%, Base Model=Qwen2-57B-A14B, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.07 | 42 | — | |
| AttackedBackbone=LLaMA2-13B-Chat2026.03 | 42 | — | |
| MoNEPruning ratio=50%, Base Model=Qwen3-30B-A3B, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.07 | 41.6 | — | |
| Ettin-Dec-1BModel Size Category=XL, Parameters=908M-1.2B, Zero-shot=true, Closed-book=true2025.07 | 41.6 | — | |
| AttackedBackbone=LLaMA2-7B-Chat2026.03 | 41.4 | — | |
| TaylorPruning Method=Taylor, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance2026.05 | 41.4 | — | |
| Out. Norm-SimPruning Method=Out. Norm-Sim, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance2026.05 | 41.4 | — | |
| TaylorBackbone=LLaMA3-8B, Pruning Ratio=25%2026.05 | 41.4 | — | |
| FTBackbone=LLaMA2-7B-Chat, Parameter Setting=LoRA Adapter2026.03 | 41.2 | — | |
| MoNEPruning ratio=50%, Base Model=Moonlight, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.07 | 40.8 | — | |
| RSPruning ratio=50%, Base Model=Deepseek-V2-Lite, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.07 | 40.8 | — | |
| BD-VAXBackbone=LLaMA2-7B-Chat, Parameter Setting=Full Params2026.03 | 40.6 | — | |
| CROWBackbone=LLaMA2-7B-Chat, Parameter Setting=Full Params2026.03 | 40.2 | — | |
| Out. Cosine-SimPruning Method=Out. Cosine-Sim, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance2026.05 | 40.2 | — | |
| PruningBackbone=LLaMA2-7B-Chat, Parameter Setting=Full Params2026.03 | 40 | — | |
| HyWIARatio=25%, Zero-shot=true2024.03 | 39.6 | — | |
| Out. Divergence-SimPruning Method=Out. Divergence-Sim, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance2026.05 | 39.6 | — | |
| Out. Norm-SimBackbone=LLaMA3-8B, Pruning Ratio=25%2026.05 | 39.6 | — | |
| SALM-DuplexFull Duplex=true2026.07 | 39.6 | — | |
| MC-SMoEPruning ratio=50%, Base Model=Qwen3-30B-A3B, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.07 | 39.4 | — | |
| QuantizationBackbone=LLaMA2-7B-Chat, Parameter Setting=Full Params2026.03 | 39.4 | — | |
| PerplexityPruning Method=Perplexity, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance2026.05 | 39.2 | — | |
| MC-SMoEPruning ratio=50%, Base Model=Qwen2-57B-A14B, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.07 | 38.8 | — | |
| LLM-Pruner Vector⋆Ratio=25%, Zero-shot=true2024.03 | 38.4 | — | |
| FLAPPruning ratio=50%, Base Model=Qwen2-57B-A14B, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.07 | 38.2 | — | |
| RSPruning ratio=50%, Base Model=Moonlight, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.07 | 38 | — | |
| LLM-Pruner Element2⋆Ratio=25%, Zero-shot=true2024.03 | 37.8 | — | |
| SmolLM2-360mModel Size Category=Large, Parameters=360-410M, Zero-shot=true, Closed-book=true2025.07 | 37.6 | — | |
| PruningBackbone=LLaMA2-13B-Chat, Parameter Setting=Full Params2026.03 | 37.4 | — | |
| Llama-3.2-1BModel Size Category=XL, Parameters=908M-1.2B, Zero-shot=true, Closed-book=true2025.07 | 37.2 | — | |
| MoNEPruning ratio=50%, Base Model=Deepseek-V2-Lite, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.07 | 36.8 | — | |
| Cosine SimilarityPruning Method=Cosine Similarity, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance2026.05 | 36.8 | — | |
| Slice-GPTBackbone=LLaMA3-8B, Pruning Ratio=25%2026.05 | 36.8 | — | |
| Cosine SimilarityBackbone=LLaMA3-8B, Pruning Ratio=25%2026.05 | 36.2 | — | |
| Out. Cosine-SimBackbone=LLaMA3-8B, Pruning Ratio=25%2026.05 | 36.2 | — | |
| FLAPPruning ratio=50%, Base Model=Moonlight, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.07 | 36 | — | |
| OLMo-1B-0724Model Size Category=XL, Parameters=908M-1.2B, Zero-shot=true, Closed-book=true2025.07 | 35.6 | — | |
| FLAPPruning ratio=50%, Base Model=Deepseek-V2-Lite, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.07 | 35.2 | — | |
| Ettin-Dec-400mModel Size Category=Large, Parameters=360-410M, Zero-shot=true, Closed-book=true2025.07 | 34.4 | — | |
| Out. Divergence-SimBackbone=LLaMA3-8B, Pruning Ratio=25%2026.05 | 34.2 | — | |
| FLAPPruning ratio=50%, Base Model=Qwen3-30B-A3B, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.07 | 34 | — | |
| Acc (Ours)Pruning Method=Accuracy-based relevance score, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance2026.05 | 33.8 | — | |
| Accuracy (C4)Backbone=LLaMA3-8B, Pruning Ratio=25%2026.05 | 33.8 | — | |
| MoNEPruning ratio=50%, Base Model=OLMoE, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.07 | 32.4 | — | |
| SmolLM2-135mModel Size Category=Base, Parameters=135-160M, Zero-shot=true, Closed-book=true2025.07 | 32.4 | — | |
| MC-SMoEPruning ratio=50%, Base Model=Moonlight, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.07 | 31.4 | — | |
| DenseBackbone=Llama-2-7B, Sparsity=0%, Evaluation Protocol=Zero-shot2025.10 | 31.4 | — | |
| L-ADMMBackbone=Llama-2-7B, Sparsity=50%, Evaluation Protocol=Zero-shot2025.10 | 31.4 | — | |
| Freeze-OmniFull Duplex=true2026.07 | 31 | — | |
| ALPSBackbone=Llama-2-7B, Sparsity=50%, Evaluation Protocol=Zero-shot2025.10 | 30.8 | — | |
| WandaBackbone=Llama-2-7B, Sparsity=50%, Evaluation Protocol=Zero-shot2025.10 | 30.6 | — | |
| PerplexityBackbone=LLaMA3-8B, Pruning Ratio=25%2026.05 | 30.5 | — | |
| AngularPruning ratio=50%, Base Model=Qwen2-57B-A14B, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.07 | 29.8 | — | |
| SparseGPTBackbone=Llama-2-7B, Sparsity=50%, Evaluation Protocol=Zero-shot2025.10 | 29.8 | — |