List your company
ResearchFormerly WizWand

Commonsense Reasoning on HellaSwag (HS) (Accuracy)

78.94HS AccuracyOriginal
20.377635.581350.78565.9887Aug 9, 2025Sep 27, 2025Nov 15, 2025Jan 4, 2026Feb 22, 2026Apr 12, 2026Jun 1, 2026
Updated 5mo ago

Evaluation results

MethodLinks
OriginalBase Model=LLaMA3-8B2026.0578.94
OriginalBackbone=LLaMA3-8B2026.0578.94
Accuracy (Arc-E)Backbone=LLaMA3-8B, Pruning Ratio=25%2026.0573.62
LLaMA3Size=3.0B, Bit=16.0, Zero-shot=true2025.0873.6
Accuracy (Ours)Backbone=LLaMA3-8B, Pruning Ratio=25%2026.0573.41
Llama-3.1-8B-InstructModel=Llama-3.1-8B-Instruct, Sparsity=Dense2026.0672.52
DeepSeek-7B-chatModel=DeepSeek-7B-chat, Sparsity=Dense2026.0670.56
Cosine SimilarityPruning Method=Cosine Similarity, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance2026.0570.24
Cosine SimilarityBackbone=LLaMA3-8B, Pruning Ratio=25%2026.0569.35
Out. Norm-SimPruning Method=Out. Norm-Sim, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance2026.0567
Out. Cosine-SimPruning Method=Out. Cosine-Sim, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance2026.0566.57
PerplexityPruning Method=Perplexity, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance2026.0565.96
Qwen2.5-7B-InstructModel=Qwen2.5-7B-Instruct, Sparsity=Dense2026.0665.48
Out. Divergence-SimPruning Method=Out. Divergence-Sim, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance2026.0565.09
Out. Cosine-SimBackbone=LLaMA3-8B, Pruning Ratio=25%2026.0564.6
Llama-3.2-3B-InstructModel=Llama-3.2-3B-Instruct, Sparsity=Dense2026.0664.53
Out. Divergence-SimBackbone=LLaMA3-8B, Pruning Ratio=25%2026.0564.37
TaylorPruning Method=Taylor, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance2026.0563.73
LLaMA3Size=1.3B, Bit=16.0, Zero-shot=true2025.0863.7
TaylorBackbone=LLaMA3-8B, Pruning Ratio=25%2026.0562.73
Out. Norm-SimBackbone=LLaMA3-8B, Pruning Ratio=25%2026.0561.73
Slice-GPTBackbone=LLaMA3-8B, Pruning Ratio=25%2026.0561.19
OPTSize=2.7B, Bit=16.0, Zero-shot=true2025.0860.6
ReplaceMe (Cosine)Model=Llama-3.1-8B-Instruct, Sparsity=25%2026.0660.22
Acc (Ours)Pruning Method=Accuracy-based relevance score, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance2026.0560.16
Accuracy (C4)Backbone=LLaMA3-8B, Pruning Ratio=25%2026.0560.16
Streamline (Layer)Model=Llama-3.1-8B-Instruct, Sparsity=25%2026.0659.9
SUBFITModel=Llama-3.1-8B-Instruct, Sparsity=25%2026.0659.33
PythiaSize=2.8B, Bit=16.0, Zero-shot=true2025.0859.3
Streamline (FFN)Model=Llama-3.1-8B-Instruct, Sparsity=25%2026.0658.31
BinaryLLMSize=3.0B, Bit=1.01, Zero-shot=true2025.0858.3
ReplaceMe (LS)Model=Llama-3.1-8B-Instruct, Sparsity=25%2026.0657.27
OPTSize=1.3B, Bit=16.0, Zero-shot=true2025.0853.7
ReplaceMe (Cosine)Model=Qwen2.5-7B-Instruct, Sparsity=25%2026.0653.17
Streamline (Layer)Model=Qwen2.5-7B-Instruct, Sparsity=25%2026.0653.1
ReplaceMe (LS)Model=Qwen2.5-7B-Instruct, Sparsity=25%2026.0652.47
Streamline (FFN)Model=Qwen2.5-7B-Instruct, Sparsity=25%2026.0651.87
SUBFITModel=Llama-3.2-3B-Instruct, Sparsity=25%2026.0651.42
Streamline (Layer)Model=Llama-3.2-3B-Instruct, Sparsity=25%2026.0650.93
ReplaceMe (Cosine)Model=DeepSeek-7B-chat, Sparsity=25%2026.0650.7
ReplaceMe (Cosine)Model=Llama-3.2-3B-Instruct, Sparsity=25%2026.0650.45
Streamline (FFN)Model=Llama-3.2-3B-Instruct, Sparsity=25%2026.0650.25
Streamline (Layer)Model=DeepSeek-7B-chat, Sparsity=25%2026.0650.02
BinaryLLMSize=1.3B, Bit=1.01, Zero-shot=true2025.0849.6
ReplaceMe (LS)Model=Llama-3.2-3B-Instruct, Sparsity=25%2026.0648.84
SUBFITModel=DeepSeek-7B-chat, Sparsity=25%2026.0648.64
SUBFITModel=Qwen2.5-7B-Instruct, Sparsity=25%2026.0648.19
PerplexityBackbone=LLaMA3-8B, Pruning Ratio=25%2026.0548.01
ReplaceMe (LS)Model=DeepSeek-7B-chat, Sparsity=25%2026.0647.15
Qwen3-4B-InstructModel=Qwen3-4B-Instruct, Sparsity=Dense2026.0643.42
BitNetSize=3.0B, Bit=1.59, Zero-shot=true2025.0842.9
SmolLMSize=135M, Bit=16.0, Zero-shot=true2025.0842.6
FBI-LLMSize=1.3B, Bit=1.01, Zero-shot=true2025.0842.3
Streamline (FFN)Model=DeepSeek-7B-chat, Sparsity=25%2026.0639.83
OneBit-OPTSize=2.7B, Bit=1.02, Zero-shot=true2025.0838.2
BitNetSize=1.3B, Bit=1.59, Zero-shot=true2025.0837.7
ReplaceMe (LS)Model=Qwen3-4B-Instruct, Sparsity=25%2026.0637.05
ReplaceMe (Cosine)Model=Qwen3-4B-Instruct, Sparsity=25%2026.0636.72
OneBit-OPTSize=1.3B, Bit=1.02, Zero-shot=true2025.0834.3
OPTSize=125M, Bit=16.0, Zero-shot=true2025.0831.3
PythiaSize=160M, Bit=16.0, Zero-shot=true2025.0831.3
BinaryLLMSize=135M, Bit=1.01, Zero-shot=true2025.0831.2
SUBFITModel=Qwen3-4B-Instruct, Sparsity=25%2026.0628.88
FBI-LLMSize=130M, Bit=1.01, Zero-shot=true2025.0828.7
Streamline (FFN)Model=Qwen3-4B-Instruct, Sparsity=25%2026.0622.93
Streamline (Layer)Model=Qwen3-4B-Instruct, Sparsity=25%2026.0622.63