List your company
ResearchFormerly WizWand

Language Understanding on MMLU (MMLU Score)

73.02MMLU ScoreBF16 Baseline
20.947234.466147.98561.5039Dec 10, 2024Jan 29, 2025Mar 20, 2025May 9, 2025Jun 28, 2025Aug 17, 2025Oct 6, 2025
Updated 7mo ago

Evaluation results

MethodLinks
BF16 BaselineAvg. Bitwidth=16.000, Backbone=Qwen3-8B2025.0773.02
SpecEM (All)Ensemble Base Models=All2024.1273.01
SSQR-4%Avg. Bitwidth=5.405, Backbone=Qwen3-8B2025.0772.63
HPTQAvg. Bitwidth=4.125, Backbone=Qwen3-8B2025.0772.28
SSQR-2%Avg. Bitwidth=4.765, Backbone=Qwen3-8B2025.0772.27
SSQR-5%Avg. Bitwidth=5.725, Backbone=Qwen3-8B2025.0772.23
SpecEM (Qwen2+Gemma2)Ensemble Base Models=Qwen2-7B-instruct, Gemma-2-9B-instruct2024.1272.18
HRTNAvg. Bitwidth=4.125, Backbone=Qwen3-8B2025.0772.15
SSQR-1%Avg. Bitwidth=4.445, Backbone=Qwen3-8B2025.0772.12
UniTEEnsemble Base Models=All2024.1271.94
SSQR-3%Avg. Bitwidth=5.085, Backbone=Qwen3-8B2025.0771.89
SpecEM (GLM4+Gemma2)Ensemble Base Models=GLM-4-9B-instruct, Gemma-2-9B-instruct2024.1271.82
Majority-VotingEnsemble Base Models=All2024.1271.78
GPTQAvg. Bitwidth=4.125, Backbone=Qwen3-8B2025.0771.76
Gemma-2-9B-instructModel Category=Base LLM2024.1271.51
HPTQAvg. Bitwidth=3.125, Backbone=Qwen3-8B2025.0770.96
SSQR-5%Avg. Bitwidth=4.725, Backbone=Qwen3-8B2025.0770.76
SpecEM (Qwen2+GLM4)Ensemble Base Models=Qwen2-7B-instruct, GLM-4-9B-instruct2024.1270.73
MOAEnsemble Base Models=All2024.1270.43
SSQR-3%Avg. Bitwidth=4.085, Backbone=Qwen3-8B2025.0769.88
RTNAvg. Bitwidth=4.125, Backbone=Qwen3-8B2025.0769.71
SSQR-2%Avg. Bitwidth=3.765, Backbone=Qwen3-8B2025.0769.66
SSQR-4%Avg. Bitwidth=4.405, Backbone=Qwen3-8B2025.0769.51
SSQR-1%Avg. Bitwidth=3.445, Backbone=Qwen3-8B2025.0768.46
Qwen2-7B-instructModel Category=Base LLM2024.1268.23
HRTNAvg. Bitwidth=3.125, Backbone=Qwen3-8B2025.0767.85
GLM-4-9B-instructModel Category=Base LLM2024.1267.16
GPTQAvg. Bitwidth=3.125, Backbone=Qwen3-8B2025.0765.8
DEITAModel=Qwen-2.5-3B, Data Selection=1%2025.0465.43
SSPLModel=Qwen-2.5-3B, Data Selection=1%2025.0465.33
RAISEModel=Qwen-2.5-3B, Data Selection=1%2025.0465.32
RANDModel=Qwen-2.5-3B, Data Selection=1%2025.0465.3
IFDModel=Qwen-2.5-3B, Data Selection=1%2025.0465.25
Full Alpaca (100%)Model=Qwen-2.5-3B, Data Selection=100%2025.0465.22
AlpaGasusModel=Qwen-2.5-3B, Data Selection=1%2025.0465.18
Base Model (0%)Model=Qwen-2.5-3B, Data Selection=0%2025.0463.19
SSQR-5%Avg. Bitwidth=3.725, Backbone=Qwen3-8B2025.0758.67
TITOKTransfer setting=Mistral 7B → Mistral 7B, k=70%, Evaluation Protocol=Zero-shot2025.1056.1
KDTransfer setting=Mistral 7B → Mistral 7B, Evaluation Protocol=Zero-shot2025.1056
VanillaTransfer setting=Mistral 7B → Mistral 7B, Evaluation Protocol=Zero-shot2025.1055.7
RAISEModel=Llama-3.2-3B, Data Selection=1%2025.0454.64
SSQR-4%Avg. Bitwidth=3.405, Backbone=Qwen3-8B2025.0754.07
TransLoRATransfer setting=Mistral 7B → Mistral 7B, Evaluation Protocol=Zero-shot2025.1053.4
HPTQAvg. Bitwidth=2.125, Backbone=Qwen3-8B2025.0752.99
RANDModel=Llama-3.2-3B, Data Selection=1%2025.0452.86
Full Alpaca (100%)Model=Llama-3.2-3B, Data Selection=100%2025.0452.76
IFDModel=Llama-3.2-3B, Data Selection=1%2025.0452.66
AlpaGasusModel=Llama-3.2-3B, Data Selection=1%2025.0452.3
DEITAModel=Llama-3.2-3B, Data Selection=1%2025.0451.9
Base Model (0%)Model=Llama-3.2-3B, Data Selection=0%2025.0451.66
SSPLModel=Llama-3.2-3B, Data Selection=1%2025.0450.11
SASFTModel=Qwen3-8B-Base, Training Set=Chinese 210k2025.0749.6
SFTModel=Qwen3-8B-Base, Training Set=Chinese 210k2025.0749.53
SSQR-3%Avg. Bitwidth=3.085, Backbone=Qwen3-8B2025.0749.32
SFT+PenaltyModel=Qwen3-8B-Base, Training Set=Chinese 210k2025.0748.64
TITOKTransfer setting=Mistral 7B → Llama3 8B, k=70%, Evaluation Protocol=Zero-shot2025.1048.5
KDTransfer setting=Mistral 7B → Llama3 8B, Evaluation Protocol=Zero-shot2025.1048.2
RTNAvg. Bitwidth=3.125, Backbone=Qwen3-8B2025.0747.9
TITOKTransfer setting=Llama3 3B → Llama3 8B, k=30%, Evaluation Protocol=Zero-shot2025.1047.8
KDTransfer setting=Llama3 3B → Llama3 8B, Evaluation Protocol=Zero-shot2025.1047.7
TITOKTransfer setting=Llama2 7B → Llama3 8B, k=70%, Evaluation Protocol=Zero-shot2025.1047.7
KDTransfer setting=Llama2 7B → Llama3 8B, Evaluation Protocol=Zero-shot2025.1047.6
TransLoRATransfer setting=Mistral 7B → Llama3 8B, Evaluation Protocol=Zero-shot2025.1047.3
SASFTModel=Gemma-2-9B, Training Set=Chinese 210k2025.0747.04
VanillaTransfer setting=Mistral 7B → Llama3 8B, Evaluation Protocol=Zero-shot2025.1046.9
VanillaTransfer setting=Llama3 3B → Llama3 8B, Evaluation Protocol=Zero-shot2025.1046.9
VanillaTransfer setting=Llama2 7B → Llama3 8B, Evaluation Protocol=Zero-shot2025.1046.9
TransLoRATransfer setting=Llama2 7B → Llama3 8B, Evaluation Protocol=Zero-shot2025.1046.8
TransLoRATransfer setting=Llama3 3B → Llama3 8B, Evaluation Protocol=Zero-shot2025.1046.7
SFT+GRPOModel=Gemma-2-9B, Training Set=Chinese 210k2025.0746.22
SFTModel=Gemma-2-9B, Training Set=Chinese 210k2025.0745.77
SFT+PenaltyModel=Gemma-2-9B, Training Set=Chinese 210k2025.0745.39
SFT+GRPOModel=Qwen3-8B-Base, Training Set=Chinese 210k2025.0744.85
RAISEModel=Llama-3.2-1B, Data Selection=1%2025.0438.14
SASFTModel=Qwen3-1.7B-Base, Training Set=Chinese 210k2025.0738.1
SFT+PenaltyModel=Qwen3-1.7B-Base, Training Set=Chinese 210k2025.0737.76
SSQR-2%Avg. Bitwidth=2.765, Backbone=Qwen3-8B2025.0737.48
IFDModel=Llama-3.2-1B, Data Selection=1%2025.0437.35
SFTModel=Qwen3-1.7B-Base, Training Set=Chinese 210k2025.0737.27
SSPLModel=Llama-3.2-1B, Data Selection=1%2025.0437.2
SFT+GRPOModel=Qwen3-1.7B-Base, Training Set=Chinese 210k2025.0736.99
AlpaGasusModel=Llama-3.2-1B, Data Selection=1%2025.0436.89
DEITAModel=Llama-3.2-1B, Data Selection=1%2025.0436.58
Full Alpaca (100%)Model=Llama-3.2-1B, Data Selection=100%2025.0435.94
RANDModel=Llama-3.2-1B, Data Selection=1%2025.0435.91
Base Model (0%)Model=Llama-3.2-1B, Data Selection=0%2025.0435.53
GPTQAvg. Bitwidth=2.125, Backbone=Qwen3-8B2025.0734.25
HRTNAvg. Bitwidth=2.125, Backbone=Qwen3-8B2025.0733.91
SFT+PenaltyModel=Llama-3.1-8B, Training Set=Chinese 210k2025.0733.37
SASFTModel=Llama-3.1-8B, Training Set=Chinese 210k2025.0733.37
SFTModel=Llama-3.1-8B, Training Set=Chinese 210k2025.0731.53
SFT+GRPOModel=Llama-3.1-8B, Training Set=Chinese 210k2025.0730.35
SFT+GRPOModel=Gemma-2-2B, Training Set=Chinese 210k2025.0728.99
SFT+PenaltyModel=Gemma-2-2B, Training Set=Chinese 210k2025.0728.8
SFTModel=Gemma-2-2B, Training Set=Chinese 210k2025.0728.58
SASFTModel=Gemma-2-2B, Training Set=Chinese 210k2025.0727.89
SSQR-1%Avg. Bitwidth=2.445, Backbone=Qwen3-8B2025.0726.71
RTNAvg. Bitwidth=2.125, Backbone=Qwen3-8B2025.0722.95