Language Understanding on MMLU (MMLU Score)
73.02MMLU ScoreBF16 Baseline
Evaluation Results
| Method | Links | |
|---|---|---|
| BF16 BaselineAvg. Bitwidth=16.000, Backbone=Qwen3-8B2025.07 | 73.02 | |
| SpecEM (All)Ensemble Base Models=All2024.12 | 73.01 | |
| SSQR-4%Avg. Bitwidth=5.405, Backbone=Qwen3-8B2025.07 | 72.63 | |
| HPTQAvg. Bitwidth=4.125, Backbone=Qwen3-8B2025.07 | 72.28 | |
| SSQR-2%Avg. Bitwidth=4.765, Backbone=Qwen3-8B2025.07 | 72.27 | |
| SSQR-5%Avg. Bitwidth=5.725, Backbone=Qwen3-8B2025.07 | 72.23 | |
| SpecEM (Qwen2+Gemma2)Ensemble Base Models=Qwen2-7B-instruct, Gemma-2-9B-instruct2024.12 | 72.18 | |
| HRTNAvg. Bitwidth=4.125, Backbone=Qwen3-8B2025.07 | 72.15 | |
| SSQR-1%Avg. Bitwidth=4.445, Backbone=Qwen3-8B2025.07 | 72.12 | |
| UniTEEnsemble Base Models=All2024.12 | 71.94 | |
| SSQR-3%Avg. Bitwidth=5.085, Backbone=Qwen3-8B2025.07 | 71.89 | |
| SpecEM (GLM4+Gemma2)Ensemble Base Models=GLM-4-9B-instruct, Gemma-2-9B-instruct2024.12 | 71.82 | |
| Majority-VotingEnsemble Base Models=All2024.12 | 71.78 | |
| GPTQAvg. Bitwidth=4.125, Backbone=Qwen3-8B2025.07 | 71.76 | |
| Gemma-2-9B-instructModel Category=Base LLM2024.12 | 71.51 | |
| HPTQAvg. Bitwidth=3.125, Backbone=Qwen3-8B2025.07 | 70.96 | |
| SSQR-5%Avg. Bitwidth=4.725, Backbone=Qwen3-8B2025.07 | 70.76 | |
| SpecEM (Qwen2+GLM4)Ensemble Base Models=Qwen2-7B-instruct, GLM-4-9B-instruct2024.12 | 70.73 | |
| MOAEnsemble Base Models=All2024.12 | 70.43 | |
| SSQR-3%Avg. Bitwidth=4.085, Backbone=Qwen3-8B2025.07 | 69.88 | |
| RTNAvg. Bitwidth=4.125, Backbone=Qwen3-8B2025.07 | 69.71 | |
| SSQR-2%Avg. Bitwidth=3.765, Backbone=Qwen3-8B2025.07 | 69.66 | |
| SSQR-4%Avg. Bitwidth=4.405, Backbone=Qwen3-8B2025.07 | 69.51 | |
| SSQR-1%Avg. Bitwidth=3.445, Backbone=Qwen3-8B2025.07 | 68.46 | |
| Qwen2-7B-instructModel Category=Base LLM2024.12 | 68.23 | |
| HRTNAvg. Bitwidth=3.125, Backbone=Qwen3-8B2025.07 | 67.85 | |
| GLM-4-9B-instructModel Category=Base LLM2024.12 | 67.16 | |
| GPTQAvg. Bitwidth=3.125, Backbone=Qwen3-8B2025.07 | 65.8 | |
| DEITAModel=Qwen-2.5-3B, Data Selection=1%2025.04 | 65.43 | |
| SSPLModel=Qwen-2.5-3B, Data Selection=1%2025.04 | 65.33 | |
| RAISEModel=Qwen-2.5-3B, Data Selection=1%2025.04 | 65.32 | |
| RANDModel=Qwen-2.5-3B, Data Selection=1%2025.04 | 65.3 | |
| IFDModel=Qwen-2.5-3B, Data Selection=1%2025.04 | 65.25 | |
| Full Alpaca (100%)Model=Qwen-2.5-3B, Data Selection=100%2025.04 | 65.22 | |
| AlpaGasusModel=Qwen-2.5-3B, Data Selection=1%2025.04 | 65.18 | |
| Base Model (0%)Model=Qwen-2.5-3B, Data Selection=0%2025.04 | 63.19 | |
| SSQR-5%Avg. Bitwidth=3.725, Backbone=Qwen3-8B2025.07 | 58.67 | |
| TITOKTransfer setting=Mistral 7B → Mistral 7B, k=70%, Evaluation Protocol=Zero-shot2025.10 | 56.1 | |
| KDTransfer setting=Mistral 7B → Mistral 7B, Evaluation Protocol=Zero-shot2025.10 | 56 | |
| VanillaTransfer setting=Mistral 7B → Mistral 7B, Evaluation Protocol=Zero-shot2025.10 | 55.7 | |
| RAISEModel=Llama-3.2-3B, Data Selection=1%2025.04 | 54.64 | |
| SSQR-4%Avg. Bitwidth=3.405, Backbone=Qwen3-8B2025.07 | 54.07 | |
| TransLoRATransfer setting=Mistral 7B → Mistral 7B, Evaluation Protocol=Zero-shot2025.10 | 53.4 | |
| HPTQAvg. Bitwidth=2.125, Backbone=Qwen3-8B2025.07 | 52.99 | |
| RANDModel=Llama-3.2-3B, Data Selection=1%2025.04 | 52.86 | |
| Full Alpaca (100%)Model=Llama-3.2-3B, Data Selection=100%2025.04 | 52.76 | |
| IFDModel=Llama-3.2-3B, Data Selection=1%2025.04 | 52.66 | |
| AlpaGasusModel=Llama-3.2-3B, Data Selection=1%2025.04 | 52.3 | |
| DEITAModel=Llama-3.2-3B, Data Selection=1%2025.04 | 51.9 | |
| Base Model (0%)Model=Llama-3.2-3B, Data Selection=0%2025.04 | 51.66 | |
| SSPLModel=Llama-3.2-3B, Data Selection=1%2025.04 | 50.11 | |
| SASFTModel=Qwen3-8B-Base, Training Set=Chinese 210k2025.07 | 49.6 | |
| SFTModel=Qwen3-8B-Base, Training Set=Chinese 210k2025.07 | 49.53 | |
| SSQR-3%Avg. Bitwidth=3.085, Backbone=Qwen3-8B2025.07 | 49.32 | |
| SFT+PenaltyModel=Qwen3-8B-Base, Training Set=Chinese 210k2025.07 | 48.64 | |
| TITOKTransfer setting=Mistral 7B → Llama3 8B, k=70%, Evaluation Protocol=Zero-shot2025.10 | 48.5 | |
| KDTransfer setting=Mistral 7B → Llama3 8B, Evaluation Protocol=Zero-shot2025.10 | 48.2 | |
| RTNAvg. Bitwidth=3.125, Backbone=Qwen3-8B2025.07 | 47.9 | |
| TITOKTransfer setting=Llama3 3B → Llama3 8B, k=30%, Evaluation Protocol=Zero-shot2025.10 | 47.8 | |
| KDTransfer setting=Llama3 3B → Llama3 8B, Evaluation Protocol=Zero-shot2025.10 | 47.7 | |
| TITOKTransfer setting=Llama2 7B → Llama3 8B, k=70%, Evaluation Protocol=Zero-shot2025.10 | 47.7 | |
| KDTransfer setting=Llama2 7B → Llama3 8B, Evaluation Protocol=Zero-shot2025.10 | 47.6 | |
| TransLoRATransfer setting=Mistral 7B → Llama3 8B, Evaluation Protocol=Zero-shot2025.10 | 47.3 | |
| SASFTModel=Gemma-2-9B, Training Set=Chinese 210k2025.07 | 47.04 | |
| VanillaTransfer setting=Mistral 7B → Llama3 8B, Evaluation Protocol=Zero-shot2025.10 | 46.9 | |
| VanillaTransfer setting=Llama3 3B → Llama3 8B, Evaluation Protocol=Zero-shot2025.10 | 46.9 | |
| VanillaTransfer setting=Llama2 7B → Llama3 8B, Evaluation Protocol=Zero-shot2025.10 | 46.9 | |
| TransLoRATransfer setting=Llama2 7B → Llama3 8B, Evaluation Protocol=Zero-shot2025.10 | 46.8 | |
| TransLoRATransfer setting=Llama3 3B → Llama3 8B, Evaluation Protocol=Zero-shot2025.10 | 46.7 | |
| SFT+GRPOModel=Gemma-2-9B, Training Set=Chinese 210k2025.07 | 46.22 | |
| SFTModel=Gemma-2-9B, Training Set=Chinese 210k2025.07 | 45.77 | |
| SFT+PenaltyModel=Gemma-2-9B, Training Set=Chinese 210k2025.07 | 45.39 | |
| SFT+GRPOModel=Qwen3-8B-Base, Training Set=Chinese 210k2025.07 | 44.85 | |
| RAISEModel=Llama-3.2-1B, Data Selection=1%2025.04 | 38.14 | |
| SASFTModel=Qwen3-1.7B-Base, Training Set=Chinese 210k2025.07 | 38.1 | |
| SFT+PenaltyModel=Qwen3-1.7B-Base, Training Set=Chinese 210k2025.07 | 37.76 | |
| SSQR-2%Avg. Bitwidth=2.765, Backbone=Qwen3-8B2025.07 | 37.48 | |
| IFDModel=Llama-3.2-1B, Data Selection=1%2025.04 | 37.35 | |
| SFTModel=Qwen3-1.7B-Base, Training Set=Chinese 210k2025.07 | 37.27 | |
| SSPLModel=Llama-3.2-1B, Data Selection=1%2025.04 | 37.2 | |
| SFT+GRPOModel=Qwen3-1.7B-Base, Training Set=Chinese 210k2025.07 | 36.99 | |
| AlpaGasusModel=Llama-3.2-1B, Data Selection=1%2025.04 | 36.89 | |
| DEITAModel=Llama-3.2-1B, Data Selection=1%2025.04 | 36.58 | |
| Full Alpaca (100%)Model=Llama-3.2-1B, Data Selection=100%2025.04 | 35.94 | |
| RANDModel=Llama-3.2-1B, Data Selection=1%2025.04 | 35.91 | |
| Base Model (0%)Model=Llama-3.2-1B, Data Selection=0%2025.04 | 35.53 | |
| GPTQAvg. Bitwidth=2.125, Backbone=Qwen3-8B2025.07 | 34.25 | |
| HRTNAvg. Bitwidth=2.125, Backbone=Qwen3-8B2025.07 | 33.91 | |
| SFT+PenaltyModel=Llama-3.1-8B, Training Set=Chinese 210k2025.07 | 33.37 | |
| SASFTModel=Llama-3.1-8B, Training Set=Chinese 210k2025.07 | 33.37 | |
| SFTModel=Llama-3.1-8B, Training Set=Chinese 210k2025.07 | 31.53 | |
| SFT+GRPOModel=Llama-3.1-8B, Training Set=Chinese 210k2025.07 | 30.35 | |
| SFT+GRPOModel=Gemma-2-2B, Training Set=Chinese 210k2025.07 | 28.99 | |
| SFT+PenaltyModel=Gemma-2-2B, Training Set=Chinese 210k2025.07 | 28.8 | |
| SFTModel=Gemma-2-2B, Training Set=Chinese 210k2025.07 | 28.58 | |
| SASFTModel=Gemma-2-2B, Training Set=Chinese 210k2025.07 | 27.89 | |
| SSQR-1%Avg. Bitwidth=2.445, Backbone=Qwen3-8B2025.07 | 26.71 | |
| RTNAvg. Bitwidth=2.125, Backbone=Qwen3-8B2025.07 | 22.95 |