Knowledge Evaluation on MMLU
82.03MMLU AccuracyW16A16-Direct
Evaluation results
| Method | Links | ||
|---|---|---|---|
| W16A16-DirectModel=Qwen3-30B2026.04 | 82.03 | 0 | |
| W8A8-DirectModel=Qwen3-30B2026.04 | 81.98 | -0.06 | |
| MXFP4-W2FP8+DynModel=Qwen3-30B2026.04 | 79.91 | -2.12 | |
| MXFP4-DynamicModel=Qwen3-30B2026.04 | 79.78 | -2.25 | |
| MXFP4-OSCModel=Qwen3-30B2026.04 | 79.28 | -2.75 | |
| MXFP4-W2FP8Model=Qwen3-30B2026.04 | 78.93 | -3.1 | |
| MXFP4-DirectModel=Qwen3-30B2026.04 | 78.23 | -3.8 | |
| W16A16-DirectModel=Qwen3-8B2026.04 | 74.97 | 0 | |
| W8A8-DirectModel=Qwen3-8B2026.04 | 74.61 | -0.36 | |
| OriginalBackbone=Qwen2.5-7B2026.02 | 74.26 | — | |
| Linear-AcTBackbone=Qwen2.5-7B2026.02 | 73.65 | — | |
| Dense baselineBase Model=Qwen2.5-7B-Instruct, Sparsity=Dense2026.06 | 73.51 | — | |
| CHaRSBackbone=Qwen2.5-7B2026.02 | 73.41 | — | |
| CHaRS-PCTBackbone=Qwen2.5-7B2026.02 | 73.31 | — | |
| MXFP4-W2FP8+DynModel=Qwen3-8B2026.04 | 72.64 | -2.33 | |
| MXFP4-OSCModel=Qwen3-8B2026.04 | 72.4 | -2.56 | |
| MXFP4-DynamicModel=Qwen3-8B2026.04 | 72.26 | -2.71 | |
| Dense baselineBase Model=Qwen3-4B-Instruct, Sparsity=Dense2026.06 | 72.18 | — | |
| SUBFITBase Model=Qwen3-4B-Instruct, Sparsity=12.5%2026.06 | 72.08 | — | |
| MXFP4-W2FP8Model=Qwen3-8B2026.04 | 70.31 | -4.66 | |
| Dense baselineBase Model=Llama-3.1-8B-Instruct, Sparsity=Dense2026.06 | 68.79 | — | |
| MXFP4-DirectModel=Qwen3-8B2026.04 | 68.24 | -6.73 | |
| SUBFITBase Model=Llama-3.1-8B-Instruct, Sparsity=12.5%2026.06 | 66.65 | — | |
| OriginalBackbone=Llama3-8B2026.02 | 65.52 | — | |
| Linear-AcTBackbone=Llama3-8B2026.02 | 64.64 | — | |
| CHaRSBackbone=Llama3-8B2026.02 | 64.48 | — | |
| CHaRS-PCTBackbone=Llama3-8B2026.02 | 64.37 | — | |
| Dense baselineBase Model=Llama-3.2-3B-Instruct, Sparsity=Dense2026.06 | 61.84 | — | |
| SUBFITBase Model=Llama-3.2-3B-Instruct, Sparsity=12.5%2026.06 | 61.07 | — | |
| No WatermarkBase Model=Mistral2026.05 | 58.9 | — | |
| SynthIDBase Model=Mistral2026.05 | 58.9 | 0 | |
| DTMBase Model=Mistral2026.05 | 58.9 | 0 | |
| TWBase Model=Mistral2026.05 | 58.9 | 0 | |
| SAFESEALBase Model=Mistral2026.05 | 58.9 | 0 | |
| EXPBase Model=Mistral2026.05 | 58.7 | -0.3 | |
| KGWBase Model=Mistral2026.05 | 58.6 | -0.5 | |
| OriginalBackbone=Gemma2-2B2026.02 | 53.03 | — | |
| Linear-AcTBackbone=Gemma2-2B2026.02 | 51.77 | — | |
| SUBFITBase Model=DeepSeek-7B-chat, Sparsity=12.5%2026.06 | 51.32 | — | |
| SUBFITBase Model=Qwen2.5-7B-Instruct, Sparsity=12.5%2026.06 | 51.05 | — | |
| Dense baselineBase Model=DeepSeek-7B-chat, Sparsity=Dense2026.06 | 50.9 | — | |
| CHaRSBackbone=Gemma2-2B2026.02 | 50.69 | — | |
| CHaRS-PCTBackbone=Gemma2-2B2026.02 | 50.21 | — | |
| FedSDR2026.05 | 47.11 | — | |
| FedSD2026.05 | 46.96 | — | |
| No WatermarkBase Model=LLaMA-22026.05 | 46.7 | — | |
| SynthIDBase Model=LLaMA-22026.05 | 46.7 | 0 | |
| DTMBase Model=LLaMA-22026.05 | 46.7 | 0 | |
| TWBase Model=LLaMA-22026.05 | 46.7 | 0 | |
| SAFESEALBase Model=LLaMA-22026.05 | 46.7 | 0 | |
| EXPBase Model=LLaMA-22026.05 | 45.9 | -1.7 | |
| Dual Upload2026.05 | 45.49 | — | |
| Baseline(FedAvg)2026.05 | 45.22 | — | |
| Dual-LoRA(FedDPA)2026.05 | 44.78 | — | |
| KGWBase Model=LLaMA-22026.05 | 44.5 | -4.7 | |
| Distill Rect2026.05 | 42.65 | — | |
| STOCModel Scale=1.7B, Prompts=5-shot, Freezing Layers=Best reported2026.05 | 42.49 | — | |
| STOCModel Scale=0.6B, Prompts=5-shot, Freezing Layers=Best reported2026.05 | 40.17 | — | |
| LAMOLModel Scale=1.7B, Prompts=5-shot, Freezing Layers=Best reported2026.05 | 39.28 | — | |
| LAMOLModel Scale=0.6B, Prompts=5-shot, Freezing Layers=Best reported2026.05 | 38.87 | — | |
| SIRBase Model=LLaMA-22026.05 | 27.9 | -40.2 | |
| NaiveModel Scale=1.7B, Prompts=5-shot, Freezing Layers=Best reported2026.05 | 23.63 | — | |
| NaiveModel Scale=0.6B, Prompts=5-shot, Freezing Layers=Best reported2026.05 | 22.49 | — | |
| SIRBase Model=Mistral2026.05 | 20.9 | -64.5 |