General Knowledge on MMLU EL (Accuracy)
44.06MMLU EL (General Knowledge) AccuracyLayerMoE
Evaluation results
| Method | Links | |
|---|---|---|
| LayerMoEBackbone=Llama-3.2-3B, Stage=Stage 2, #Experts=842026.03 | 44.06 | |
| NeuronMoE-S1Backbone=Llama-3.2-3B, Stage=Stage 1, #Experts=49 (-42%)2026.03 | 43.95 | |
| NeuronMoEBackbone=Llama-3.2-3B, Stage=Stage 2, #Experts=49 (-42%)2026.03 | 43.66 | |
| LayerMoE-S1Backbone=Llama-3.2-3B, Stage=Stage 1, #Experts=842026.03 | 43.41 | |
| NeuronMoE-ENBackbone=Llama-3.2-3B, Stage=Stage 1, #Experts=37 (-56%)2026.03 | 43.26 | |
| Llama-3.2-3B DenseBackbone=Llama-3.2-3B, Stage=Dense, #Experts=-2026.03 | 41.17 | |
| LayerMoE-S1Backbone=Qwen-1.5-1.8B, Stage=Stage 1, #Experts=722026.03 | 31.26 | |
| NeuronMoE-S1Backbone=Qwen-1.5-1.8B, Stage=Stage 1, #Experts=36 (-50%)2026.03 | 29.23 |