Reading Comprehension on Belebele EN
75.33AccuracyLayerMoE
Evaluation Results
| Method | Links | |
|---|---|---|
| LayerMoETarget Language=Turkish, Backbone=Llama-3.2-3B, Number of Experts=842026.03 | 75.33 | |
| LayerMoETarget Language=Hungarian, Backbone=Llama-3.2-3B, Number of Experts=842026.03 | 75.33 | |
| NeuronMoE-S1Backbone=Llama-3.2-3B, Stage=Stage 1, #Experts=49 (-42%)2026.03 | 75.33 | |
| NeuronMoEBackbone=Llama-3.2-3B, Stage=Stage 2, #Experts=49 (-42%)2026.03 | 75.11 | |
| NeuronMoETarget Language=Hungarian, Backbone=Llama-3.2-3B, Number of Experts=47 (-44%)2026.03 | 74.78 | |
| LayerMoEBackbone=Llama-3.2-3B, Stage=Stage 2, #Experts=842026.03 | 74.78 | |
| NeuronMoETarget Language=Turkish, Backbone=Llama-3.2-3B, Number of Experts=50 (-40%)2026.03 | 74.44 | |
| Llama-3.2-3BTraining Status=No training2025.12 | 74.2 | |
| DenseTarget Language=Turkish, Backbone=Llama-3.2-3B2026.03 | 74.11 | |
| DenseTarget Language=Hungarian, Backbone=Llama-3.2-3B2026.03 | 74.11 | |
| Llama-3.2-3B DenseBackbone=Llama-3.2-3B, Stage=Dense, #Experts=-2026.03 | 74.11 | |
| LayerMoE-S1Backbone=Llama-3.2-3B, Stage=Stage 1, #Experts=842026.03 | 73.89 | |
| NeuronMoE-ENBackbone=Llama-3.2-3B, Stage=Stage 1, #Experts=37 (-56%)2026.03 | 73.56 | |
| Llama-3.2-3BTraining Status=Continued pretraining, Tokenizer Strategy=prune+ext (16k, naive)2025.12 | 66 | |
| Llama-3.2-3BTraining Status=Continued pretraining, Tokenizer Strategy=prune+ext (16k, ours)2025.12 | 65 | |
| Llama-3.2-3BTraining Status=Continued pretraining, Tokenizer Strategy=default2025.12 | 64.7 | |
| NeuronMoE-S1Backbone=Qwen-1.5-1.8B, Stage=Stage 1, #Experts=36 (-50%)2026.03 | 58.67 | |
| LayerMoE-S1Backbone=Qwen-1.5-1.8B, Stage=Stage 1, #Experts=722026.03 | 54.89 | |
| Llama-3.2-1BTraining Status=No training2025.12 | 35.7 | |
| Llama-3.2-1BTraining Status=Continued pretraining, Tokenizer Strategy=prune+ext (16k, ours)2025.12 | 27.1 | |
| Llama-3.2-1BTraining Status=Continued pretraining, Tokenizer Strategy=default2025.12 | 26.9 | |
| Llama-3.2-1BTraining Status=Continued pretraining, Tokenizer Strategy=prune+ext (16k, naive)2025.12 | 25.7 |