Knowledge on MMLU-Redux (test)
69.8Knowledge AccuracyReM-MoA*
Evaluation Results
| Method | Links | |
|---|---|---|
| ReM-MoA*Width (N)=8, Depth (L)=3, Trend=↑↑, Distilled Reviewer=true2026.06 | 69.8 | |
| ReM-MoA*Width (N)=7, Depth (L)=3, Trend=↑↑, Distilled Reviewer=true2026.06 | 69.5 | |
| ReM-MoA*Width (N)=6, Depth (L)=3, Trend=↑↑, Distilled Reviewer=true2026.06 | 69.3 | |
| ReM-MoAWidth (N)=8, Depth (L)=3, Trend=↑→2026.06 | 69.2 | |
| ReM-MoAWidth (N)=7, Depth (L)=3, Trend=↑→2026.06 | 69 | |
| ReM-MoAWidth (N)=6, Depth (L)=3, Trend=↑→2026.06 | 68.8 | |
| ReM-MoA*Width (N)=5, Depth (L)=3, Trend=↑↑, Distilled Reviewer=true2026.06 | 68.7 | |
| ReM-MoAWidth (N)=5, Depth (L)=3, Trend=↑→2026.06 | 68.3 | |
| ReM-MoA*Width (N)=4, Depth (L)=3, Trend=↑↑, Distilled Reviewer=true2026.06 | 68.2 | |
| ReM-MoAWidth (N)=4, Depth (L)=3, Trend=↑→2026.06 | 67.8 | |
| ReM-MoA*Width (N)=3, Depth (L)=3, Trend=↑↑, Distilled Reviewer=true2026.06 | 67.4 | |
| ReM-MoAWidth (N)=3, Depth (L)=3, Trend=↑→2026.06 | 67 | |
| RMoAWidth (N)=4, Depth (L)=3, Trend=→2026.06 | 65.5 | |
| RMoAWidth (N)=5, Depth (L)=3, Trend=→2026.06 | 65.5 | |
| RMoAWidth (N)=3, Depth (L)=3, Trend=→2026.06 | 65.4 | |
| RMoAWidth (N)=6, Depth (L)=3, Trend=→2026.06 | 65.4 | |
| RMoAWidth (N)=7, Depth (L)=3, Trend=→2026.06 | 65.3 | |
| RMoAWidth (N)=8, Depth (L)=3, Trend=→2026.06 | 65.2 | |
| AttentionMoAWidth (N)=5, Depth (L)=3, Trend=→2026.06 | 65.2 | |
| AttentionMoAWidth (N)=6, Depth (L)=3, Trend=→2026.06 | 65.2 | |
| AttentionMoAWidth (N)=4, Depth (L)=3, Trend=→2026.06 | 65.1 | |
| AttentionMoAWidth (N)=7, Depth (L)=3, Trend=→2026.06 | 65.1 | |
| AttentionMoAWidth (N)=8, Depth (L)=3, Trend=→2026.06 | 65 | |
| AttentionMoAWidth (N)=3, Depth (L)=3, Trend=→2026.06 | 64.8 | |
| Standard MoAWidth (N)=3, Depth (L)=3, Trend=↓2026.06 | 58.2 | |
| Standard MoAWidth (N)=4, Depth (L)=3, Trend=↓2026.06 | 57.2 | |
| Standard MoAWidth (N)=5, Depth (L)=3, Trend=↓2026.06 | 56.5 | |
| Standard MoAWidth (N)=6, Depth (L)=3, Trend=↓2026.06 | 55.8 | |
| Standard MoAWidth (N)=7, Depth (L)=3, Trend=↓2026.06 | 55.2 | |
| Standard MoAWidth (N)=8, Depth (L)=3, Trend=↓2026.06 | 54.5 | |
| MoE w/ optimal ARModel Type=MoE, Activation rate=20.07, Compute=2.86e21, Data reuse=-2025.06 | 30.93 | |
| Dense baselineModel Type=Dense, Compute=5.45e212025.06 | 28.9 | |
| MoE w/ optimal ARModel Type=MoE, Activation rate=20.07, Compute=2.86e21, Data reuse=strict2025.06 | 23.73 |