Reasoning on MMLU
82AccuracyORCH
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| ORCHrouting=EMA-router2026.02 | 82 | 12,315 | 7 | |
| ORCHagents=3, routing=fixed route2026.02 | 81.3 | 11,775 | 4.6 | |
| Llama 3.1 8BAttn CR=N/A2025.08 | 62.9 | — | — | |
| Matrix PCABase Model=Llama 3.1 8B, Attn CR=20%2025.08 | 60.5 | — | — | |
| SVD-LLMBase Model=Llama 3.1 8B, Attn CR=20%2025.08 | 59 | — | — | |
| TiToKTransfer=Mistral 7B → Mistral 7B, Evaluation Mode=Zero-shot2025.10 | 56.1 | — | — | |
| KD (+MinED)Transfer=Mistral 7B → Mistral 7B, Evaluation Mode=Zero-shot2025.10 | 56 | — | — | |
| VanillaTransfer=Mistral 7B → Mistral 7B, Evaluation Mode=Zero-shot2025.10 | 55.7 | — | — | |
| Llama 3.2 3BAttn CR=N/A2025.08 | 54.3 | — | — | |
| TransLoRATransfer=Mistral 7B → Mistral 7B, Evaluation Mode=Zero-shot2025.10 | 53.4 | — | — | |
| SVD-LLMBase Model=Llama 3.2 3B, Attn CR=20%2025.08 | 50.9 | — | — | |
| Matrix PCABase Model=Llama 3.2 3B, Attn CR=20%2025.08 | 50.6 | — | — | |
| TiToKTransfer=Mistral 7B → Llama3 8B, Evaluation Mode=Zero-shot2025.10 | 48.5 | — | — | |
| KD (+MinED)Transfer=Mistral 7B → Llama3 8B, Evaluation Mode=Zero-shot2025.10 | 48.2 | — | — | |
| TiToKTransfer=Llama3 3B → Llama3 8B, Evaluation Mode=Zero-shot2025.10 | 47.8 | — | — | |
| KD (+MinED)Transfer=Llama3 3B → Llama3 8B, Evaluation Mode=Zero-shot2025.10 | 47.7 | — | — | |
| TiToKTransfer=Llama2 7B → Llama3 8B, Evaluation Mode=Zero-shot2025.10 | 47.7 | — | — | |
| KD (+MinED)Transfer=Llama2 7B → Llama3 8B, Evaluation Mode=Zero-shot2025.10 | 47.6 | — | — | |
| TransLoRATransfer=Mistral 7B → Llama3 8B, Evaluation Mode=Zero-shot2025.10 | 47.3 | — | — | |
| VanillaTransfer=Mistral 7B → Llama3 8B, Evaluation Mode=Zero-shot2025.10 | 46.9 | — | — | |
| VanillaTransfer=Llama3 3B → Llama3 8B, Evaluation Mode=Zero-shot2025.10 | 46.9 | — | — | |
| VanillaTransfer=Llama2 7B → Llama3 8B, Evaluation Mode=Zero-shot2025.10 | 46.9 | — | — | |
| TransLoRATransfer=Llama2 7B → Llama3 8B, Evaluation Mode=Zero-shot2025.10 | 46.8 | — | — | |
| TransLoRATransfer=Llama3 3B → Llama3 8B, Evaluation Mode=Zero-shot2025.10 | 46.7 | — | — | |
| Llama 3.2 1BAttn CR=N/A2025.08 | 37 | — | — | |
| Matrix PCABase Model=Llama 3.2 1B, Attn CR=20%2025.08 | 33 | — | — | |
| SVD-LLMBase Model=Llama 3.2 1B, Attn CR=20%2025.08 | 29.5 | — | — | |
| Matrix PCABase Model=Llama 3.2 1B, Attn CR=30%2025.08 | 28.8 | — | — | |
| Self-Improving PretrainingTraining Dataset=SlimPajama, Pretraining Objective=Quality2026.01 | 28.3 | — | — | |
| Self-Improving PretrainingTraining Dataset=SlimPajama, Pretraining Objective=Factuality2026.01 | 27.9 | — | — | |
| SVD-LLMBase Model=Llama 3.2 1B, Attn CR=30%2025.08 | 27.6 | — | — | |
| Llama Pretrain BaselineTraining Dataset=RedPajama, Pretraining Objective=Safety2026.01 | 27.5 | — | — | |
| Llama Pretrain BaselineTraining Dataset=SlimPajama, Pretraining Objective=Standard next token prediction2026.01 | 26.7 | — | — | |
| Self-Improving PretrainingTraining Dataset=RedPajama, Pretraining Objective=Safety2026.01 | 26.7 | — | — | |
| Llama BaseTraining Dataset=Original Llama Pretraining, Pretraining Objective=Standard next token prediction2026.01 | 26.4 | — | — |