List your company
ResearchFormerly WizWand

Reasoning on MMLU

82AccuracyORCH
24.17639.18854.269.212Aug 6, 2025Sep 5, 2025Oct 5, 2025Nov 4, 2025Dec 4, 2025Jan 3, 2026Feb 2, 2026
Updated 8mo ago

Evaluation results

MethodLinks
ORCHrouting=EMA-router2026.028212,3157
ORCHagents=3, routing=fixed route2026.0281.311,7754.6
Llama 3.1 8BAttn CR=N/A2025.0862.9——
Matrix PCABase Model=Llama 3.1 8B, Attn CR=20%2025.0860.5——
SVD-LLMBase Model=Llama 3.1 8B, Attn CR=20%2025.0859——
TiToKTransfer=Mistral 7B → Mistral 7B, Evaluation Mode=Zero-shot2025.1056.1——
KD (+MinED)Transfer=Mistral 7B → Mistral 7B, Evaluation Mode=Zero-shot2025.1056——
VanillaTransfer=Mistral 7B → Mistral 7B, Evaluation Mode=Zero-shot2025.1055.7——
Llama 3.2 3BAttn CR=N/A2025.0854.3——
TransLoRATransfer=Mistral 7B → Mistral 7B, Evaluation Mode=Zero-shot2025.1053.4——
SVD-LLMBase Model=Llama 3.2 3B, Attn CR=20%2025.0850.9——
Matrix PCABase Model=Llama 3.2 3B, Attn CR=20%2025.0850.6——
TiToKTransfer=Mistral 7B → Llama3 8B, Evaluation Mode=Zero-shot2025.1048.5——
KD (+MinED)Transfer=Mistral 7B → Llama3 8B, Evaluation Mode=Zero-shot2025.1048.2——
TiToKTransfer=Llama3 3B → Llama3 8B, Evaluation Mode=Zero-shot2025.1047.8——
KD (+MinED)Transfer=Llama3 3B → Llama3 8B, Evaluation Mode=Zero-shot2025.1047.7——
TiToKTransfer=Llama2 7B → Llama3 8B, Evaluation Mode=Zero-shot2025.1047.7——
KD (+MinED)Transfer=Llama2 7B → Llama3 8B, Evaluation Mode=Zero-shot2025.1047.6——
TransLoRATransfer=Mistral 7B → Llama3 8B, Evaluation Mode=Zero-shot2025.1047.3——
VanillaTransfer=Mistral 7B → Llama3 8B, Evaluation Mode=Zero-shot2025.1046.9——
VanillaTransfer=Llama3 3B → Llama3 8B, Evaluation Mode=Zero-shot2025.1046.9——
VanillaTransfer=Llama2 7B → Llama3 8B, Evaluation Mode=Zero-shot2025.1046.9——
TransLoRATransfer=Llama2 7B → Llama3 8B, Evaluation Mode=Zero-shot2025.1046.8——
TransLoRATransfer=Llama3 3B → Llama3 8B, Evaluation Mode=Zero-shot2025.1046.7——
Llama 3.2 1BAttn CR=N/A2025.0837——
Matrix PCABase Model=Llama 3.2 1B, Attn CR=20%2025.0833——
SVD-LLMBase Model=Llama 3.2 1B, Attn CR=20%2025.0829.5——
Matrix PCABase Model=Llama 3.2 1B, Attn CR=30%2025.0828.8——
Self-Improving PretrainingTraining Dataset=SlimPajama, Pretraining Objective=Quality2026.0128.3——
Self-Improving PretrainingTraining Dataset=SlimPajama, Pretraining Objective=Factuality2026.0127.9——
SVD-LLMBase Model=Llama 3.2 1B, Attn CR=30%2025.0827.6——
Llama Pretrain BaselineTraining Dataset=RedPajama, Pretraining Objective=Safety2026.0127.5——
Llama Pretrain BaselineTraining Dataset=SlimPajama, Pretraining Objective=Standard next token prediction2026.0126.7——
Self-Improving PretrainingTraining Dataset=RedPajama, Pretraining Objective=Safety2026.0126.7——
Llama BaseTraining Dataset=Original Llama Pretraining, Pretraining Objective=Standard next token prediction2026.0126.4——