Aggregate Out-of-Domain Evaluation on Average GPQA, LSAT, MMLU-1000
0.38Pass@1Agentica-24k
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Agentica-24kTraining Regime=Base Model2025.11 | 0.38 | 4,519 | 1.12 | |
| Agentica-24k + SLPO (Direct)Training Regime=SLPO (Direct)2025.11 | 0.36 | 2,189 | 2.47 | |
| ORION-AG-SLPOTraining Regime=SLPO, Base Model=Agentica-24k2025.11 | 0.35 | 739 | 6.54 | |
| DeepSeek R1 1.5b + SLPO (Direct)Training Regime=SLPO (Direct)2025.11 | 0.34 | 3,177 | 1.7 | |
| DeepSeek R1 1.5bTraining Regime=Base Model2025.11 | 0.33 | 5,484 | 1 | |
| ORION-DS-GRPOTraining Regime=GRPO, Base Model=DeepSeek R1 1.5b2025.11 | 0.33 | 373 | 12.96 | |
| ORION-DS-SLPOTraining Regime=SLPO, Base Model=DeepSeek R1 1.5b2025.11 | 0.33 | 431 | 14.45 | |
| ORION-AG-GRPOTraining Regime=GRPO, Base Model=Agentica-24k2025.11 | 0.33 | 361 | 13.3 | |
| L1 MaxTraining Regime=Baseline2025.11 | 0.32 | 453 | 10.61 | |
| ORION-DSTraining Regime=Mentalese SFT, Base Model=DeepSeek R1 1.5b2025.11 | 0.27 | 175 | 29.91 | |
| ORION-AGTraining Regime=Mentalese SFT, Base Model=Agentica-24k2025.11 | 0.27 | 133 | 36.05 |