Logical Reasoning Question Answering on LSAT
0.29Pass@1Agentica-24k
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Agentica-24kTraining Regime=Base Model2025.11 | 0.29 | 7,614 | 1.08 | |
| DeepScaleRmax context length=4K2026.04 | 0.2864 | 6,934 | — | |
| GRPO-4Kmax context length=4K2026.04 | 0.2845 | 5,025 | — | |
| LAPO-Imax context length=4K2026.04 | 0.2842 | 5,382 | — | |
| SASmax context length=4K, ratio r=0.32026.04 | 0.2832 | 4,312 | — | |
| ThinkPrune-4kmax context length=4K2026.04 | 0.2813 | 5,131 | — | |
| Agentica-24k + SLPO (Direct)Training Regime=SLPO (Direct)2025.11 | 0.28 | 4,077 | 2.01 | |
| L1-Maxmax context length=4K2026.04 | 0.2666 | 1,949 | — | |
| DeepSeek R1 1.5b + SLPO (Direct)Training Regime=SLPO (Direct)2025.11 | 0.25 | 5,608 | 1.46 | |
| ORION-AG-SLPOTraining Regime=SLPO, Base Model=Agentica-24k2025.11 | 0.25 | 1,200 | 6.83 | |
| DeepSeek R1 1.5bTraining Regime=Base Model2025.11 | 0.24 | 8,192 | 1 | |
| ORION-DS-GRPOTraining Regime=GRPO, Base Model=DeepSeek R1 1.5b2025.11 | 0.23 | 587 | 13.95 | |
| ORION-DS-SLPOTraining Regime=SLPO, Base Model=DeepSeek R1 1.5b2025.11 | 0.23 | 851 | 9.63 | |
| ORION-AG-GRPOTraining Regime=GRPO, Base Model=Agentica-24k2025.11 | 0.23 | 564 | 14.52 | |
| L1 MaxTraining Regime=Baseline2025.11 | 0.22 | 554 | 14.79 | |
| ORION-AGTraining Regime=Mentalese SFT, Base Model=Agentica-24k2025.11 | 0.21 | 201 | 40.76 | |
| ORION-DSTraining Regime=Mentalese SFT, Base Model=DeepSeek R1 1.5b2025.11 | 0.2 | 316 | 25.92 |