LLM Routing on MMLU-PRO, SUPERGPQA, MEDMCQA, and BBEH (test)
91.5MMLU-PRO Top-1 AccORACLE
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| ORACLEModel Pool Size=Pool Large2026.01 | 91.5 | — | 77.6 | — | 96.3 | — | 66.4 | — | 83 | — | |
| ORACLEModel Pool Size=Pool Small2026.01 | 84.4 | — | 67.8 | — | 91.2 | — | 56.7 | — | 75 | — | |
| TOP-1 / TOP-3Model Pool Size=Pool Large2026.01 | 79.4 | 80.9 | 54.4 | 54.7 | 92.7 | 88.5 | 34 | 34.3 | 65.1 | 64.6 | |
| SMOOTHIEModel Pool Size=Pool Large, Router Input Category=Query-only Routers, Training Query Generator=Exaone-3.5-7.8B-Instruct2026.01 | 74.8 | 78.2 | 43.6 | 48.1 | 76.1 | 79.6 | 31.6 | 31.6 | 56.5 | 59.4 | |
| CASCALModel Pool Size=Pool Large, Router Input Category=Query-only Routers, Training Query Generator=Exaone-3.5-7.8B-Instruct2026.01 | 74.6 | 78.7 | 42.6 | 49.2 | 81.6 | 84.1 | 32.1 | 32.3 | 57.7 | 61.1 | |
| LLMRANKModel Pool Size=Pool Large, Router Input Category=Query-answer Routers, Training Query Generator=Exaone-3.5-7.8B-Instruct2026.01 | 73.8 | — | 40.8 | — | 79.2 | — | 34.5 | — | 57.1 | — | |
| CASCAL-GTModel Pool Size=Pool Large, Router Input Category=Query-answer Routers, Training Query Generator=Exaone-3.5-7.8B-Instruct2026.01 | 71.9 | 77.5 | 39.2 | 44.5 | 80.7 | 84.1 | 34.3 | 32.6 | 56.5 | 59.7 | |
| AVENGERSModel Pool Size=Pool Large, Router Input Category=Query-answer Routers, Training Query Generator=Exaone-3.5-7.8B-Instruct2026.01 | 71.9 | — | 49.4 | — | 86.6 | — | 27.5 | — | 58.9 | — | |
| TOP-1 / TOP-3Model Pool Size=Pool Small2026.01 | 66.2 | 65.7 | 37.3 | 35.7 | 71 | 70.2 | 25.1 | 25 | 49.9 | 49.2 | |
| CASCALModel Pool Size=Pool Small, Router Input Category=Query-only Routers, Training Query Generator=Exaone-3.5-7.8B-Instruct2026.01 | 57.1 | 61.7 | 32.1 | 31.2 | 60.8 | 65.3 | 20.6 | 22.8 | 42.7 | 45.3 | |
| AVENGERSModel Pool Size=Pool Small, Router Input Category=Query-answer Routers, Training Query Generator=Exaone-3.5-7.8B-Instruct2026.01 | 54.7 | — | 25.9 | — | 59.8 | — | 22.4 | — | 40.7 | — | |
| SMOOTHIEModel Pool Size=Pool Small, Router Input Category=Query-only Routers, Training Query Generator=Exaone-3.5-7.8B-Instruct2026.01 | 54.3 | 57.8 | 25.7 | 27.6 | 59.7 | 65.3 | 21.7 | 22.7 | 40.4 | 43.4 | |
| LLMRANKModel Pool Size=Pool Small, Router Input Category=Query-answer Routers, Training Query Generator=Exaone-3.5-7.8B-Instruct2026.01 | 53.9 | — | 28.8 | — | 57 | — | 24.4 | — | 41 | — | |
| CASCAL-GTModel Pool Size=Pool Small, Router Input Category=Query-answer Routers, Training Query Generator=Exaone-3.5-7.8B-Instruct2026.01 | 52.9 | 58.9 | 25.6 | 30.8 | 56.8 | 65.3 | 21.4 | 23.4 | 39.2 | 44.6 |