Continual Model Routing on CMRBench v1 (test)
85.7D-AccFrom Scratch CARvE
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| From Scratch CARvESetting=Upper bound (non-continual)2026.05 | 85.7 | 50.4 | 56.4 | 0.8 | 0.8 | 0.5 | |
| Joint Training CARvESetting=Upper bound (non-continual)2026.05 | 84.6 | 49.5 | 55.4 | — | — | — | |
| Cumulative CARvESetting=Upper bound (non-continual)2026.05 | 83.5 | 48.1 | 54.2 | 0.8 | 1 | 0.5 | |
| CARvEBackbone=LLaMA2-7B, Replay Percentage=20%2026.05 | 82.9 | 46.4 | 51.9 | 6.4 | 7.3 | 3 | |
| Cumulative Zero-ShotSetting=Upper bound (non-continual)2026.05 | 81.9 | 45.9 | 54.6 | 0.6 | 1.2 | 0.8 | |
| CARvEBackbone=Qwen2.5-7B, Replay Percentage=10%2026.05 | 81.5 | 42.4 | 48.1 | 11.2 | 12.3 | 6.2 | |
| CARvEBackbone=LLaMA2-7B, Replay Percentage=10%2026.05 | 80.7 | 43.1 | 48.5 | 11.4 | 12.5 | 5.9 | |
| CARvE EWCBackbone=LLaMA2-7B, Replay Percentage=10%2026.05 | 80.5 | 42.2 | 47.7 | 9.1 | 9.8 | 6.1 | |
| From Scratch Zero-ShotSetting=Upper bound (non-continual)2026.05 | 80.1 | 43.9 | 53 | 0.2 | 1.1 | 1.9 | |
| Joint Training Zero-ShotSetting=Upper bound (non-continual)2026.05 | 79.3 | 43.1 | 51.6 | — | — | — | |
| CARvEBackbone=Qwen3-4B, Replay Percentage=10%2026.05 | 78.6 | 36.7 | 42.1 | 13.5 | 14.3 | 9.1 | |
| CARvEBackbone=LLaMA2-7B, Replay Percentage=5%2026.05 | 78.5 | 40.5 | 45.4 | 15.9 | 17.4 | 10.1 | |
| Random ReplayBackbone=Qwen2.5-7B, Replay Percentage=10%2026.05 | 78.2 | 40.7 | 49.2 | 14.2 | 16.5 | 10.3 | |
| Random ReplayBackbone=LLaMA2-7B, Replay Percentage=20%2026.05 | 78.1 | 41.3 | 49.8 | 8.6 | 11.1 | 7.8 | |
| Joint Training RATSetting=Upper bound (non-continual)2026.05 | 77.3 | 39.9 | 48.4 | — | — | — | |
| Random ReplayBackbone=Qwen3-4B, Replay Percentage=10%2026.05 | 77.2 | 39.6 | 47.9 | 14.9 | 18.2 | 11.2 | |
| Random ReplayBackbone=LLaMA2-7B, Replay Percentage=10%2026.05 | 75.9 | 39.1 | 47.3 | 14 | 16.8 | 13.1 | |
| Random ReplayBackbone=LLaMA2-7B, Replay Percentage=5%2026.05 | 70.2 | 36.5 | 44.2 | 19.5 | 23.2 | 24 | |
| EWCBackbone=LLaMA2-7B2026.05 | 66.2 | 31.3 | 38.4 | 26.6 | 32.7 | 31.4 | |
| Sequential FinetuningBackbone=LLaMA2-7B2026.05 | 64.3 | 28 | 34.8 | 34.6 | 41.8 | 37.2 | |
| LwFBackbone=LLaMA2-7B2026.05 | 56.4 | 28.8 | 35.9 | 20.8 | 25.7 | 39.5 | |
| HuggingGPTBackbone=Qwen3-32B, Setting=LLM-controller baseline2026.05 | 51.7 | — | — | — | — | — | |
| BGE-M3Setting=Retrieval baseline2026.05 | 44 | 13.6 | 16.2 | 2.5 | 3.9 | 3.3 | |
| Gorilla RAGSetting=Retrieval baseline2026.05 | 43 | 6.7 | 10.4 | 0 | 0.8 | 0.1 | |
| Model Merging (TIES)Backbone=LLaMA2-7B2026.05 | 28.6 | 7.6 | 10.9 | 15 | 19.7 | 32.6 |