Budgeted hybrid routing on Colloquial domain (test)
1Spearman Correlation (Avg.)Gain Oracle
Evaluation results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gain OracleBudget p=0.32026.04 | 1 | 100 | 100 | 100 | 100 | 100 | 15.2 | 7.77 | 28.51 | 7.3 | 17.22 | |
| Quality OracleBudget p=0.32026.04 | 0.61 | 72.73 | 69.8 | 66.98 | 74.07 | 80.06 | 12.94 | 6.11 | 24.36 | 6.35 | 14.96 | |
| RouteLMTType=Learned, Form=In-model, Target=Δ, Route=top-p, Extra model=No, Budget p=0.32026.04 | 0.3 | 51.28 | 48.72 | 50.47 | 52.99 | 52.96 | 8.87 | 3.68 | 18.5 | 3.9 | 9.41 | |
| RouteLMT-QType=Learned, Form=In-model, Target=Q, Route=bottom-p, Extra model=No, Budget p=0.32026.04 | 0.29 | 50.91 | 49 | 48.91 | 51.85 | 53.89 | 8.92 | 3.48 | 18.17 | 3.86 | 10.19 | |
| sentinel-src-24Type=Learned, Form=External, Target=Q, Route=bottom-p, Extra model=Yes, Budget p=0.32026.04 | 0.26 | 49.28 | 44.73 | 48.6 | 52.71 | 51.09 | 8.08 | 3.12 | 17.72 | 2.76 | 8.72 | |
| XLM-R-ΔType=Learned, Form=External, Target=Δ, Route=top-p, Extra model=Yes, Budget p=0.32026.04 | 0.24 | 47.88 | 49 | 47.66 | 45.01 | 49.84 | 8.09 | 3.52 | 17.47 | 2.21 | 9.15 | |
| LengthType=Heuristic, Route=top-p, Extra model=No, Budget p=0.32026.04 | 0.21 | 43.74 | 45.3 | 45.79 | 45.87 | 38.01 | 7.19 | 3.3 | 17.17 | 2.16 | 6.13 | |
| sentinel-src-25Type=Learned, Form=External, Target=Q, Route=bottom-p, Extra model=Yes, Budget p=0.32026.04 | 0.21 | 45.36 | 43.59 | 43.93 | 48.43 | 45.48 | 7.4 | 2.75 | 16.36 | 3.19 | 7.32 | |
| XLM-R-QType=Learned, Form=External, Target=Q, Route=bottom-p, Extra model=Yes, Budget p=0.32026.04 | 0.19 | 45.32 | 47.86 | 46.11 | 42.45 | 44.86 | 7.39 | 3.44 | 16.66 | 1.84 | 7.62 | |
| RarityType=Heuristic, Score semantics=surprisal, Route=top-p, Extra model=No, Budget p=0.32026.04 | 0.14 | 38.41 | 33.9 | 36.45 | 42.17 | 41.12 | 6.47 | 1.99 | 14.15 | 2.85 | 6.88 | |
| EntropyType=Heuristic, Score semantics=entropy, Route=top-p, Extra model=No, Budget p=0.32026.04 | 0.03 | 34.2 | 32.76 | 31.46 | 36.75 | 35.83 | 5.79 | 1.54 | 13.71 | 2.28 | 5.64 | |
| RandomBudget p=0.3, Route=top-p, Extra model=No2026.04 | 0 | 30 | 30 | 30 | 30 | 30 | 4.86 | 1.49 | 11.84 | 1.38 | 4.74 |