Machine Translation on 49 low-performing Closely-Related Languages (CRLs) (test)
49.5AUS Scoreoff-the-shelf
Evaluation Results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| off-the-shelfParadigm=Baselines, Adaptation Method=off-the-shelf2025.01 | 49.5 | — | 58.4 | 60.7 | 76.5 | 47.6 | 61.1 | 74 | |
| off-the-shelfBackbone=M2M, Condition=Baseline2025.01 | 10.8 | 20.2 | 9.6 | 5 | 12.6 | 5.9 | 10.7 | — | |
| off-the-shelfBackbone=Aya-23, Condition=Baseline2025.01 | 7.6 | 23.5 | 17.1 | 7.9 | 20.4 | 7.3 | 14 | — | |
| cloudParadigm=M→D, Adaptation Method=cloud2025.01 | 5.5 | — | 12.7 | 8.2 | 3 | 7.8 | 6.7 | 2.9 | |
| shellParadigm=M→D, Adaptation Method=shell2025.01 | 5.3 | — | 12.7 | 8.3 | 2.8 | 7.5 | 6.6 | 2.7 | |
| M↔DBackbone=Aya-23, variant=-cloud-all2025.01 | 5.1 | -4.5 | 7.4 | 2.1 | 4.6 | 2.5 | 2.9 | — | |
| M↔DBackbone=Aya-23, variant=-cloud-func2025.01 | 3.8 | -0.8 | 9.8 | 3.1 | 6 | 2.8 | 4.1 | — | |
| D→MBackbone=Aya-23, variant=-all2025.01 | 3.7 | -6.9 | 3.1 | 0 | 4.4 | -3.5 | 0.2 | — | |
| M↔DBackbone=Aya-23, variant=-cloud-cont2025.01 | 3.3 | -4.3 | 5.3 | 1.8 | 1.6 | 2.6 | 1.7 | — | |
| D→MBackbone=Aya-23, variant=-func2025.01 | 2.4 | -1.8 | 4.7 | 0.3 | 5.6 | -3.3 | 1.3 | — | |
| allParadigm=D→M, Adaptation Method=all2025.01 | 2.4 | — | — | 1.3 | -3.4 | 1.7 | 1.6 | -7.7 | |
| M→DBackbone=Aya-23, variant=-cloud2025.01 | 2.2 | 0.1 | 7.7 | 3.2 | 2.7 | 3.5 | 3.2 | — | |
| M→DBackbone=Aya-23, variant=-shell2025.01 | 2.1 | 0.4 | 7.9 | 2.9 | 2.8 | 3 | 3.2 | — | |
| M→DBackbone=M2M, variant=-shell2025.01 | 1.9 | 1 | 11.5 | 2.5 | 5 | 2.7 | 4.1 | — | |
| cloud-funcParadigm=M↔D, Adaptation Method=cloud-func2025.01 | 1.6 | — | — | 8 | 4.1 | 9.2 | 5.4 | 13.4 | |
| M↔DBackbone=M2M, variant=-cloud-cont2025.01 | 1.4 | -1.5 | 9 | 2.6 | 5.1 | 3.7 | 3.4 | — | |
| M→DBackbone=M2M, variant=-cloud2025.01 | 1.3 | 0.9 | 9.1 | 1.6 | 5.4 | 2.7 | 3.5 | — | |
| M↔DBackbone=M2M, variant=-cloud-func2025.01 | 1.1 | 1.3 | 14.1 | 2.3 | 13 | 4.6 | 6.1 | — | |
| M↔DBackbone=M2M, variant=-cloud-all2025.01 | 0.9 | -1.2 | 13.1 | 3.9 | 12.2 | 6.1 | 5.8 | — | |
| D→MBackbone=Aya-23, variant=-cont2025.01 | 0.8 | -6.2 | -1 | -0.6 | -1.1 | -3.8 | -1.9 | — | |
| ft-hrlnBackbone=M2M, Condition=Baseline adaptation2025.01 | 0.3 | 0.3 | 0.5 | -0.1 | 0.4 | 0.9 | 0.4 | — | |
| ft-randaugBackbone=M2M, Condition=Baseline adaptation2025.01 | 0.2 | 0.2 | 0.6 | 0.2 | 0.4 | 0.9 | 0.4 | — | |
| D→MBackbone=M2M, variant=-all2025.01 | 0 | -1.4 | 9.8 | 3 | 11.4 | 5.6 | 4.7 | — | |
| D→MBackbone=M2M, variant=-func2025.01 | -0.1 | 0.8 | 8 | 1 | 12 | 3.7 | 4.3 | — | |
| D→MBackbone=M2M, variant=-cont2025.01 | -0.4 | -2.2 | 1.6 | 1.4 | 0.3 | 2.5 | 0.5 | — | |
| ft-randaugBackbone=Aya-23, Condition=Baseline adaptation2025.01 | -0.4 | -2.3 | 1.9 | 1.1 | -1.8 | 2.1 | 0.1 | — | |
| ft-hrlnBackbone=Aya-23, Condition=Baseline adaptation2025.01 | -0.6 | -1.8 | 1.4 | 1.2 | -1.8 | 2.3 | 0.1 | — | |
| funcParadigm=D→M, Adaptation Method=func2025.01 | -1.3 | — | — | 0.1 | 3.1 | 4 | 2 | 4.5 | |
| randaug-shellParadigm=Baselines, Adaptation Method=randaug-shell2025.01 | -1.6 | — | 0.6 | 4 | 4.3 | -1.8 | 1.9 | 0.3 | |
| cloud-contParadigm=M↔D, Adaptation Method=cloud-cont2025.01 | -1.7 | — | — | 5.9 | 1.6 | 7.4 | 7.4 | 9.2 | |
| ft-hrlParadigm=Baselines, Adaptation Method=ft-hrl2025.01 | -2 | — | 2.8 | 4.3 | -1.7 | 6.2 | 1.6 | 0.2 | |
| cloud-allParadigm=M↔D, Adaptation Method=cloud-all2025.01 | -2.8 | — | — | 5.6 | 2.8 | 7.4 | 7.4 | 10 | |
| contParadigm=D→M, Adaptation Method=cont2025.01 | -6.5 | — | — | -3.7 | -1.3 | 0.4 | -2.1 | -2.1 |