Machine Translation Evaluation on TAC summary-level 2008-2011 (test)
67.3Pearson Correlation (Pyramid)FrugalScore
Evaluation results
| Method | Links | ||
|---|---|---|---|
| FrugalScoreBackbone=BERT-Tiny, Teacher=MoverScore (BERT-Base), Params=4.4M, Runtime=1m 28s2021.10 | 67.3 | 55.1 | |
| MoverScoreBackbone=BERT-Base, Params=110M, Runtime=301m 29s2021.10 | 66.5 | 55.4 | |
| FrugalScoreBackbone=BERT-Medium, Teacher=MoverScore (BERT-Base), Params=41.7M, Runtime=3m 41s2021.10 | 66.2 | 55.1 | |
| FrugalScoreBackbone=BERT-Small, Teacher=BERTScore (DeBERTa-XLarge), Params=29.1M, Runtime=2m 29s2021.10 | 66 | 54.9 | |
| FrugalScoreBackbone=BERT-Small, Teacher=MoverScore (BERT-Base), Params=29.1M, Runtime=2m 29s2021.10 | 65.9 | 54.7 | |
| FrugalScoreBackbone=BERT-Medium, Teacher=BERTScore (DeBERTa-XLarge), Params=41.7M, Runtime=3m 41s2021.10 | 65.5 | 54.9 | |
| FrugalScoreBackbone=BERT-Tiny, Teacher=BERTScore (BERT-Base), Params=4.4M, Runtime=1m 28s2021.10 | 64.9 | 53.5 | |
| FrugalScoreBackbone=BERT-Medium, Teacher=BERTScore (BERT-Base), Params=41.7M, Runtime=3m 41s2021.10 | 64.8 | 54.2 | |
| BERTScoreBackbone=BERT-Base, Params=110M, Runtime=3m 28s2021.10 | 64.7 | 54.7 | |
| FrugalScoreBackbone=BERT-Small, Teacher=BERTScore (BERT-Base), Params=29.1M, Runtime=2m 29s2021.10 | 64.7 | 53.7 | |
| BERTScoreBackbone=DeBERTa-XLarge, Params=900M, Runtime=6m 20s2021.10 | 64.5 | 56 | |
| BERTScoreBackbone=RoBERTa-Large, Params=355M, Runtime=5m 17s2021.10 | 64.2 | 55.4 | |
| FrugalScoreBackbone=BERT-Small, Teacher=BERTScore (RoBERTa-Large), Params=29.1M, Runtime=2m 29s2021.10 | 64.1 | 53.8 | |
| FrugalScoreBackbone=BERT-Medium, Teacher=BERTScore (RoBERTa-Large), Params=41.7M, Runtime=3m 41s2021.10 | 63.9 | 52.1 | |
| BERTScoreBackbone=BERT-Medium, Params=41.7M, Runtime=2m 28s2021.10 | 62.7 | 52.4 | |
| FrugalScoreBackbone=BERT-Tiny, Teacher=BERTScore (DeBERTa-XLarge), Params=4.4M, Runtime=1m 28s2021.10 | 61.7 | 51 | |
| BERTScoreBackbone=BERT-Small, Params=29.1M, Runtime=2m 20s2021.10 | 61.6 | 51.5 | |
| FrugalScoreBackbone=BERT-Tiny, Teacher=BERTScore (RoBERTa-Large), Params=4.4M, Runtime=1m 28s2021.10 | 60 | 50.1 | |
| BERTScoreBackbone=BERT-Tiny, Params=4.4M, Runtime=1m 27s2021.10 | 55.4 | 47.5 |