Machine Translation on ZHUANGBENCH (test)
48.3BLEU (easy)GPT-4
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| GPT-4Evaluation Protocol=DIPMT++, Few-shot Context=3-shot2024.02 | 48.3 | 43.2 | 35 | 31 | 22.8 | 21.8 | 31.9 | 29.1 | |
| Qwen-72B-chatEvaluation Protocol=DIPMT++, Few-shot Context=3-shot2024.02 | 43.6 | 39.4 | 30.1 | 28 | 18.7 | 19.9 | 27.3 | 26.4 | |
| Qwen-14B-chatEvaluation Protocol=DIPMT++, Few-shot Context=3-shot2024.02 | 34.3 | 31 | 21.6 | 20.8 | 9.1 | 9.7 | 19.5 | 17.8 | |
| GPT-3.5Evaluation Protocol=DIPMT++, Few-shot Context=3-shot2024.02 | 34.3 | 31.8 | 17.1 | 18 | 16.3 | 17.5 | 20.1 | 20.5 | |
| Qwen-72B-chatEvaluation Protocol=DIPMT++, Few-shot Context=3-shot2024.02 | 31.1 | 58.1 | 13.9 | 43.4 | 9.9 | 40.4 | 16.4 | 45.1 | |
| GPT-4Evaluation Protocol=DIPMT++, Few-shot Context=3-shot2024.02 | 30.7 | 57.3 | 15.1 | 45.4 | 7.4 | 41.7 | 15.7 | 46.1 | |
| Llama-2-7B-chatEvaluation Protocol=Finetune2024.02 | 29 | 54.7 | 5.9 | 34.2 | 1 | 25.5 | 6.5 | 33.2 | |
| Qwen-14B-chatEvaluation Protocol=DIPMT++, Few-shot Context=3-shot2024.02 | 28.2 | 55.8 | 10.6 | 42.5 | 4.8 | 34.9 | 12.6 | 41.7 | |
| Llama-2-70B-chatEvaluation Protocol=DIPMT++, Few-shot Context=3-shot2024.02 | 26.4 | 25.4 | 12.5 | 13.9 | 6.2 | 8.7 | 12.7 | 13.8 | |
| GPT-3.5Evaluation Protocol=DIPMT++, Few-shot Context=3-shot2024.02 | 25.7 | 53.8 | 11.3 | 42.6 | 7.6 | 39.6 | 13.3 | 43.5 | |
| Qwen-72B-chatEvaluation Protocol=DIPMT, Few-shot Context=3-shot2024.02 | 23 | 21.3 | 19.8 | 17.9 | 9.8 | 10.4 | 16.3 | 15.1 | |
| Llama-2-13B-chatEvaluation Protocol=DIPMT++, Few-shot Context=3-shot2024.02 | 22.8 | 20.6 | 9.1 | 11.5 | 6 | 7.5 | 10.7 | 11.6 | |
| Qwen-7B-chatEvaluation Protocol=DIPMT++, Few-shot Context=3-shot2024.02 | 22.4 | 25.2 | 11.7 | 15.5 | 5.5 | 8.5 | 11.4 | 14.3 | |
| Llama-2-13B-chatEvaluation Protocol=DIPMT++, Few-shot Context=3-shot2024.02 | 21.5 | 51.6 | 7.1 | 37.7 | 3 | 32.4 | 9 | 38.2 | |
| Llama-2-70B-chatEvaluation Protocol=DIPMT++, Few-shot Context=3-shot2024.02 | 21.5 | 50.7 | 8.2 | 41.3 | 2.4 | 34.6 | 9.3 | 40.6 | |
| Llama-2-7B-chatEvaluation Protocol=Finetune2024.02 | 21.2 | 19.4 | 10.1 | 11.2 | 5.8 | 7.6 | 11.3 | 11.2 | |
| Qwen-14B-chatEvaluation Protocol=DIPMT, Few-shot Context=3-shot2024.02 | 20.6 | 18.1 | 17 | 16 | 5.9 | 7.7 | 12.7 | 12.8 | |
| Llama-2-7B-chatEvaluation Protocol=DIPMT++, Few-shot Context=3-shot2024.02 | 19.8 | 19.3 | 9.3 | 10.9 | 4.3 | 6.5 | 9.7 | 10.7 | |
| Qwen-7B-chatEvaluation Protocol=DIPMT++, Few-shot Context=3-shot2024.02 | 17.6 | 48.4 | 5.1 | 37 | 2.8 | 31.5 | 7.6 | 36.9 | |
| Llama-2-7B-chatEvaluation Protocol=DIPMT++, Few-shot Context=3-shot2024.02 | 14.8 | 47.4 | 4.4 | 33.5 | 1.1 | 28 | 5.9 | 33.9 | |
| Qwen-7B-chatEvaluation Protocol=DIPMT, Few-shot Context=3-shot2024.02 | 12.8 | 13.9 | 10.2 | 11.2 | 3.3 | 5.7 | 8.3 | 9.3 | |
| Qwen-72B-chatEvaluation Protocol=DIPMT, Few-shot Context=3-shot2024.02 | 9.2 | 36.6 | 4.5 | 31.7 | 3.1 | 29.4 | 5.1 | 31.7 | |
| Qwen-14B-chatEvaluation Protocol=DIPMT, Few-shot Context=3-shot2024.02 | 8.4 | 35.4 | 4.2 | 30.8 | 2.1 | 26.1 | 4.4 | 29.6 | |
| Qwen-7B-chatEvaluation Protocol=DIPMT, Few-shot Context=3-shot2024.02 | 3.9 | 28.2 | 1 | 24.4 | 0.7 | 23.3 | 1.8 | 24.8 | |
| mT5-largeEvaluation Protocol=Finetune2024.02 | 3.3 | 21 | 0.8 | 15 | 0.3 | 12.5 | 1.1 | 15.1 | |
| mT5-largeEvaluation Protocol=Finetune2024.02 | 2.5 | 4.6 | 0.4 | 2.1 | 1.1 | 1.9 | 1.3 | 2.6 | |
| GPT-4Evaluation Protocol=Direct, Few-shot Context=3-shot2024.02 | 0.9 | 3.9 | 0.8 | 3.7 | 1.7 | 3.9 | 1.4 | 3.9 | |
| GPT-3.5Evaluation Protocol=Direct, Few-shot Context=3-shot2024.02 | 0.6 | 3.8 | 0.5 | 3.2 | 0.2 | 3.4 | 0.3 | 3.5 | |
| GPT-3.5Evaluation Protocol=Direct, Few-shot Context=3-shot2024.02 | 0.2 | 14 | 0.3 | 15.8 | 0.5 | 17.2 | 0.3 | 16.1 | |
| mT5-baseEvaluation Protocol=Finetune2024.02 | 0.2 | 1.2 | 0.2 | 1.4 | 0.2 | 1.3 | 0.2 | 1.3 | |
| mT5-baseEvaluation Protocol=Finetune2024.02 | 0.1 | 7.5 | 0.2 | 7.9 | 0 | 7.9 | 0.1 | 7.8 | |
| GPT-4Evaluation Protocol=Direct, Few-shot Context=3-shot2024.02 | 0.1 | 6.2 | 0.1 | 7.2 | 0.1 | 7.2 | 0 | 7 |