Literary Translation on O. Henry Collection (out-of-domain)
86.22CRF ScoreClaude Opus 4.5
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Claude Opus 4.5Params=–, trained on MetaphorTrans=false, thinking-enabled=false2026.06 | 86.22 | 4.26 | 80.65 | |
| Kimi K2Params=1T (32B), trained on MetaphorTrans=false, thinking-enabled=true2026.06 | 85.55 | 4.16 | 78.56 | |
| GPT-5.2Params=–, trained on MetaphorTrans=false, thinking-enabled=false2026.06 | 84.47 | 4.12 | 77.74 | |
| Claude Sonnet 4.5Params=–, trained on MetaphorTrans=false, thinking-enabled=false2026.06 | 84.27 | 4.05 | 76.5 | |
| DeepSeek V3.1Params=671B (37B), trained on MetaphorTrans=false, thinking-enabled=false2026.06 | 83.56 | 4.12 | 77.94 | |
| Qwen3-235B-A22BParams=235B (22B), variant=Instruct, trained on MetaphorTrans=false, thinking-enabled=false2026.06 | 81.61 | 3.94 | 74.01 | |
| LitMT-14BParams=14B, trained on MetaphorTrans=true, thinking-enabled=false2026.06 | 79.65 | 3.94 | 73.71 | |
| LitMT-8BParams=8B, trained on MetaphorTrans=true, thinking-enabled=false2026.06 | 76.42 | 3.77 | 70.38 | |
| Qwen3-32BParams=32B, trained on MetaphorTrans=false, thinking-enabled=false2026.06 | 75.51 | 3.56 | 65.81 | |
| ExTrans-7BParams=7B, trained on MetaphorTrans=true, thinking-enabled=true2026.06 | 75.04 | 3.62 | 67.39 | |
| DeepTrans-7BParams=7B, trained on MetaphorTrans=true, thinking-enabled=true2026.06 | 74.58 | 3.46 | 64.17 | |
| DRT-14BParams=14B, trained on MetaphorTrans=true, thinking-enabled=true2026.06 | 73.67 | 3.52 | 65.19 | |
| MiniMax M2Params=230B (10B), trained on MetaphorTrans=false, thinking-enabled=false2026.06 | 73.49 | 3.35 | 61.79 | |
| Qwen3-14BParams=14B, trained on MetaphorTrans=false, thinking-enabled=false2026.06 | 73.38 | 3.49 | 64.32 | |
| gpt-oss-120BParams=120B (5.1B), trained on MetaphorTrans=false, thinking-enabled=true, reasoning effort=medium2026.06 | 71.35 | 3.34 | 61.49 | |
| DRT-7BParams=7B, trained on MetaphorTrans=true, thinking-enabled=true2026.06 | 70.3 | 3.31 | 60.67 | |
| Qwen3-8BParams=8B, trained on MetaphorTrans=false, thinking-enabled=false2026.06 | 70.2 | 3.3 | 60.56 | |
| Llama3.3-70BParams=70B, trained on MetaphorTrans=false, thinking-enabled=false2026.06 | 68.57 | 3.18 | 57.96 | |
| LMT-60-8BParams=8B, trained on MetaphorTrans=false, thinking-enabled=false2026.06 | 66.78 | 3.15 | 56.88 | |
| gpt-oss-20BParams=20B (3.6B), trained on MetaphorTrans=false, thinking-enabled=true, reasoning effort=medium2026.06 | 63.68 | 2.91 | 51.63 | |
| X-ALMA-13B (G6)Params=13B, trained on MetaphorTrans=false, thinking-enabled=false2026.06 | 59.05 | 2.78 | 48.59 |