Machine Translation on CALT (test)

49.26BLEU-4 (fil)Gemini-2.5 Flash

Evaluation results

Method
Gemini-2.5 Flash2026.04	49.26	48.73	35.79	36.91	39.24	47.48	47.2	35.03	36.07	38.63	35.89	30.83	13.53	23.92	24.62	42.68	41.93	30.61	32.09	34.66	70.14	70.87	60.21	61.85	60.31	71.67	72.77	63.93	62.39	60.88
DeepSeek-V32026.04	46.19	41.83	25.57	26.68	41.29	44.12	41.38	25.25	26.29	40.9	35.15	30.06	12.75	22.95	28.27	39.8	36.95	22.37	24.43	36.86	68.04	67.1	51.41	53.66	67.43	70.02	68.81	54.82	54.21	68.87
GPT-4o2026.04	45.26	47.77	28.1	30.48	45.03	43.6	45.81	27.38	29.53	44.5	33.62	31.4	12.33	23.57	28.25	39.36	41.13	24.2	26.76	39.62	67.8	70.34	53.66	56.59	69.44	68.59	70.84	56.27	57.04	70.41
Qwen-2.5 32B2026.04	43.56	47.87	23.27	20.43	46.23	42.01	46.61	22.48	19.5	44.63	34.97	31.85	11.84	20.61	29.08	37.77	41.35	20.36	18.13	39.8	66.61	70.57	46.72	43.29	69.76	67.73	71.73	50.5	45.09	71.13
Claude-3.5 Sonnet2026.04	42.97	47.35	32.2	30.92	45.14	43.38	46.54	32.65	30.14	44.55	35.03	31.17	14	24.44	28.45	38.71	41.3	28.38	28.79	39.65	67.52	70.29	58.53	58.88	69.23	68.28	71.55	60.72	57.68	70.4
DeepSeek-R1 32B2026.04	37.98	43.29	12.97	8.87	42.57	37.14	42.27	12.71	8.43	41.45	32.4	29.92	10.04	16.31	28.11	33.62	37.62	12.77	9.81	36.97	61.6	66.75	34.07	27.37	66.95	63.32	68.54	36.09	27.36	68.65
Qwen 2.5-7B2026.04	30.21	36.28	6.17	6.43	35.22	29.75	36.41	5.67	5.42	35.07	31.99	28.56	9.86	16.41	27.81	27.88	33.68	7.39	7.95	33.28	54.51	62.56	20.84	23.25	63	54.95	62.82	22.24	23.35	63.28
MERIT-3B2026.04	29.71	34.73	5.15	4.56	31.2	27.2	33.16	4.28	3.54	29.25	31.91	26.63	8.4	13.68	21.18	25.52	30.22	5.81	5.53	26.98	49.88	56.46	16.55	16.93	50.5	46.7	45.58	11.45	16.12	32.87
NLLB-200 3.3B2026.04	25.05	25.27	15.86	20.83	25.3	24.21	23.64	15.19	20.13	22.97	31.45	25.73	11.49	18.68	25.51	22.48	23.57	14.92	18.69	22.43	46.1	45.28	36.27	42.28	45.73	48.61	54.34	44.8	48.93	52.59
DeepSeek-R1 7B2026.04	14.77	20.94	0.79	0.37	12.53	15.31	24.17	0.86	0.46	16.16	24.58	23.32	8.67	5.67	18.52	15.43	21.6	2.2	1.35	14.87	36.02	49.35	8.45	6.46	37.61	34.22	49.07	1.46	4.1	36.54
M2M-100 1.2B2026.04	2.13	9.53	0.05	0	3.33	1.32	9.47	0.01	0	3.19	9.88	21.65	4.69	0	13.01	5.16	18.21	0.26	0.01	6.49	4.65	14	0.57	0.11	6.1	16.71	1.68	10.04	16.34	9.51
DeepSeek-R1 1.5B2026.04	0.07	0.08	0.05	1.06	0.05	0.03	0.06	0.01	0.11	0.02	0.77	0.9	1.8	3.8	0.55	0.33	0.38	0.36	1.87	0.24	1.99	2.34	2.17	3.8	1.43	27.97	26.07	20.48	19.68	29.48