Intrinsic Tokenization on FLORES+ (test) benchmark leaderboard · SOTA2 Research

Intrinsic Tokenization on FLORES+ (test)

69.2Vocabulary UtilisationBias

Evaluation Results

Method	Links
Bias2026.05		69.2	0.0093	6.79	4.29	156.7	1.96	121.9	0.0082
BPE2026.05		67.7	0.009	6.75	4.34	131.1	1.91	123.5	0.0081
Det2026.05		63.8	0.0085	6.73	4.31	127.8	1.93	123.4	0.0081
Int2026.05		62	0.0051	6.41	4.54	57.6	1.52	136.7	0.0073
Bias2026.05		48.1	0.0154	7.37	4.84	233.9	2.19	102.9	0.0097
BPE2026.05		46.5	0.0142	7.25	4.9	192.2	2.11	107.9	0.0093
Det2026.05		44.3	0.0138	7.29	4.86	196.3	2.16	105.8	0.0095
Int2026.05		42.7	0.0084	6.76	4.8	89.1	1.77	124.4	0.008
Bias2026.05		29.4	0.0216	7.85	5.27	322.5	2.39	89.4	0.0112
BPE2026.05		28.7	0.0201	7.71	5.27	272.5	2.3	93.5	0.0107
Det2026.05		27.7	0.02	7.76	5.25	285.4	2.36	91	0.011
Int2026.05		26.6	0.0132	7.22	5.13	153.1	2.06	108.8	0.0092
Bias2026.05		17.5	0.0334	8.91	6.37	519.6	2.65	69.1	0.0145
BPE2026.05		16.7	0.0301	8.79	6.68	432.4	2.57	72.7	0.0138
Det2026.05		16.6	0.0313	8.84	6.33	475.6	2.63	69.9	0.0143
Int2026.05		15.8	0.0195	7.82	5.55	245.6	2.3	90.1	0.0111
Bias2026.05		10	0.0459	9.74	6.85	779.8	2.92	57.6	0.0174
Det2026.05		9.7	0.0442	9.7	6.83	742.1	2.91	57.9	0.0173
BPE2026.05		9.4	0.0405	9.52	7.25	632.3	2.83	61	0.0164
Int2026.05		9.3	0.0321	8.72	6.37	469.7	2.66	69.6	0.0144
Bias2026.05		5.9	0.0624	10.46	7.6	1,140.2	3.19	49.9	0.02
Det2026.05		5.8	0.0608	10.41	7.56	1,107	3.19	49.8	0.0201
BPE2026.05		5.4	0.0548	10.28	7.71	951	3.1	51.9	0.0193
Int2026.05		5.2	0.0391	9.09	6.74	590.3	2.8	63.9	0.0157