Tokenization Disambiguation on IMDB English benchmark leaderboard · SOTA2 Research

Tokenization Disambiguation on IMDB English

6.58Average Perplexity (PPL)Verification-based

Evaluation Results

Method	Links
Verification-based2026.04		6.58	0	0	2.32	86	15.17	209.59	100
MWIS2026.04		6.87	0.09	0.75	2.12	81	7.64	55.92	100
Discop2026.04		7.26	0	0	2.28	88	4.9	0	—
ReTokSync2026.04		7.26	0	0	2.28	88	4.94	0.82	100
Stepwise Verification2026.04		7.36	0	0.05	2.31	89	6.65	35.71	100
Syncpool2026.04		7.65	0	0	1.68	64	5.05	3.06	100
MWIS2026.04		9.34	0.11	0.75	2.61	82	9.01	80.92	100
Verification-based2026.04		11	0	0	2.82	86	441.21	8,759.64	100
Discop2026.04		11.33	0	0	2.95	91	4.98	0	—
ReTokSync2026.04		11.36	0	0	2.95	91	5.05	1.41	99.9
MWIS2026.04		11.74	0.15	0.83	2.93	80	23.89	361.2	100
Stepwise Verification2026.04		11.93	0	0.15	3.06	91	11.82	137.35	100
Syncpool2026.04		11.98	0	0	1.01	33	5.15	3.41	100
Verification-based2026.04		13.54	0	0	3.13	87	552.64	10,714.87	100
Stepwise Verification2026.04		16.37	0	0.01	3.45	91	32.39	525.29	100
Syncpool2026.04		17.06	0	0	0.53	14	5.21	0.58	100
Discop2026.04		17.29	0	0	3.5	91	5.18	0	—
ReTokSync2026.04		17.29	0	0	3.5	91	5.27	1.74	100
Basic2026.04		19.93	0.96	9.15	2.72	96	7.64	55.92	100
Basic2026.04		59.05	1.24	8.77	4.05	98	8.75	75.7	100
Basic2026.04		219.67	1.36	7.58	5.72	98	21.69	318.73	100