Tokenizer Evaluation on mC4 unbalanced 30-lang benchmark leaderboard · SOTA2 Research

Tokenizer Evaluation on mC4 unbalanced 30-lang

0.0296Compression RatePA-BPE (window+hybrid-127k)

Evaluation Results

Method	Links
PA-BPE (window+hybrid-127k)2025.08		0.0296	0.1231	49.3	3.698	0.46	0.032	0.53	0.596
BPE2025.08		0.0294	0.12	48.3	3.696	0.46	0.056	0.529	0.598
PA-BPE (window+hybrid)2025.08		0.0294	0.1217	49	3.709	0.46	0.03	0.524	0.59
PA-BPE (hybrid-127k)2025.08		0.0294	0.1188	47.9	3.722	0.46	0.027	0.521	0.59
PA-BPE (window)2025.08		0.0292	0.1203	48.9	3.725	0.47	0.032	0.514	0.58
PA-BPE (hybrid)2025.08		0.0291	0.1157	47.1	3.739	0.47	0.026	0.513	0.582
PA-BPE2025.08		0.0288	0.1143	47	3.752	0.47	0.028	0.501	0.57
PA-BPE (window+hybrid)2025.08		0.0266	0.0761	67.9	4.127	0.49	0.031	0.43	0.517
PA-BPE (hybrid)2025.08		0.0264	0.0737	66.3	4.116	0.5	0.027	0.426	0.515
BPE2025.08		0.0262	0.0718	65.1	4.178	0.5	0.067	0.434	0.527
PA-BPE (window)2025.08		0.0262	0.0749	67.9	4.152	0.5	0.029	0.417	0.507
PA-BPE2025.08		0.026	0.0727	66.4	4.138	0.5	0.027	0.418	0.508
UnigramLM2025.08		0.0193	0.0457	56.3	4.536	0.29	0.095	0.147	0.262