ResearchBenchmarksIntrinsic tokenization quality evaluation on CulturaX 18 languages sampleIntrinsic tokenization quality evaluation on CulturaX 18 languages sampleFollow4.89WFText3.11163.57334.0354.4967Jun 18, 2026Evaluation resultsMethodMethodLinksWFPCWATLTTRRE1RE2RE∞CRVUZipfDTPTFGTextVocabulary size=300k,...Vocabulary size=300k, Aggregation=macro-averaged2026.064.89922.02138.816.373.992.2620.293.336IPAVocabulary size=300k,...Vocabulary size=300k, Aggregation=macro-averaged2026.063.18902.72169.146.063.72.9820.161.7217