List your company
ResearchFormerly WizWand

Tokenization Intrinsic Evaluation on FLORES+ 30-lang

0.027Compression RatePA-BPE (ratios)
0.0269640.0272070.027450.027693Aug 6, 2025
Updated 4mo ago

Evaluation results

MethodLinks
PA-BPE (ratios)Vocabulary Size=128k, Training Data=unbalanced 30-lang dataset, Method Variant=ratios2025.080.0270.073764.94.3980.490.040.5320.669
BPEVocabulary Size=128k, Training Data=unbalanced 30-lang dataset2025.080.02750.0777673.990.490.0640.5370.659
PA-BPEVocabulary Size=128k, Training Data=unbalanced 30-lang dataset2025.080.02760.081970.44.080.490.0070.5390.671
PA-BPE (window)Vocabulary Size=128k, Training Data=unbalanced 30-lang dataset, Method Variant=window2025.080.02780.083871.64.0420.480.0090.5460.677
PA-BPE (hybrid)Vocabulary Size=128k, Training Data=unbalanced 30-lang dataset, Method Variant=hybrid2025.080.02780.081769.84.0560.490.0150.5410.667
PA-BPE (hybrid+window)Vocabulary Size=128k, Training Data=unbalanced 30-lang dataset, Method Variant=hybrid+window2025.080.02790.083270.84.0290.480.0190.5460.67