ResearchBenchmarksVocabulary Allocation on 6 languages Averaged (test)Follow2,198ARTokMix1,814.241,913.872,013.52,113.13May 26, 2023Evaluation ResultsMethodMethodLinksARCPTTokMixTokenizer method=TokMixTokenizer method=TokMix2023.052,1983.34BPETokenizer method=BPETokenizer method=BPE2023.052,1934.47UnigramTokenizer method=UnigramTokenizer method=Unigram2023.052,0423.17NoOverlapTokenizer method=NoOve...Tokenizer method=NoOverlap2023.051,8293.16