ResearchBenchmarksVocabulary Overlap on 20 languages Same scriptFollow0.58JSDUnigram0.57640.60070.6250.6493May 26, 2023Evaluation ResultsMethodMethodLinksJSDUnigramTokenizer=UnigramTokenizer=Unigram2023.050.58TokMixTokenizer=TokMixTokenizer=TokMix2023.050.64BPETokenizer=BPETokenizer=BPE2023.050.67