ResearchBenchmarksVocabulary Overlap on 20 languages Different scriptFollow0.75JSDUnigram0.74680.76840.790.8116May 26, 2023Evaluation ResultsMethodMethodLinksJSDUnigramTokenizer=UnigramTokenizer=Unigram2023.050.75TokMixTokenizer=TokMixTokenizer=TokMix2023.050.8BPETokenizer=BPETokenizer=BPE2023.050.83