ResearchBenchmarksVocabulary Overlap on 20 languages All transfersFollow0.73JSDUnigram0.72680.74840.770.7916May 26, 2023Evaluation ResultsMethodMethodLinksJSDUnigramTokenizer=UnigramTokenizer=Unigram2023.050.73TokMixTokenizer=TokMixTokenizer=TokMix2023.050.78BPETokenizer=BPETokenizer=BPE2023.050.81