ResearchBenchmarksCross-language Vocabulary Overlap on 6 languages All transfersCross-language Vocabulary Overlap on 6 languages All transfersFollow0.74JSDUnigram0.72960.79980.870.9402May 26, 2023Evaluation resultsMethodMethodLinksJSDUnigramTokenizer=UnigramTokenizer=Unigram2023.050.74TokMixTokenizer=TokMixTokenizer=TokMix2023.050.77BPETokenizer=BPETokenizer=BPE2023.050.8NoOverlapTokenizer=NoOverlapTokenizer=NoOverlap2023.051