ResearchBenchmarksCross-language Vocabulary Overlap on 6 languages Different scriptCross-language Vocabulary Overlap on 6 languages Different scriptFollow0.77JSDUnigram0.76080.82290.8850.9471May 26, 2023Evaluation resultsMethodMethodLinksJSDUnigramTokenizer=UnigramTokenizer=Unigram2023.050.77TokMixTokenizer=TokMixTokenizer=TokMix2023.050.8BPETokenizer=BPETokenizer=BPE2023.050.83NoOverlapTokenizer=NoOverlapTokenizer=NoOverlap2023.051