ResearchBenchmarksCross-language Vocabulary Overlap on 6 languages (Same script split)Cross-language Vocabulary Overlap on 6 languages (Same script split)Follow0.62JSDUnigram0.60480.70740.810.9126May 26, 2023Evaluation resultsMethodMethodLinksJSDUnigramTokenizer=UnigramTokenizer=Unigram2023.050.62TokMixTokenizer=TokMixTokenizer=TokMix2023.050.65BPETokenizer=BPETokenizer=BPE2023.050.68NoOverlapTokenizer=NoOverlapTokenizer=NoOverlap2023.051