ResearchBenchmarksSentence Retrieval on 6 languages (Different script split)Sentence Retrieval on 6 languages (Different script split)Follow0.23AccuracyTokMix0.118720.147610.17650.20539May 26, 2023Evaluation resultsMethodMethodLinksAccuracyTokMixTokenizer=TokMixTokenizer=TokMix2023.050.23UnigramTokenizer=UnigramTokenizer=Unigram2023.050.21BPETokenizer=BPETokenizer=BPE2023.050.209NoOverlapTokenizer=NoOverlapTokenizer=NoOverlap2023.050.123