ResearchBenchmarksSentence Retrieval on 6 languages Same script splitSentence Retrieval on 6 languages Same script splitFollow43.9AccuracyUnigram27.36431.65735.9540.243May 26, 2023Evaluation resultsMethodMethodLinksAccuracyUnigramTokenizer=UnigramTokenizer=Unigram2023.0543.9TokMixTokenizer=TokMixTokenizer=TokMix2023.0543.4BPETokenizer=BPETokenizer=BPE2023.0540.7NoOverlapTokenizer=NoOverlapTokenizer=NoOverlap2023.0528