Multilingual Information Retrieval on Belebele
0.926nDCG@20embeddinggemma-300m + SHIFT
Evaluation results
| Method | Links | ||
|---|---|---|---|
| embeddinggemma-300m + SHIFTArchi=encoder, SHIFT=true, Source Query Language=English2026.06 | 0.926 | 92.8 | |
| embeddinggemma-300mArchi=encoder, SHIFT=false, Source Query Language=English2026.06 | 0.924 | 92.6 | |
| gte-Qwen2-1.5B-instruct + SHIFTArchi=decoder, SHIFT=true, Source Query Language=English2026.06 | 0.914 | 91.2 | |
| multilingual-e5-large + SHIFTArchi=encoder, SHIFT=true, Source Query Language=English2026.06 | 0.91 | 90.7 | |
| gte-Qwen2-1.5B-instructArchi=decoder, SHIFT=false, Source Query Language=English2026.06 | 0.904 | 90.1 | |
| Qwen3-Embedding-0.6B + SHIFTArchi=decoder, SHIFT=true, Source Query Language=English2026.06 | 0.895 | 89.2 | |
| Qwen3-Embedding-0.6BArchi=decoder, SHIFT=false, Source Query Language=English2026.06 | 0.89 | 88.7 | |
| bge-m3 + SHIFTArchi=encoder, SHIFT=true, Source Query Language=English2026.06 | 0.876 | 87.4 | |
| bge-m3Archi=encoder, SHIFT=false, Source Query Language=English2026.06 | 0.874 | 87.1 | |
| multilingual-e5-largeArchi=encoder, SHIFT=false, Source Query Language=English2026.06 | 0.816 | 81.2 | |
| llama-nemotron-embed-1b-v2 + SHIFTArchi=decoder, SHIFT=true, Source Query Language=English2026.06 | 0.7 | 69.7 | |
| llama-nemotron-embed-1b-v2Archi=decoder, SHIFT=false, Source Query Language=English2026.06 | 0.688 | 68.1 | |
| LangSAE EditingQuery Language=Spanish2026.01 | 0.6653 | 67.4 | |
| jina-embeddings-v3Query Language=Spanish2026.01 | 0.6649 | 67.39 | |
| SAE ReconstructedQuery Language=Spanish2026.01 | 0.6624 | 66.9 | |
| LangSAE EditingQuery Language=Italian2026.01 | 0.652 | 65.68 | |
| jina-embeddings-v3Query Language=Italian2026.01 | 0.6488 | 65.29 | |
| SAE ReconstructedQuery Language=Italian2026.01 | 0.6471 | 65.1 | |
| LangSAE EditingQuery Language=Portuguese2026.01 | 0.6333 | 63.88 | |
| jina-embeddings-v3Query Language=Portuguese2026.01 | 0.6318 | 63.88 | |
| SAE ReconstructedQuery Language=Portuguese2026.01 | 0.6304 | 63.54 | |
| LangSAE EditingQuery Language=French2026.01 | 0.6134 | 61.86 | |
| jina-embeddings-v3Query Language=French2026.01 | 0.6112 | 61.86 | |
| SAE ReconstructedQuery Language=French2026.01 | 0.6108 | 61.59 | |
| LangSAE EditingQuery Language=English2026.01 | 0.5839 | 58.73 | |
| SAE ReconstructedQuery Language=English2026.01 | 0.5836 | 58.74 | |
| LangSAE EditingQuery Language=Japanese2026.01 | 0.5809 | 58.28 | |
| jina-embeddings-v3Query Language=English2026.01 | 0.5806 | 58.76 | |
| jina-embeddings-v3Query Language=Japanese2026.01 | 0.5786 | 57.57 | |
| SAE ReconstructedQuery Language=Japanese2026.01 | 0.5767 | 57.68 | |
| LangSAE EditingQuery Language=Chinese2026.01 | 0.5648 | 56.17 | |
| SAE ReconstructedQuery Language=Chinese2026.01 | 0.5611 | 55.38 | |
| jina-embeddings-v3Query Language=Chinese2026.01 | 0.5529 | 54.3 | |
| jina-embeddings-v3Query Language=Arabic2026.01 | 0.5504 | 55.74 | |
| LangSAE EditingQuery Language=Arabic2026.01 | 0.5474 | 55.86 | |
| SAE ReconstructedQuery Language=Arabic2026.01 | 0.545 | 55.3 | |
| LangSAE EditingQuery Language=Macro Average2026.01 | 0.5432 | 54.23 | |
| SAE ReconstructedQuery Language=Macro Average2026.01 | 0.5405 | 53.86 | |
| jina-embeddings-v3Query Language=Macro Average2026.01 | 0.5401 | 53.9 | |
| LangSAE EditingQuery Language=Hindi2026.01 | 0.3802 | 36.83 | |
| SAE ReconstructedQuery Language=Hindi2026.01 | 0.3782 | 36.66 | |
| jina-embeddings-v3Query Language=Hindi2026.01 | 0.3752 | 36.56 | |
| LangSAE EditingQuery Language=Russian2026.01 | 0.2111 | 17.74 | |
| SAE ReconstructedQuery Language=Russian2026.01 | 0.2103 | 17.69 | |
| jina-embeddings-v3Query Language=Russian2026.01 | 0.2071 | 17.6 |