Multilingual Reranking on Project-Freelancer Interactions (test)
99.7RecallGemma
Evaluation results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GemmaModel size=1B, Inference time (@1k)=13.3m, Loss=L_MSE2026.01 | 99.7 | 7.2 | 94.2 | 55.7 | 59.3 | 63.5 | 97.5 | 0.141 | 0.024 | 0.169 | 12.8 | |
| Ours (CMMD)Model size=45M, Inference time (@1k)=287ms, Loss=L_CMMD2026.01 | 94.9 | 27.1 | 93.1 | 51.7 | 63.1 | 67.5 | 97.3 | 0.131 | 0.004 | 0.034 | 5.7 | |
| QwenModel size=0.6B, Inference time (@1k)=7.06m, Loss=L_MSE2026.01 | 94.7 | 21.4 | 92.7 | 45.6 | 57.2 | 61.4 | 97 | 0.145 | 0.024 | 0.071 | 6.9 | |
| Ours (CLID + MSE)Model size=45M, Inference time (@1k)=287ms, Loss=L_CLID + MSE2026.01 | 90.2 | 40.5 | 92.9 | 50.6 | 62.7 | 67.2 | 97.2 | 0.14 | 0.051 | 0.032 | 6.8 | |
| OursModel size=45M, Inference time (@1k)=287ms, Loss=L_MSE2026.01 | 89.7 | 38.3 | 92.6 | 48.8 | 57.8 | 62.2 | 96.9 | 0.151 | 0.074 | 0.061 | 8.7 | |
| GemmaModel size=1B, Inference time (@1k)=13.3m, Loss=L_CE2026.01 | 86.2 | 48.7 | 91.7 | 45 | 42.5 | 43.8 | 95.8 | 0.184 | 0.043 | 0 | 11.3 | |
| QwenModel size=0.6B, Inference time (@1k)=7.06m, Loss=Zero-shot2026.01 | 77.4 | 71.7 | 94.2 | 55.2 | 56.7 | 61 | 97.3 | 0.291 | 0.007 | 0.517 | 22.4 | |
| GemmaModel size=1B, Inference time (@1k)=13.3m, Loss=Zero-shot2026.01 | 45.6 | 55 | 89.4 | 29.6 | 39.2 | 42.8 | 94.8 | 5.876 | 5.875 | 2.8 | 587.6 |