Book Plot Retrieval on NDP v1
68.98Recall@10SitEmb-v1.5-Qwen3 + book note data
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| SitEmb-v1.5-Qwen3 + book note dataSize=8B, Context Setting=+ Situated Context, Maximum Length=81922025.08 | 68.98 | 79.32 | 86.68 | |
| SitEmb-v1.5-Qwen3 + book note dataSize=8B, Context Setting=+ Situated Summ., Maximum Length=81922025.08 | 67.81 | 76.37 | 84.37 | |
| SitEmb-v1.5-Qwen3 + book note dataSize=8B, Context Setting=Chunk-Only, Maximum Length=81922025.08 | 66.81 | 74.36 | 84.32 | |
| SitEmb-v1.5-Qwen3Size=8B, Context Setting=+ Situated Context, Maximum Length=81922025.08 | 63.32 | 72.75 | 83.59 | |
| SitEmb-v1.5-Qwen3Size=8B, Context Setting=Chunk-Only, Maximum Length=81922025.08 | 61.66 | 69.06 | 79.54 | |
| voyage-context-3Size=unk, Context Setting=+ Situated Summ., Maximum Length=81922025.08 | 61.37 | 70.36 | 82.84 | |
| SitEmb-v1.5-Qwen3Size=8B, Context Setting=+ Situated Summ., Maximum Length=81922025.08 | 61.26 | 70.02 | 81.57 | |
| voyage-context-3Size=unk, Context Setting=+ Situated Context, Maximum Length=81922025.08 | 60.46 | 69.5 | 82.19 | |
| voyage-context-3Size=unk, Context Setting=Chunk-Only, Maximum Length=81922025.08 | 58.54 | 68.47 | 79.09 | |
| Qwen3-EmbeddingSize=8B, Context Setting=Chunk-Only, Maximum Length=81922025.08 | 51.58 | 61.32 | 73.47 | |
| SitEmb-v1-M3Size=1B, Context Setting=+ Situated Summ., Maximum Length=81922025.08 | 51.47 | 63.69 | 76.76 | |
| NV-Embed-v2Size=7B, Context Setting=Chunk-Only, Maximum Length=81922025.08 | 51.38 | 60.11 | 71.16 | |
| Qwen3-EmbeddingSize=8B, Context Setting=+ Situated Summ., Maximum Length=81922025.08 | 51.2 | 61.63 | 76.1 | |
| SitEmb-v1-M3Size=1B, Context Setting=+ Situated Context, Maximum Length=81922025.08 | 50.85 | 60.57 | 76.4 | |
| NV-Embed-v2Size=7B, Context Setting=+ Situated Summ., Maximum Length=81922025.08 | 49.25 | 58.33 | 70.85 | |
| SitEmb-v1-M3Size=1B, Context Setting=Chunk-Only, Maximum Length=81922025.08 | 48.79 | 58.45 | 73.29 | |
| Qwen3-EmbeddingSize=8B, Context Setting=+ Situated Context, Maximum Length=81922025.08 | 48.01 | 58.91 | 71.86 | |
| GTE-Qwen2Size=7B, Context Setting=Chunk-Only, Maximum Length=81922025.08 | 46.19 | 55.79 | 71.15 | |
| Jina-v3Size=0.5B, Context Setting=+ Situated Summ., Maximum Length=81922025.08 | 45.48 | 56.12 | 69.9 | |
| E5-MistralSize=7B, Context Setting=+ Situated Summ., Maximum Length=81922025.08 | 44.58 | 54.32 | 68.27 | |
| E5-MistralSize=7B, Context Setting=Chunk-Only, Maximum Length=81922025.08 | 43.18 | 51.93 | 66.65 | |
| Jina-v3Size=0.5B, Context Setting=Chunk-Only, Maximum Length=81922025.08 | 42.65 | 52.67 | 67.92 | |
| M3Size=0.5B, Context Setting=Chunk-Only, Maximum Length=81922025.08 | 42.55 | 53.33 | 66.51 | |
| GTE-Qwen2Size=7B, Context Setting=+ Situated Summ., Maximum Length=81922025.08 | 42.44 | 49.88 | 60.85 | |
| M3Size=0.5B, Context Setting=+ Situated Summ., Maximum Length=81922025.08 | 41.83 | 53.94 | 67.78 | |
| Jina-v3Size=0.5B, Context Setting=+ Situated Context, Maximum Length=81922025.08 | 34.1 | 44.27 | 58.33 | |
| NV-Embed-v2Size=7B, Context Setting=+ Situated Context, Maximum Length=81922025.08 | 21.01 | 30.2 | 42.18 | |
| GTE-Qwen2Size=7B, Context Setting=+ Situated Context, Maximum Length=81922025.08 | 19.01 | 29.77 | 49.34 | |
| E5-MistralSize=7B, Context Setting=+ Situated Context, Maximum Length=81922025.08 | 14.77 | 24.49 | 32.68 | |
| M3Size=0.5B, Context Setting=+ Situated Context, Maximum Length=81922025.08 | 9.48 | 15.68 | 24.7 |