Narrative Retrieval on CARDS
0.505MAPSpecFi-CS (gemma-3-27b-it abliterated + Qwen3-Embedding-4b)
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| SpecFi-CS (gemma-3-27b-it abliterated + Qwen3-Embedding-4b)Setup=SpecFi-CS, Model=gemma-3-27b-it abliterated, Embedding=Qwen3-Embedding-4b2026.03 | 0.505 | 0.738 | 0.66 | 0.49 | 24.079 | |
| static* (gpt-4o + text-embedding-3-large)Setup=static*, Model=gpt-4o, Embedding=text-embedding-3-large2026.03 | 0.488 | 0.713 | 0.649 | 0.487 | 16.006 | |
| SpecFi-CS (gemma-3-27b-it + Qwen3-Embedding-4b)Setup=SpecFi-CS, Model=gemma-3-27b-it, Embedding=Qwen3-Embedding-4b2026.03 | 0.468 | 0.709 | 0.631 | 0.492 | 27.033 | |
| static* (gemma-3-27b-it abliterated + Qwen3-Embedding-4b)Setup=static*, Model=gemma-3-27b-it abliterated, Embedding=Qwen3-Embedding-4b2026.03 | 0.464 | 0.679 | 0.637 | 0.468 | 40.739 | |
| SpecFi-DR (gemma-3-27b-it abliterated + Qwen3-Embedding-4b)Setup=SpecFi-DR, Model=gemma-3-27b-it abliterated, Embedding=Qwen3-Embedding-4b2026.03 | 0.457 | 0.693 | 0.619 | 0.453 | 34.902 | |
| static* (gemma-3-27b-it + Qwen3-Embedding-4b)Setup=static*, Model=gemma-3-27b-it, Embedding=Qwen3-Embedding-4b2026.03 | 0.435 | 0.635 | 0.616 | 0.435 | 38.284 | |
| SpecFi-CS (gpt-4o + text-embedding-3-large)Setup=SpecFi-CS, Model=gpt-4o, Embedding=text-embedding-3-large2026.03 | 0.426 | 0.66 | 0.597 | 0.456 | 15.124 | |
| SpecFi-DR (gemma-3-27b-it + Qwen3-Embedding-4b)Setup=SpecFi-DR, Model=gemma-3-27b-it, Embedding=Qwen3-Embedding-4b2026.03 | 0.424 | 0.63 | 0.581 | 0.432 | 39.321 | |
| SpecFi-DR (gpt-4o + text-embedding-3-large)Setup=SpecFi-DR, Model=gpt-4o, Embedding=text-embedding-3-large2026.03 | 0.421 | 0.682 | 0.6 | 0.44 | 17.19 | |
| zero shot (gpt-4o + text-embedding-3-large)Setup=zero shot, Model=gpt-4o, Embedding=text-embedding-3-large2026.03 | 0.321 | 0.509 | 0.487 | 0.37 | 29.091 | |
| zero shot (gemma-3-27b-it + Qwen3-Embedding-4b)Setup=zero shot, Model=gemma-3-27b-it, Embedding=Qwen3-Embedding-4b2026.03 | 0.313 | 0.469 | 0.456 | 0.371 | 18.867 | |
| zero shot (gemma-3-27b-it abliterated + Qwen3-Embedding-4b)Setup=zero shot, Model=gemma-3-27b-it abliterated, Embedding=Qwen3-Embedding-4b2026.03 | 0.295 | 0.428 | 0.436 | 0.308 | 17.205 |