Information Retrieval and Reading Comprehension on JRT Retrieval Benchmark Suite 1.3B scale (test)
44.83SWDECARVE + SWA
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| CARVE + SWAModel category=Hybrid models, SWA ratio=3:1 ratio, Model scale=1.3B, Input truncation=2K tokens2026.06 | 44.83 | 46.54 | 58.12 | 63.47 | 28.54 | 24.81 | 45.89 | |
| GDN-2 + SWAModel category=Hybrid models, SWA ratio=3:1 ratio, Model scale=1.3B, Input truncation=2K tokens2026.06 | 41.96 | 44.7 | 54.68 | 62.38 | 26.31 | 23.67 | 42.28 | |
| KDA + SWAModel category=Hybrid models, SWA ratio=3:1 ratio, Model scale=1.3B, Input truncation=2K tokens2026.06 | 39.83 | 40.1 | 53.59 | 59.89 | 25.27 | 22.18 | 40.14 | |
| Mamba-3 SISO + SWAModel category=Hybrid models, SWA ratio=3:1 ratio, Model scale=1.3B, Input truncation=2K tokens2026.06 | 35.3 | 46.42 | 54.95 | 59.54 | 25.91 | 23.96 | 41.01 | |
| Mamba-2 + SWAModel category=Hybrid models, SWA ratio=3:1 ratio, Model scale=1.3B, Input truncation=2K tokens2026.06 | 34.67 | 40.74 | 52.31 | 60.13 | 25.91 | 24.68 | 39.74 | |
| Gated DeltaNet + SWAModel category=Hybrid models, SWA ratio=3:1 ratio, Model scale=1.3B, Input truncation=2K tokens2026.06 | 33.18 | 42.28 | 50.86 | 60.6 | 25.78 | 21.95 | 39.11 | |
| Mamba-3 MIMO + SWAModel category=Hybrid models, SWA ratio=3:1 ratio, Model scale=1.3B, Input truncation=2K tokens2026.06 | 32.33 | 44.7 | 55.31 | 59 | 26.26 | 23.08 | 40.11 | |
| Transformer (full attn)Model category=Hybrid models, SWA ratio=3:1 ratio, Model scale=1.3B, Input truncation=2K tokens2026.06 | 32.21 | 38.67 | 54.78 | 58.09 | 22.49 | 22.18 | 38.07 | |
| CARVEModel category=Recurrent models, Model scale=1.3B, Input truncation=2K tokens2026.06 | 25.18 | 38.24 | 21.34 | 62.14 | 20.43 | 19.84 | 31.09 | |
| GDN-2Model category=Recurrent models, Model scale=1.3B, Input truncation=2K tokens2026.06 | 23.65 | 36.75 | 19.98 | 61.37 | 19.64 | 17.87 | 29.88 | |
| KDAModel category=Recurrent models, Model scale=1.3B, Input truncation=2K tokens2026.06 | 22.49 | 35.1 | 14.9 | 58.12 | 19.58 | 21.8 | 28.67 | |
| Gated DeltaNetModel category=Recurrent models, Model scale=1.3B, Input truncation=2K tokens2026.06 | 17.9 | 32.67 | 18.52 | 59.6 | 20.16 | 19.69 | 28.09 | |
| Mamba-3 SISOModel category=Recurrent models, Model scale=1.3B, Input truncation=2K tokens2026.06 | 17.62 | 35.07 | 11.08 | 58.89 | 18.18 | 21.32 | 27.03 | |
| Mamba-2Model category=Recurrent models, Model scale=1.3B, Input truncation=2K tokens2026.06 | 17.24 | 32.38 | 14.53 | 58.35 | 18.91 | 19.6 | 26.84 | |
| Mamba-3 MIMOModel category=Recurrent models, Model scale=1.3B, Input truncation=2K tokens2026.06 | 16.68 | 36.65 | 17.44 | 59.06 | 19.16 | 21.08 | 28.35 |