Recall-intensive Retrieval on SWDE, SQuAD, FDA, TQA, NQ, and Drop Suite
76.85Performance on SWDEQwen3-8B
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Qwen3-8BArchitecture=Transformer, Training Tokens (B)=360002025.10 | 76.85 | 59.89 | 73.48 | 72.63 | 45.01 | 36.61 | 60.75 | |
| Qwen2.5-7BArchitecture=Transformer, Training Tokens (B)=180002025.10 | 75.45 | 54.28 | 82.38 | 77.49 | 48.43 | 38.28 | 62.72 | |
| Llama-3-8BArchitecture=Transformer, Training Tokens (B)=150002025.10 | 72.07 | 54.22 | 82.92 | 74.17 | 43.24 | 33.88 | 60.08 | |
| Zamba-7BArchitecture=Hybrid, Training Tokens (B)=1000, Full-attention layers=132025.10 | 71.7 | 50.69 | 76.02 | 70.56 | 42.1 | 25.78 | 56.14 | |
| StripedHyena-7BArchitecture=Hybrid, Full-attention layers=162025.10 | 69.82 | 53.81 | 75.3 | 71.86 | 42.41 | 32.34 | 57.59 | |
| NHA-Llama-3-8BArchitecture=Hybrid, Training Tokens (B)=10, Full-attention layers=42025.10 | 67.67 | 55.16 | 73.84 | 73.05 | 42.1 | 34.02 | 57.64 | |
| Mistral-7BArchitecture=Transformer, Training Tokens (B)=80002025.10 | 65.32 | 54.01 | 75.3 | 73.16 | 42.76 | 27.55 | 56.35 | |
| Zamba2-7BArchitecture=Hybrid, Training Tokens (B)=2100, Full-attention layers=92025.10 | 64.48 | 50.15 | 71.48 | 71.86 | 40.13 | 29.23 | 54.56 | |
| FalconMamba-7BArchitecture=Linear/Subquadratic, Training Tokens (B)=55002025.10 | 59.51 | 49.31 | 48.32 | 75.06 | 37.85 | 30.09 | 50.02 | |
| RWKV-6-World-7BArchitecture=Linear/Subquadratic, Training Tokens (B)=14202025.10 | 51.55 | 46.62 | 56.86 | 69.85 | 37.03 | 28.7 | 48.44 | |
| Oryx-TMFamily=Oryx-TM, Mixer=Transformer, Context Length=2K2026.05 | 50.1 | 41.7 | 57.6 | 63.9 | 28.3 | 22.5 | 44 | |
| NHA-Qwen2.5-7BArchitecture=Hybrid, Training Tokens (B)=10, Full-attention layers=42025.10 | 44.61 | 47.26 | 62.58 | 71.98 | 37.47 | 37.47 | 50.23 | |
| Oryx-TGFamily=Oryx-TG, Mixer=Transformer, Context Length=2K2026.05 | 44.5 | 41.2 | 46.4 | 64.8 | 27.8 | 21.7 | 41.1 | |
| Mamba-7BArchitecture=Linear/Subquadratic, Training Tokens (B)=12002025.10 | 43.3 | 46.93 | 37.06 | 71.56 | 33.39 | 25.3 | 42.92 | |
| BaselineFamily=Baseline, Mixer=Transformer, Context Length=2K2026.05 | 42.3 | 42.8 | 55.2 | 65.9 | 26.7 | 22.9 | 42.6 | |
| Gated DeltaNet-H2Input sequence truncation=2K tokens2025.10 | 38.2 | 40.4 | 50.7 | 63.3 | 24.8 | 23.3 | 40.1 | |
| NirvanaInput sequence truncation=2K tokens2025.10 | 37.8 | 41 | 51.1 | 62.8 | 24.8 | 22.9 | 40.1 | |
| Gated DeltaNet-H1Input sequence truncation=2K tokens2025.10 | 35.6 | 39.7 | 52 | 60.1 | 24.6 | 22.2 | 39 | |
| Nirvana-noTriggerInput sequence truncation=2K tokens2025.10 | 35.1 | 39.8 | 50.5 | 60 | 22.3 | 21.7 | 38.2 | |
| SambaInput sequence truncation=2K tokens2025.10 | 33 | 39.2 | 50.5 | 57.7 | 23.5 | 20.2 | 37.3 | |
| Transformer++Input sequence truncation=2K tokens2025.10 | 29.5 | 38 | 52.2 | 58.3 | 22.5 | 21.6 | 37 | |
| Oryx-TGFamily=Oryx-TG, Mixer=Gated DeltaNet, Context Length=2K2026.05 | 29.3 | 37.9 | 18.2 | 64 | 23.5 | 22 | 32.5 | |
| BaselineFamily=Baseline, Mixer=Gated DeltaNet, Context Length=2K2026.05 | 27.4 | 36.4 | 24.2 | 63.3 | 23.8 | 21.6 | 32.8 | |
| Gated DeltaNetInput sequence truncation=2K tokens2025.10 | 25.4 | 34.8 | 23.7 | 60 | 20 | 19.8 | 30.6 | |
| Oryx-TMFamily=Oryx-TM, Mixer=Mamba-2, Context Length=2K2026.05 | 22.2 | 36.4 | 19.4 | 63.7 | 21.8 | 22.3 | 31 | |
| BaselineFamily=Baseline, Mixer=Mamba-2, Context Length=2K2026.05 | 20.6 | 36.1 | 21 | 61.7 | 22.4 | 20.3 | 30.3 | |
| Mamba2Input sequence truncation=2K tokens2025.10 | 19.1 | 33.6 | 25.3 | 61 | 20.8 | 19.2 | 29.8 | |
| DeltaNetInput sequence truncation=2K tokens2025.10 | 17.9 | 30.9 | 18.4 | 53.9 | 17.3 | 18.6 | 26.2 | |
| RetNetInput sequence truncation=2K tokens2025.10 | 14 | 28.5 | 7 | 54.4 | 16.2 | 17.3 | 22.9 | |
| MambaInput sequence truncation=2K tokens2025.10 | 9.8 | 25.8 | 3.7 | 54.3 | 14.9 | 17.4 | 21 | |
| HGRN2Input sequence truncation=2K tokens2025.10 | 8.3 | 25.3 | 4.8 | 51.2 | 14.2 | 16.9 | 20.1 |