In-context retrieval on Real-world data
50.1SQuAD AccuracyStaticHybrid
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| StaticHybridHybrid Ratio=1:3, Components=full attention and SWA2026.03 | 50.1 | 32.7 | 31.3 | 65.1 | 67.9 | 56.5 | 50.6 | |
| SWA-CPTMode=continual pretraining, Attention=sliding window2026.03 | 49.7 | 28.4 | 32.2 | 65.7 | 24 | 49.1 | 41.5 | |
| SwiAttnArchitecture=Switch Attention, Pretraining=continual (8B tokens)2026.03 | 49.4 | 35.2 | 32.4 | 65.2 | 75.1 | 60.2 | 52.9 | |
| SWA-ZSMode=zero-shot, Attention=sliding window2026.03 | 49.2 | 30.9 | 31.1 | 64.6 | 22.5 | 52 | 41.7 | |
| FullAttnAttention=full attention2026.03 | 49.2 | 35.1 | 31.1 | 64.6 | 76.4 | 60.8 | 52.9 | |
| Hybrid M2RNNModel scale=7B, Active parameters=1B, Architecture=MoE2026.03 | 48.4 | 31.4 | 31.4 | 64.6 | 78 | 79.8 | 55.6 | |
| Hybrid Mamba-2Model scale=7B, Active parameters=1B, Architecture=MoE2026.03 | 48.2 | 31.4 | 32.7 | 67.9 | 73.1 | 77.1 | 55.1 | |
| Hybrid Gated DeltaNet + M2RNN-5Model scale=7B, Active parameters=1B, Architecture=MoE, Number of M2RNN layers=52026.03 | 47.1 | 32 | 31.2 | 65.5 | 77.2 | 77.9 | 55.1 | |
| Hybrid Mamba-2 + M2RNN-5Model scale=7B, Active parameters=1B, Architecture=MoE, Number of M2RNN layers=52026.03 | 46.8 | 31.5 | 28.7 | 66 | 78.5 | 79.7 | 55.2 | |
| Hybrid Mamba-2 + M2RNN-1Model scale=7B, Active parameters=1B, Architecture=MoE, Number of M2RNN layers=12026.03 | 46.6 | 30.6 | 33.6 | 67.1 | 78.4 | 75 | 55.2 | |
| Hybrid Gated DeltaNetModel scale=7B, Active parameters=1B, Architecture=MoE2026.03 | 46.4 | 31.2 | 28.3 | 63.6 | 67.1 | 71.9 | 51.4 | |
| Hybrid Gated DeltaNet + M2RNN-1Model scale=7B, Active parameters=1B, Architecture=MoE, Number of M2RNN layers=12026.03 | 46.4 | 29.6 | 32.1 | 66.4 | 73.1 | 77.4 | 54.2 | |
| Mamba-2Model scale=7B, Active parameters=1B, Architecture=MoE2026.03 | 41.7 | 26.5 | 30 | 65 | 33 | 54.4 | 41.8 | |
| Gated DeltaNetModel scale=7B, Active parameters=1B, Architecture=MoE2026.03 | 40.3 | 24.2 | 29.9 | 64.3 | 32.4 | 61.8 | 42.2 | |
| TransformerNumber of Parameters=1.3B, Training Tokens=100B, Batch Size=1M, Input tokens=2K2026.05 | 39.27 | 26.8 | 21.85 | 57.23 | 51.77 | 46.67 | 40.6 | |
| M2RNNModel scale=7B, Active parameters=1B, Architecture=MoE2026.03 | 37.9 | 21.1 | 28.2 | 62.5 | 27.6 | 49.9 | 37.9 | |
| MDNNumber of Parameters=1.3B, Training Tokens=100B, Batch Size=1M, Input tokens=2K2026.05 | 35.67 | 26.09 | 20.36 | 56.54 | 35.42 | 42.74 | 36.14 | |
| Mamba2Number of Parameters=1.3B, Training Tokens=100B, Batch Size=1M, Input tokens=2K2026.05 | 35.24 | 22.43 | 22.71 | 53.73 | 25.16 | 35.43 | 32.45 | |
| CombaNumber of Parameters=1.3B, Training Tokens=100B, Batch Size=1M, Input tokens=2K2026.05 | 35.2 | 25.97 | 21.37 | 55.39 | 35.24 | 38.33 | 35.25 | |
| KDANumber of Parameters=1.3B, Training Tokens=100B, Batch Size=1M, Input tokens=2K2026.05 | 33.79 | 24.77 | 22.09 | 54.68 | 28.16 | 32.24 | 32.62 | |
| GDNNumber of Parameters=1.3B, Training Tokens=100B, Batch Size=1M, Input tokens=2K2026.05 | 33.59 | 24.8 | 21.27 | 55.69 | 27.52 | 33.93 | 32.8 | |
| TransformerNumber of Parameters=400M, Training Tokens=15B, Batch Size=0.5M, Input tokens=2K2026.05 | 29.66 | 17.96 | 18.11 | 41.59 | 43.32 | 31.87 | 30.42 | |
| KDANumber of Parameters=400M, Training Tokens=15B, Batch Size=0.5M, Input tokens=2K2026.05 | 28.12 | 15.14 | 20.08 | 41.35 | 18.44 | 23.71 | 24.47 | |
| MDNNumber of Parameters=400M, Training Tokens=15B, Batch Size=0.5M, Input tokens=2K2026.05 | 28.01 | 16.95 | 19.89 | 43.01 | 28.07 | 24.65 | 26.76 | |
| GDNNumber of Parameters=400M, Training Tokens=15B, Batch Size=0.5M, Input tokens=2K2026.05 | 27.24 | 14.76 | 18.69 | 40.88 | 14.99 | 20.99 | 22.93 | |
| CombaNumber of Parameters=400M, Training Tokens=15B, Batch Size=0.5M, Input tokens=2K2026.05 | 27.18 | 16.03 | 19.02 | 43.78 | 17.08 | 20.99 | 24.01 | |
| Mamba2Number of Parameters=400M, Training Tokens=15B, Batch Size=0.5M, Input tokens=2K2026.05 | 27.01 | 13.78 | 17.97 | 38.92 | 11.81 | 17.24 | 21.12 | |
| GRUModel scale=7B, Active parameters=1B, Architecture=MoE2026.03 | 27 | 16.7 | 24.6 | 59.3 | 5.2 | 18.4 | 25.2 | |
| Transformer++Model scale=7B, Active parameters=1B, Architecture=MoE2026.03 | 22.1 | 24 | 27.6 | 62.2 | 67 | 70.1 | 45.5 | |
| RNNModel scale=7B, Active parameters=1B, Architecture=MoE2026.03 | 20 | 12.2 | 20.4 | 53.1 | 0.8 | 9.8 | 19.4 |