Common-sense Reasoning on LAMBADA
74.9AccuracyLoLA-8B (ours)
Evaluation Results
| Method | Links | |
|---|---|---|
| LoLA-8B (ours)Architecture=Subquadratic (Distilled), Tokens (B)=0.04, Evaluation Protocol=zero-shot, η=64, λ=642025.05 | 74.9 | |
| Llama-3.1-8BArchitecture=Transformer, Tokens (B)=15000, Evaluation Protocol=zero-shot2025.05 | 73 | |
| Llamba-8BArchitecture=Subquadratic (Distilled), Tokens (B)=12, Evaluation Protocol=zero-shot2025.05 | 69.4 | |
| LoLCATs-8BArchitecture=Subquadratic (Distilled), Tokens (B)=0.04, Evaluation Protocol=zero-shot2025.05 | 67.6 | |
| Falcon3-Mamba-7BArchitecture=Subquadratic (Pretrained), Tokens (B)=7300, Evaluation Protocol=zero-shot2025.05 | 67.5 | |
| RecurrentGemma-9BArchitecture=Subquadratic (Pretrained), Tokens (B)=2000, Evaluation Protocol=zero-shot2025.05 | 54.1 | |
| HADESZero-shot=true, Parameters=218M2026.03 | 39.24 | |
| Mamba2Zero-shot=true, Parameters=370M2026.03 | 36.46 | |
| DeltaNetZero-shot=true, Parameters=370M2026.03 | 35.75 | |
| RetNetZero-shot=true, Parameters=370M2026.03 | 35.36 | |
| Linear TransformerZero-shot=true, Parameters=370M2026.03 | 24.06 | |
| Mamba1Zero-shot=true, Parameters=370M2026.03 | 22.43 |