Commonsense Reasoning on Lambada (Accuracy and Perplexity)
66.8AccuracyFinch-1.6B
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Finch-1.6BTokens (B)=1100, Zero-shot=true, Training Strategy=Subquadratic, Training Source=various sources2025.05 | 66.8 | — | |
| Mamba2-1.3BTokens (B)=315, Zero-shot=true, Training Strategy=Subquadratic, Training Source=various sources2025.05 | 65.7 | — | |
| Mamba1-1.4BTokens (B)=315, Zero-shot=true, Training Strategy=Subquadratic, Training Source=various sources2025.05 | 64.9 | — | |
| LoLA-1BTokens (B)=0.04, Zero-shot=true, Training Strategy=Subquadratic, Distillation Source=Llama-3.2-1B2025.05 | 61.9 | 5.3 | |
| Llama-3.2-1BTokens (B)=9000, Zero-shot=true, Training Strategy=Transformers2025.05 | 60.1 | — | |
| xLSTM-1.4BTokens (B)=300, Zero-shot=true, Training Strategy=Subquadratic, Training Source=various sources2025.05 | 57.8 | — | |
| Phi-1.5-1.3BTokens (B)=150, Zero-shot=true, Training Strategy=Transformers2025.05 | 53.4 | — | |
| LoLCATs-Llama-1BTokens (B)=0.04, Zero-shot=true, Training Strategy=Subquadratic, Distillation Source=Llama-3.2-1B2025.05 | 53.4 | 9.3 | |
| RecurrentGemma-2BTokens (B)=2000, Zero-shot=true, Training Strategy=Subquadratic, Training Source=various sources2025.05 | 52.5 | — | |
| Samba-1.3BTokens (B)=100, Zero-shot=true, Training Strategy=Subquadratic, Training Source=various sources2025.05 | 51.7 | — | |
| Phi-Mamba-1.5BTokens (B)=3, Zero-shot=true, Training Strategy=Subquadratic, Distillation Source=Phi-1.5-1.3B2025.05 | 50.1 | — | |
| Llamba-1BTokens (B)=8, Zero-shot=true, Training Strategy=Subquadratic, Distillation Source=Llama-3.2-1B2025.05 | 48.4 | — | |
| Gated-DeltaNet-1.3BTokens (B)=100, Zero-shot=true, Training Strategy=Subquadratic, Training Source=FineWeb-Edu2025.05 | 46.7 | 12.2 | |
| Mamba2-1.3BTokens (B)=100, Zero-shot=true, Training Strategy=Subquadratic, Training Source=FineWeb-Edu2025.05 | 45.7 | 12.6 | |
| Mamba1-1.3BTokens (B)=100, Zero-shot=true, Training Strategy=Subquadratic, Training Source=FineWeb-Edu2025.05 | 44 | 15.1 | |
| DeltaNet-1.3BTokens (B)=100, Zero-shot=true, Training Strategy=Subquadratic, Training Source=FineWeb-Edu2025.05 | 42.5 | 16.9 | |
| RetNet-1.3BTokens (B)=100, Zero-shot=true, Training Strategy=Subquadratic, Training Source=FineWeb-Edu2025.05 | 40.5 | 17.3 | |
| HGRN2-1.3BTokens (B)=100, Zero-shot=true, Training Strategy=Subquadratic, Training Source=FineWeb-Edu2025.05 | 39.5 | 17.7 |