Multi-task Evaluation on Aggregate (LAMBADA, HellaSwag, PIQA, ARC, WinoGrande)
51.9Avg AccuracyMistral (Full-Attention)
Evaluation results
| Method | Links | |
|---|---|---|
| Mistral (Full-Attention)Model Scale=1.4B2024.07 | 51.9 | |
| BMoJo (Fading + Eidetic)Model Scale=1.4B2024.07 | 49.1 | |
| BMoJo (Fading)Model Scale=1.4B2024.07 | 48.9 | |
| Mamba (SSM)Model Scale=1.4B2024.07 | 48.6 | |
| Hybrid (Sliding Attention + SSM)Model Scale=1.4B2024.07 | 44.8 | |
| Mistral (Full-Attention)Model Scale=370M2024.07 | 41.4 | |
| Mamba (SSM)Model Scale=370M2024.07 | 41.2 | |
| BMoJo (Fading)Model Scale=370M2024.07 | 40.7 | |
| BMoJo (Fading + Eidetic)Model Scale=370M2024.07 | 40.5 | |
| Hybrid (Sliding Attention + SSM)Model Scale=370M2024.07 | 39.3 |