Word Prediction on Lambada (Robustness Evaluation)
82.2Accuracy (Original)Hymba-1.5b
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Hymba-1.5bBit-Flips=2, Attack=COBRA2025.12 | 82.2 | 30 | |
| Mamba-2.8bBit-Flips=1, Attack=COBRA2025.12 | 78.34 | 4 | |
| Mamba2-2.7bBit-Flips=9, Attack=COBRA2025.12 | 77.78 | 15.33 | |
| Quamba-2.8b-w8a8Bit-Flips=1, Attack=COBRA2025.12 | 77.55 | 22 | |
| Quamba-2.8b-w4a16Bit-Flips=1, Attack=COBRA2025.12 | 76.03 | 0 | |
| Mamba2-1.3bBit-Flips=5, Attack=COBRA2025.12 | 75.39 | 30.67 | |
| Mamba-1.4bBit-Flips=1, Attack=COBRA2025.12 | 74.64 | 0 | |
| Quamba-1.4b-w4a16Bit-Flips=1, Attack=COBRA2025.12 | 73.39 | 57.33 | |
| Mamba2-370mBit-Flips=4, Attack=COBRA2025.12 | 67.98 | 17.33 | |
| Mamba-370mBit-Flips=4, Attack=COBRA2025.12 | 67.86 | 5.33 | |
| Quamba-1.4b-w8a8Bit-Flips=4, Attack=COBRA2025.12 | 54.67 | 46.67 |