Long-context language understanding on LongBench 20 samples/task
1.91NarrQA PerformanceMamba-1.4B
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Mamba-1.4BPlatform=Local / RTX 3070, Precision=FP16, Samples per task=202026.04 | 1.91 | 4.6 | 0 | 14.98 | 2,868.74 | |
| Pythia-410MPlatform=Local / RTX 3070, Precision=FP16, Samples per task=202026.04 | 1.9 | 3.92 | 0 | 15.5 | 624.43 | |
| Mamba-370MPlatform=Remote / RTX 3090, Precision=FP16, Samples per task=202026.04 | 1.41 | 4.43 | 0 | 14.72 | 4,890 | |
| Mamba-370MPlatform=Local / RTX 3070, Precision=FP16, Samples per task=202026.04 | 1.35 | 3.5 | 0 | 13.7 | 1,384.25 |