Mathematical and Code Reasoning on ZeroEval (test)
67.85GSM8K AccuracyMamba-Llama3
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Mamba-Llama3Attention percentage=50%2024.08 | 67.85 | 27.88 | |
| Mamba2-Llama3Attention percentage=50%2024.08 | 59.36 | 24.88 | |
| Falcon Mamba-7BType=Instruct-tuned2024.08 | 41.32 | 8.88 | |
| Mamba-Llama3Attention percentage=25%2024.08 | 40.64 | 15.62 | |
| RecurrentGemma-9BType=Instruct-tuned2024.08 | 38.51 | 26.25 | |
| Mamba2-Llama3Attention percentage=25%2024.08 | 38.13 | 13.25 | |
| Mamba2-Llama3Attention percentage=12.5%2024.08 | 35.03 | 10.25 | |
| Mamba-Llama3Attention percentage=12.5%2024.08 | 26.91 | 11.25 |