Chat Evaluation on MT-Bench 1.0 (test)
8MT-Bench ScoreLlama-3.1-Instruct
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| Llama-3.1-InstructSize=8B, Alignment=RLHF2024.08 | 8 | — | — | |
| Llama-3-InstructSize=8B, Alignment=RLHF2024.08 | 8 | — | — | |
| GPT-3.5-turboAlignment=RLHF2024.08 | 7.94 | — | — | |
| Mamba-Llama3.1Attention %=50%, Size=8B2024.08 | 7.7 | 8 | 7.3 | |
| Mamba2-Llama3.1Attention %=50%, Size=8B2024.08 | 7.6 | 8.1 | 7 | |
| Mamba-Llama3Attention %=50%, Size=8B, Alignment=DPO2024.08 | 7.35 | 7.82 | 6.88 | |
| ZephyrSize=7B, Alignment=DPO2024.08 | 7.34 | — | — | |
| Mamba2-Llama3Attention %=50%, Size=8B, Alignment=DPO2024.08 | 7.32 | 7.93 | 6.7 | |
| Mamba-ZephyrAttention %=50%, Size=7B, Alignment=DPO2024.08 | 7.31 | — | — | |
| Mamba-ZephyrAttention %=25%, Size=7B, Alignment=DPO2024.08 | 7.03 | — | — | |
| Mamba-Llama3.2Attention %=50%, Size=3B2024.08 | 6.9 | 7.6 | 6.1 | |
| Mamba-Llama3Attention %=25%, Size=8B, Alignment=DPO2024.08 | 6.86 | 7.56 | 6.15 | |
| Mamba2-Llama3Attention %=25%, Size=8B, Alignment=DPO2024.08 | 6.74 | 7.24 | 6.24 | |
| Mamba2-Llama3.2Attention %=50%, Size=3B2024.08 | 6.5 | 7.1 | 5.8 | |
| Mamba2-Llama3Attention %=12.5%, Size=8B, Alignment=DPO2024.08 | 6.48 | 6.83 | 6.13 | |
| Mamba-Llama3Attention %=12.5%, Size=8B, Alignment=DPO2024.08 | 6.46 | 6.91 | 6.01 | |
| Mamba-ZephyrAttention %=12.5%, Size=7B, Alignment=DPO2024.08 | 6.4 | — | — | |
| Falcon Mamba InstructSize=7B, Alignment=SFT2024.08 | 6.4 | 7.25 | 5.55 | |
| Mamba2-Llama3Attention %=0%, Size=8B, Alignment=DPO2024.08 | 5.64 | 6.16 | 5.11 |