Instruction Following on MT-Bench v1.0 (test)
61.2MT-Bench ScoreCaR
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| CaRBase Model=Qwen-2, Data Ratio=5%2025.02 | 61.2 | — | — | |
| NOVABase Model=Qwen-2, Data Ratio=5%2025.02 | 60.8 | — | — | |
| NuggetsBase Model=Qwen-2, Data Ratio=5%2025.02 | 60.3 | — | — | |
| CaRBase Model=Qwen-2, Data Ratio=10%2025.02 | 60.1 | — | — | |
| IFDBase Model=Qwen-2, Data Ratio=10%2025.02 | 59.8 | — | — | |
| IFDBase Model=Qwen-2, Data Ratio=5%2025.02 | 59.5 | — | — | |
| IFDBase Model=Qwen-2, Data Ratio=15%2025.02 | 59.3 | — | — | |
| NOVABase Model=Qwen-2, Data Ratio=15%2025.02 | 59.2 | — | — | |
| NuggetsBase Model=Qwen-2, Data Ratio=10%2025.02 | 58.8 | — | — | |
| NuggetsBase Model=Qwen-2, Data Ratio=15%2025.02 | 58.5 | — | — | |
| NOVABase Model=Qwen-2, Data Ratio=10%2025.02 | 58.4 | — | — | |
| CaRBase Model=Qwen-2, Data Ratio=15%2025.02 | 57.5 | — | — | |
| CaRBase Model=LLaMA-1, Data Ratio=15%2025.02 | 50.3 | — | — | |
| VanillaBase Model=Qwen-2, Data Ratio=100%2025.02 | 50.2 | — | — | |
| CaRBase Model=LLaMA-1, Data Ratio=5%2025.02 | 50.1 | — | — | |
| NOVABase Model=LLaMA-1, Data Ratio=5%2025.02 | 49.8 | — | — | |
| CaRBase Model=LLaMA-1, Data Ratio=10%2025.02 | 49.5 | — | — | |
| NuggetsBase Model=LLaMA-1, Data Ratio=15%2025.02 | 49.5 | — | — | |
| SELF-EVALBase Model=Qwen-22025.02 | 49.5 | — | — | |
| NOVABase Model=LLaMA-1, Data Ratio=10%2025.02 | 49.3 | — | — | |
| NuggetsBase Model=LLaMA-1, Data Ratio=5%2025.02 | 48.6 | — | — | |
| IFDBase Model=LLaMA-1, Data Ratio=15%2025.02 | 48.5 | — | — | |
| NuggetsBase Model=LLaMA-1, Data Ratio=10%2025.02 | 48.4 | — | — | |
| IFDBase Model=LLaMA-1, Data Ratio=5%2025.02 | 48.3 | — | — | |
| NOVABase Model=LLaMA-1, Data Ratio=15%2025.02 | 48.3 | — | — | |
| IFDBase Model=LLaMA-1, Data Ratio=10%2025.02 | 47.9 | — | — | |
| VanillaBase Model=LLaMA-1, Data Ratio=100%2025.02 | 47.8 | — | — | |
| FLAME-DPOBase Model=Qwen-22025.02 | 47.8 | — | — | |
| SELF-EVALBase Model=LLaMA-12025.02 | 42.2 | — | — | |
| FLAME-DPOBase Model=LLaMA-12025.02 | 40.6 | — | — | |
| BAdamBase Model=Llama 3-8B, Initial Learning Rate=1e-6, Training Epoch=Epoch 32024.04 | 6.67 | — | — | |
| BAdamBase Model=Llama 3-8B, Initial Learning Rate=1e-6, Training Epoch=Epoch 12024.04 | 6.65 | — | — | |
| BAdamBase Model=Llama 3-8B, Initial Learning Rate=1e-6, Training Epoch=Average2024.04 | 6.65 | — | — | |
| BAdamBase Model=Llama 3-8B, Initial Learning Rate=1e-6, Training Epoch=Epoch 22024.04 | 6.64 | — | — | |
| BAdamBase Model=Llama 3-8B, Initial Learning Rate=1e-5, Training Epoch=Average2024.04 | 6.3 | — | — | |
| LoRABase Model=Llama 3-8B, Initial Learning Rate=1e-6, Training Epoch=Average2024.04 | 6.27 | — | — | |
| LoRABase Model=Llama 3-8B, Initial Learning Rate=1e-5, Training Epoch=Average2024.04 | 6.27 | — | — | |
| AdamBase Model=Llama 3-8B, Initial Learning Rate=1e-6, Training Epoch=Average2024.04 | 6.23 | — | — | |
| GaloreBase Model=Llama 3-8B, Initial Learning Rate=1e-5, Training Epoch=Average2024.04 | 5.82 | — | — | |
| GaloreBase Model=Llama 3-8B, Initial Learning Rate=1e-6, Training Epoch=Average2024.04 | 5.71 | — | — | |
| LOMOBase Model=Llama 3-8B, Initial Learning Rate=1e-6, Training Epoch=Average2024.04 | 5.69 | — | — | |
| LOMOBase Model=Llama 3-8B, Initial Learning Rate=1e-5, Training Epoch=Average2024.04 | 5.51 | — | — | |
| Base ModelBase Model=Llama 3-8B2024.04 | 5.46 | — | — | |
| BAdamBase Model=Llama 2-7B, Initial Learning Rate=1e-5, Training Epoch=Epoch 22024.04 | 5.21 | — | — | |
| AdamBase Model=Llama 2-7B, Initial Learning Rate=1e-5, Training Epoch=Epoch 32024.04 | 5.16 | — | — | |
| BAdamBase Model=Llama 2-7B, Initial Learning Rate=1e-5, Training Epoch=Average2024.04 | 4.96 | — | — | |
| BAdamBase Model=Llama 2-7B, Initial Learning Rate=1e-5, Training Epoch=Epoch 32024.04 | 4.87 | — | — | |
| BAdamBase Model=Llama 2-7B, Initial Learning Rate=1e-5, Training Epoch=Epoch 12024.04 | 4.79 | — | — | |
| AdamBase Model=Llama 2-7B, Initial Learning Rate=1e-5, Training Epoch=Average2024.04 | 4.76 | — | — | |
| AdamBase Model=Llama 2-7B, Initial Learning Rate=1e-5, Training Epoch=Epoch 22024.04 | 4.73 | — | — | |
| AdamBase Model=Llama 2-7B, Initial Learning Rate=1e-5, Training Epoch=Epoch 12024.04 | 4.41 | — | — | |
| Base ModelBase Model=Llama 2-7B2024.04 | 3.93 | — | — | |
| EESD (+Beta-TS)Target Model=LLaMA-2-13B, Draft Model=Self, Trainable Params=481M2024.06 | — | 62.15 | 2.03 | |
| EESD (+Beta-TS)Target Model=Vicuna-13B, Draft Model=Self, Trainable Params=481M2024.06 | — | 52.82 | 1.82 | |
| EESD (+Cali-TS)Target Model=LLaMA-2-13B, Draft Model=Self, Trainable Params=481M+262M2024.06 | — | 64.31 | 2.11 | |
| EESD (+Cali-TS)Target Model=Vicuna-13B, Draft Model=Self, Trainable Params=481M+262M2024.06 | — | 54.83 | 1.89 | |
| MedusaTarget Model=LLaMA-2-13B, Draft Model=Self, Trainable Params=760M2024.06 | — | 43.08 | 1.89 | |
| MedusaTarget Model=Vicuna-13B, Draft Model=Self, Trainable Params=761M2024.06 | — | 30.11 | 1.71 | |
| Self SDTarget Model=LLaMA-2-13B, Draft Model=Self2024.06 | — | 89.25 | 1.58 | |
| Self SDTarget Model=Vicuna-13B, Draft Model=Self2024.06 | — | 86.2 | 1.46 | |
| Vanilla SDTarget Model=LLaMA-2-13B, Draft Model=LLaMA-2-7B, Trainable Params=7B+2024.06 | — | 90.01 | 1.02 | |
| Vanilla SDTarget Model=Vicuna-13B, Draft Model=Vicuna-7B, Trainable Params=7B+2024.06 | — | 81.5 | 0.91 |