List your company
ResearchFormerly WizWand

Instruction Following on MT-Bench v1.0 (test)

61.2MT-Bench ScoreCaR
1.639217.102132.56548.0279Apr 3, 2024May 25, 2024Jul 16, 2024Sep 7, 2024Oct 29, 2024Dec 20, 2024Feb 11, 2025
Updated 8mo ago

Evaluation results

MethodLinks
CaRBase Model=Qwen-2, Data Ratio=5%2025.0261.2——
NOVABase Model=Qwen-2, Data Ratio=5%2025.0260.8——
NuggetsBase Model=Qwen-2, Data Ratio=5%2025.0260.3——
CaRBase Model=Qwen-2, Data Ratio=10%2025.0260.1——
IFDBase Model=Qwen-2, Data Ratio=10%2025.0259.8——
IFDBase Model=Qwen-2, Data Ratio=5%2025.0259.5——
IFDBase Model=Qwen-2, Data Ratio=15%2025.0259.3——
NOVABase Model=Qwen-2, Data Ratio=15%2025.0259.2——
NuggetsBase Model=Qwen-2, Data Ratio=10%2025.0258.8——
NuggetsBase Model=Qwen-2, Data Ratio=15%2025.0258.5——
NOVABase Model=Qwen-2, Data Ratio=10%2025.0258.4——
CaRBase Model=Qwen-2, Data Ratio=15%2025.0257.5——
CaRBase Model=LLaMA-1, Data Ratio=15%2025.0250.3——
VanillaBase Model=Qwen-2, Data Ratio=100%2025.0250.2——
CaRBase Model=LLaMA-1, Data Ratio=5%2025.0250.1——
NOVABase Model=LLaMA-1, Data Ratio=5%2025.0249.8——
CaRBase Model=LLaMA-1, Data Ratio=10%2025.0249.5——
NuggetsBase Model=LLaMA-1, Data Ratio=15%2025.0249.5——
SELF-EVALBase Model=Qwen-22025.0249.5——
NOVABase Model=LLaMA-1, Data Ratio=10%2025.0249.3——
NuggetsBase Model=LLaMA-1, Data Ratio=5%2025.0248.6——
IFDBase Model=LLaMA-1, Data Ratio=15%2025.0248.5——
NuggetsBase Model=LLaMA-1, Data Ratio=10%2025.0248.4——
IFDBase Model=LLaMA-1, Data Ratio=5%2025.0248.3——
NOVABase Model=LLaMA-1, Data Ratio=15%2025.0248.3——
IFDBase Model=LLaMA-1, Data Ratio=10%2025.0247.9——
VanillaBase Model=LLaMA-1, Data Ratio=100%2025.0247.8——
FLAME-DPOBase Model=Qwen-22025.0247.8——
SELF-EVALBase Model=LLaMA-12025.0242.2——
FLAME-DPOBase Model=LLaMA-12025.0240.6——
BAdamBase Model=Llama 3-8B, Initial Learning Rate=1e-6, Training Epoch=Epoch 32024.046.67——
BAdamBase Model=Llama 3-8B, Initial Learning Rate=1e-6, Training Epoch=Epoch 12024.046.65——
BAdamBase Model=Llama 3-8B, Initial Learning Rate=1e-6, Training Epoch=Average2024.046.65——
BAdamBase Model=Llama 3-8B, Initial Learning Rate=1e-6, Training Epoch=Epoch 22024.046.64——
BAdamBase Model=Llama 3-8B, Initial Learning Rate=1e-5, Training Epoch=Average2024.046.3——
LoRABase Model=Llama 3-8B, Initial Learning Rate=1e-6, Training Epoch=Average2024.046.27——
LoRABase Model=Llama 3-8B, Initial Learning Rate=1e-5, Training Epoch=Average2024.046.27——
AdamBase Model=Llama 3-8B, Initial Learning Rate=1e-6, Training Epoch=Average2024.046.23——
GaloreBase Model=Llama 3-8B, Initial Learning Rate=1e-5, Training Epoch=Average2024.045.82——
GaloreBase Model=Llama 3-8B, Initial Learning Rate=1e-6, Training Epoch=Average2024.045.71——
LOMOBase Model=Llama 3-8B, Initial Learning Rate=1e-6, Training Epoch=Average2024.045.69——
LOMOBase Model=Llama 3-8B, Initial Learning Rate=1e-5, Training Epoch=Average2024.045.51——
Base ModelBase Model=Llama 3-8B2024.045.46——
BAdamBase Model=Llama 2-7B, Initial Learning Rate=1e-5, Training Epoch=Epoch 22024.045.21——
AdamBase Model=Llama 2-7B, Initial Learning Rate=1e-5, Training Epoch=Epoch 32024.045.16——
BAdamBase Model=Llama 2-7B, Initial Learning Rate=1e-5, Training Epoch=Average2024.044.96——
BAdamBase Model=Llama 2-7B, Initial Learning Rate=1e-5, Training Epoch=Epoch 32024.044.87——
BAdamBase Model=Llama 2-7B, Initial Learning Rate=1e-5, Training Epoch=Epoch 12024.044.79——
AdamBase Model=Llama 2-7B, Initial Learning Rate=1e-5, Training Epoch=Average2024.044.76——
AdamBase Model=Llama 2-7B, Initial Learning Rate=1e-5, Training Epoch=Epoch 22024.044.73——
AdamBase Model=Llama 2-7B, Initial Learning Rate=1e-5, Training Epoch=Epoch 12024.044.41——
Base ModelBase Model=Llama 2-7B2024.043.93——
EESD (+Beta-TS)Target Model=LLaMA-2-13B, Draft Model=Self, Trainable Params=481M2024.06—62.152.03
EESD (+Beta-TS)Target Model=Vicuna-13B, Draft Model=Self, Trainable Params=481M2024.06—52.821.82
EESD (+Cali-TS)Target Model=LLaMA-2-13B, Draft Model=Self, Trainable Params=481M+262M2024.06—64.312.11
EESD (+Cali-TS)Target Model=Vicuna-13B, Draft Model=Self, Trainable Params=481M+262M2024.06—54.831.89
MedusaTarget Model=LLaMA-2-13B, Draft Model=Self, Trainable Params=760M2024.06—43.081.89
MedusaTarget Model=Vicuna-13B, Draft Model=Self, Trainable Params=761M2024.06—30.111.71
Self SDTarget Model=LLaMA-2-13B, Draft Model=Self2024.06—89.251.58
Self SDTarget Model=Vicuna-13B, Draft Model=Self2024.06—86.21.46
Vanilla SDTarget Model=LLaMA-2-13B, Draft Model=LLaMA-2-7B, Trainable Params=7B+2024.06—90.011.02
Vanilla SDTarget Model=Vicuna-13B, Draft Model=Vicuna-7B, Trainable Params=7B+2024.06—81.50.91