List your company
ResearchFormerly WizWand

Instruction Following on IF-Eval 0-shot

81.13ScoreUltraMix-190k
25.666840.065954.46568.8641Nov 14, 2025Dec 8, 2025Jan 1, 2026Jan 25, 2026Feb 18, 2026Mar 14, 2026Apr 8, 2026
Updated 6mo ago

Evaluation results

MethodLinks
UltraMix-190kBase Model=Llama-3.1-8B-TuluSFT, Training Mixture/Protocol=UM-190k2025.1181.13
TuluDPOBase Model=Llama-3.1-8B-TuluSFT, Training Mixture/Protocol=TuluDPO2025.1180.35
UltraMix-187kBase Model=Llama-3.1-8B-TuluSFT, Training Mixture/Protocol=UM-187k2025.1180.19
UltraMix-190kBase Model=Qwen-2.5-7B-TuluSFT, Training Mixture/Protocol=UM-190k2025.1179.88
UltraMix-170kBase Model=Llama-3.1-8B-TuluSFT, Training Mixture/Protocol=UM-170k2025.1179.38
UltraMix-187kBase Model=Qwen-2.5-7B-TuluSFT, Training Mixture/Protocol=UM-187k2025.1178.67
TuluDPOBase Model=Qwen-2.5-7B-TuluSFT, Training Mixture/Protocol=TuluDPO2025.1178.04
UltraMix-170kBase Model=Qwen-2.5-7B-TuluSFT, Training Mixture/Protocol=UM-170k2025.1177.28
Instella-3B-SFT (UM-190k)DPO Training Dataset=UM-190k, Evaluation Protocol=0-shot2025.1175.35
Instella-3B-SFT (UM-187k)DPO Training Dataset=UM-187k, Evaluation Protocol=0-shot2025.1174.52
Instella-3B-SFT (TuluDPO)DPO Training Dataset=TuluDPO, Evaluation Protocol=0-shot2025.1174.02
UM-190kBase Model=OLMo-2-7B-SFT, DPO Training Dataset=UM-190k2025.1173.31
UM-187kBase Model=OLMo-2-7B-SFT, DPO Training Dataset=UM-187k2025.1173.25
UltraMixTraining Method=DPO, Dataset=UM-190k, Evaluation Protocol=0-shot2025.1172.81
Instella-3B-SFT (UM-170k)DPO Training Dataset=UM-170k, Evaluation Protocol=0-shot2025.1172.54
UltraMixTraining Method=DPO, Dataset=UM-187k, Evaluation Protocol=0-shot2025.1172.49
UM-170kBase Model=OLMo-2-7B-SFT, DPO Training Dataset=UM-170k2025.1172.48
SFTBase Model=Llama-3.1-8B-TuluSFT, Training Mixture/Protocol=SFT2025.1172.45
TuluDPOBase Model=OLMo-2-7B-SFT, DPO Training Dataset=TuluDPO2025.1171.96
Instella-3B-SFT (ORPO)DPO Training Dataset=ORPO, Evaluation Protocol=0-shot2025.1171.34
TuluDPOTraining Method=DPO, Dataset=TuluDPO, Evaluation Protocol=0-shot2025.1170.19
ORPOTraining Method=ORPO, Dataset=ORPO, Evaluation Protocol=0-shot2025.1169.74
ORPOBase Model=OLMo-2-7B-SFT, DPO Training Dataset=ORPO2025.1169.31
UltraMixTraining Method=DPO, Dataset=UM-170k, Evaluation Protocol=0-shot2025.1169.25
UltraFBBase Model=OLMo-2-7B-SFT, DPO Training Dataset=UltraFB2025.1168.51
SmolLM-3-3B-SFTTraining Method=SFT, Evaluation Protocol=0-shot2025.1168.47
CodePrefBase Model=OLMo-2-7B-SFT, DPO Training Dataset=CodePref2025.1168.24
MARS-7BModel Scale=7B, Block Size (B)=4, One-token mode (τ=1.0)=true, Decoding Strategy=greedy, Max New Tokens=2562026.0468.2
SFTBase Model=Qwen-2.5-7B-TuluSFT, Training Mixture/Protocol=SFT2025.1168.06
SFTBase Model=OLMo-2-7B-SFT, DPO Training Dataset=SFT2025.1167.42
HelpSteerBase Model=OLMo-2-7B-SFT, DPO Training Dataset=HelpSteer2025.1167.42
Instella-3B-SFT (HelpSteer)DPO Training Dataset=HelpSteer, Evaluation Protocol=0-shot2025.1167.24
UM-190kTraining Dataset=UM-190k2025.1167.13
UM-187kTraining Dataset=UM-187k2025.1167.04
AR SFTModel Scale=7B, Training Protocol=AR SFT, One-token mode (τ=1.0)=true, Decoding Strategy=greedy, Max New Tokens=2562026.0467
Instella-3B-SFT (SFT)DPO Training Dataset=None (SFT Baseline), Evaluation Protocol=0-shot2025.1166.84
Instella-3B-SFT (UltraFB)DPO Training Dataset=UltraFB, Evaluation Protocol=0-shot2025.1165.57
TuluDPOTraining Dataset=TuluDPO2025.1165.56
UM-170kTraining Dataset=UM-170k2025.1165.12
Instella-3B-SFT (CodePref)DPO Training Dataset=CodePref, Evaluation Protocol=0-shot2025.1164.8
HelpSteerTraining Method=DPO, Dataset=HelpSteer, Evaluation Protocol=0-shot2025.1164.41
Base (no fine-tune)Model Scale=7B, Training Protocol=Base (no fine-tune), One-token mode (τ=1.0)=true, Decoding Strategy=greedy, Max New Tokens=2562026.0463.4
UltraFBTraining Method=DPO, Dataset=UltraFB, Evaluation Protocol=0-shot2025.1162.61
ORPOTraining Dataset=ORPO2025.1162.49
CodePrefTraining Method=DPO, Dataset=CodePref, Evaluation Protocol=0-shot2025.1161.4
UltraFBTraining Dataset=UltraFB2025.1160.76
HelpSteerTraining Dataset=HelpSteer2025.1158.07
CodePrefTraining Dataset=CodePref2025.1156.28
SFTTraining Dataset=SFT2025.1153.49
MARS-0.5BModel Scale=0.5B, Block Size (B)=4, SFT Loss Configuration=True, One-token mode (τ=1.0)=true, Decoding Strategy=greedy, Max New Tokens=2562026.0451.3
MARS-0.5B w/o SFT lossModel Scale=0.5B, Block Size (B)=4, SFT Loss Configuration=False, One-token mode (τ=1.0)=true, Decoding Strategy=greedy, Max New Tokens=2562026.0448.5
AR SFT (5 ep)Model Scale=0.5B, Training Protocol=AR SFT, Training Epochs=5, One-token mode (τ=1.0)=true, Decoding Strategy=greedy, Max New Tokens=2562026.0448.4
AR SFT (10 ep)Model Scale=0.5B, Training Protocol=AR SFT, Training Epochs=10, One-token mode (τ=1.0)=true, Decoding Strategy=greedy, Max New Tokens=2562026.0447.8
Block DiffusionModel Scale=0.5B, Block Size (B)=4, One-token mode (τ=1.0)=true, Decoding Strategy=greedy, Max New Tokens=2562026.0447.1
Base (no fine-tune)Model Scale=0.5B, Training Protocol=Base (no fine-tune), One-token mode (τ=1.0)=true, Decoding Strategy=greedy, Max New Tokens=2562026.0427.8