Instruction Following on IF-Eval 0-shot
81.13ScoreUltraMix-190k
Evaluation Results
| Method | Links | |
|---|---|---|
| UltraMix-190kBase Model=Llama-3.1-8B-TuluSFT, Training Mixture/Protocol=UM-190k2025.11 | 81.13 | |
| TuluDPOBase Model=Llama-3.1-8B-TuluSFT, Training Mixture/Protocol=TuluDPO2025.11 | 80.35 | |
| UltraMix-187kBase Model=Llama-3.1-8B-TuluSFT, Training Mixture/Protocol=UM-187k2025.11 | 80.19 | |
| UltraMix-190kBase Model=Qwen-2.5-7B-TuluSFT, Training Mixture/Protocol=UM-190k2025.11 | 79.88 | |
| UltraMix-170kBase Model=Llama-3.1-8B-TuluSFT, Training Mixture/Protocol=UM-170k2025.11 | 79.38 | |
| UltraMix-187kBase Model=Qwen-2.5-7B-TuluSFT, Training Mixture/Protocol=UM-187k2025.11 | 78.67 | |
| TuluDPOBase Model=Qwen-2.5-7B-TuluSFT, Training Mixture/Protocol=TuluDPO2025.11 | 78.04 | |
| UltraMix-170kBase Model=Qwen-2.5-7B-TuluSFT, Training Mixture/Protocol=UM-170k2025.11 | 77.28 | |
| Instella-3B-SFT (UM-190k)DPO Training Dataset=UM-190k, Evaluation Protocol=0-shot2025.11 | 75.35 | |
| Instella-3B-SFT (UM-187k)DPO Training Dataset=UM-187k, Evaluation Protocol=0-shot2025.11 | 74.52 | |
| Instella-3B-SFT (TuluDPO)DPO Training Dataset=TuluDPO, Evaluation Protocol=0-shot2025.11 | 74.02 | |
| UM-190kBase Model=OLMo-2-7B-SFT, DPO Training Dataset=UM-190k2025.11 | 73.31 | |
| UM-187kBase Model=OLMo-2-7B-SFT, DPO Training Dataset=UM-187k2025.11 | 73.25 | |
| UltraMixTraining Method=DPO, Dataset=UM-190k, Evaluation Protocol=0-shot2025.11 | 72.81 | |
| Instella-3B-SFT (UM-170k)DPO Training Dataset=UM-170k, Evaluation Protocol=0-shot2025.11 | 72.54 | |
| UltraMixTraining Method=DPO, Dataset=UM-187k, Evaluation Protocol=0-shot2025.11 | 72.49 | |
| UM-170kBase Model=OLMo-2-7B-SFT, DPO Training Dataset=UM-170k2025.11 | 72.48 | |
| SFTBase Model=Llama-3.1-8B-TuluSFT, Training Mixture/Protocol=SFT2025.11 | 72.45 | |
| TuluDPOBase Model=OLMo-2-7B-SFT, DPO Training Dataset=TuluDPO2025.11 | 71.96 | |
| Instella-3B-SFT (ORPO)DPO Training Dataset=ORPO, Evaluation Protocol=0-shot2025.11 | 71.34 | |
| TuluDPOTraining Method=DPO, Dataset=TuluDPO, Evaluation Protocol=0-shot2025.11 | 70.19 | |
| ORPOTraining Method=ORPO, Dataset=ORPO, Evaluation Protocol=0-shot2025.11 | 69.74 | |
| ORPOBase Model=OLMo-2-7B-SFT, DPO Training Dataset=ORPO2025.11 | 69.31 | |
| UltraMixTraining Method=DPO, Dataset=UM-170k, Evaluation Protocol=0-shot2025.11 | 69.25 | |
| UltraFBBase Model=OLMo-2-7B-SFT, DPO Training Dataset=UltraFB2025.11 | 68.51 | |
| SmolLM-3-3B-SFTTraining Method=SFT, Evaluation Protocol=0-shot2025.11 | 68.47 | |
| CodePrefBase Model=OLMo-2-7B-SFT, DPO Training Dataset=CodePref2025.11 | 68.24 | |
| MARS-7BModel Scale=7B, Block Size (B)=4, One-token mode (τ=1.0)=true, Decoding Strategy=greedy, Max New Tokens=2562026.04 | 68.2 | |
| SFTBase Model=Qwen-2.5-7B-TuluSFT, Training Mixture/Protocol=SFT2025.11 | 68.06 | |
| SFTBase Model=OLMo-2-7B-SFT, DPO Training Dataset=SFT2025.11 | 67.42 | |
| HelpSteerBase Model=OLMo-2-7B-SFT, DPO Training Dataset=HelpSteer2025.11 | 67.42 | |
| Instella-3B-SFT (HelpSteer)DPO Training Dataset=HelpSteer, Evaluation Protocol=0-shot2025.11 | 67.24 | |
| UM-190kTraining Dataset=UM-190k2025.11 | 67.13 | |
| UM-187kTraining Dataset=UM-187k2025.11 | 67.04 | |
| AR SFTModel Scale=7B, Training Protocol=AR SFT, One-token mode (τ=1.0)=true, Decoding Strategy=greedy, Max New Tokens=2562026.04 | 67 | |
| Instella-3B-SFT (SFT)DPO Training Dataset=None (SFT Baseline), Evaluation Protocol=0-shot2025.11 | 66.84 | |
| Instella-3B-SFT (UltraFB)DPO Training Dataset=UltraFB, Evaluation Protocol=0-shot2025.11 | 65.57 | |
| TuluDPOTraining Dataset=TuluDPO2025.11 | 65.56 | |
| UM-170kTraining Dataset=UM-170k2025.11 | 65.12 | |
| Instella-3B-SFT (CodePref)DPO Training Dataset=CodePref, Evaluation Protocol=0-shot2025.11 | 64.8 | |
| HelpSteerTraining Method=DPO, Dataset=HelpSteer, Evaluation Protocol=0-shot2025.11 | 64.41 | |
| Base (no fine-tune)Model Scale=7B, Training Protocol=Base (no fine-tune), One-token mode (τ=1.0)=true, Decoding Strategy=greedy, Max New Tokens=2562026.04 | 63.4 | |
| UltraFBTraining Method=DPO, Dataset=UltraFB, Evaluation Protocol=0-shot2025.11 | 62.61 | |
| ORPOTraining Dataset=ORPO2025.11 | 62.49 | |
| CodePrefTraining Method=DPO, Dataset=CodePref, Evaluation Protocol=0-shot2025.11 | 61.4 | |
| UltraFBTraining Dataset=UltraFB2025.11 | 60.76 | |
| HelpSteerTraining Dataset=HelpSteer2025.11 | 58.07 | |
| CodePrefTraining Dataset=CodePref2025.11 | 56.28 | |
| SFTTraining Dataset=SFT2025.11 | 53.49 | |
| MARS-0.5BModel Scale=0.5B, Block Size (B)=4, SFT Loss Configuration=True, One-token mode (τ=1.0)=true, Decoding Strategy=greedy, Max New Tokens=2562026.04 | 51.3 | |
| MARS-0.5B w/o SFT lossModel Scale=0.5B, Block Size (B)=4, SFT Loss Configuration=False, One-token mode (τ=1.0)=true, Decoding Strategy=greedy, Max New Tokens=2562026.04 | 48.5 | |
| AR SFT (5 ep)Model Scale=0.5B, Training Protocol=AR SFT, Training Epochs=5, One-token mode (τ=1.0)=true, Decoding Strategy=greedy, Max New Tokens=2562026.04 | 48.4 | |
| AR SFT (10 ep)Model Scale=0.5B, Training Protocol=AR SFT, Training Epochs=10, One-token mode (τ=1.0)=true, Decoding Strategy=greedy, Max New Tokens=2562026.04 | 47.8 | |
| Block DiffusionModel Scale=0.5B, Block Size (B)=4, One-token mode (τ=1.0)=true, Decoding Strategy=greedy, Max New Tokens=2562026.04 | 47.1 | |
| Base (no fine-tune)Model Scale=0.5B, Training Protocol=Base (no fine-tune), One-token mode (τ=1.0)=true, Decoding Strategy=greedy, Max New Tokens=2562026.04 | 27.8 |