Multi-task Language Modeling on MergeBench
39.56Instruction ScoreTask-specific FT
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Task-specific FTModel Type=Finetuned2026.02 | 39.56 | 69.83 | 44.33 | 41.73 | 80.46 | 55.18 | |
| RegMeanBase Model=Llama-3.1-8B2025.08 | 25.9 | 55.1 | 51.5 | 51.4 | 33.6 | 43.5 | |
| TIES + OrthoMerge-CMerging Strategy=TIES, OrthoMerge Variant=C2026.02 | 25.32 | 55.8 | 40.21 | 42.26 | 44.24 | 41.57 | |
| TIES-MergingBase Model=Llama-3.2-3B2025.08 | 22 | 42.8 | 40.5 | 48.1 | 41.2 | 38.9 | |
| TIESMerging Strategy=TIES2026.02 | 20.89 | 50.8 | 39.61 | 42.11 | 42.38 | 39.16 | |
| TIES + OrthoMerge-GMerging Strategy=TIES, OrthoMerge Variant=G2026.02 | 20.33 | 51.48 | 40.28 | 42.08 | 42.51 | 39.34 | |
| Task ArithmeticBase Model=Llama-3.2-3B2025.08 | 19.8 | 37.1 | 40.1 | 48.1 | 43.6 | 37.7 | |
| TSV-M + OrthoMerge-CMerging Strategy=TSV-M, OrthoMerge Variant=C2026.02 | 19.59 | 55.88 | 41.69 | 42.32 | 50.87 | 42.07 | |
| TSV-MMerging Strategy=TSV-M2026.02 | 19.4 | 55.72 | 40.44 | 42.2 | 49.82 | 41.52 | |
| TSV-M + OrthoMerge-GMerging Strategy=TSV-M, OrthoMerge Variant=G2026.02 | 18.67 | 51.71 | 40.65 | 41.89 | 47.23 | 40.03 | |
| Model SoupsBase Model=Llama-3.1-8B2025.08 | 13.9 | 67.9 | 50.2 | 54.4 | 56.2 | 48.5 | |
| TIES-MergingBase Model=Llama-3.1-8B2025.08 | 13.1 | 67.1 | 49.2 | 54.9 | 59.5 | 48.8 | |
| RegMean++Base Model=Llama-3.1-8B2025.08 | 11.1 | 65.8 | 52.3 | 53.1 | 46.3 | 45.7 | |
| TA + OrthoMerge-GMerging Strategy=Task Arithmetic, OrthoMerge Variant=G2026.02 | 11.09 | 43.9 | 37.96 | 42.06 | 42.41 | 35.48 | |
| TAMerging Strategy=Task Arithmetic2026.02 | 10.53 | 40.4 | 37.22 | 42.26 | 40.4 | 34.16 | |
| TA + OrthoMerge-CMerging Strategy=Task Arithmetic, OrthoMerge Variant=C2026.02 | 9.8 | 40.03 | 37.75 | 42.29 | 41.69 | 34.31 | |
| Model SoupsBase Model=Llama-3.2-3B2025.08 | 8.7 | 36.2 | 37.1 | 47.8 | 36.5 | 33.3 | |
| Task ArithmeticBase Model=Llama-3.1-8B2025.08 | 8.7 | 62.2 | 47.5 | 54.8 | 53.5 | 45.3 | |
| RegMeanBase Model=Llama-3.2-3B2025.08 | 8.3 | 35.5 | 39.2 | 47.3 | 39.8 | 34 | |
| Llama-3.2-3BModel Type=Pre-trained Base Model2026.02 | 7.58 | 28.51 | 27.44 | 40.72 | 31.41 | 27.18 | |
| RegMean++Base Model=Llama-3.2-3B2025.08 | 5.9 | 32.1 | 38.1 | 47.1 | 36.3 | 31.9 |