Multi-task Language Modeling on MergeBench benchmark leaderboard · SOTA2 Research

Multi-task Language Modeling on MergeBench

39.56Instruction ScoreTask-specific FT

Evaluation Results

Method	Links
Task-specific FT2026.02		39.56	69.83	44.33	41.73	80.46	55.18
RegMean2025.08		25.9	55.1	51.5	51.4	33.6	43.5
TIES + OrthoMerge-C2026.02		25.32	55.8	40.21	42.26	44.24	41.57
TIES-Merging2025.08		22	42.8	40.5	48.1	41.2	38.9
TIES2026.02		20.89	50.8	39.61	42.11	42.38	39.16
TIES + OrthoMerge-G2026.02		20.33	51.48	40.28	42.08	42.51	39.34
Task Arithmetic2025.08		19.8	37.1	40.1	48.1	43.6	37.7
TSV-M + OrthoMerge-C2026.02		19.59	55.88	41.69	42.32	50.87	42.07
TSV-M2026.02		19.4	55.72	40.44	42.2	49.82	41.52
TSV-M + OrthoMerge-G2026.02		18.67	51.71	40.65	41.89	47.23	40.03
Model Soups2025.08		13.9	67.9	50.2	54.4	56.2	48.5
TIES-Merging2025.08		13.1	67.1	49.2	54.9	59.5	48.8
RegMean++2025.08		11.1	65.8	52.3	53.1	46.3	45.7
TA + OrthoMerge-G2026.02		11.09	43.9	37.96	42.06	42.41	35.48
TA2026.02		10.53	40.4	37.22	42.26	40.4	34.16
TA + OrthoMerge-C2026.02		9.8	40.03	37.75	42.29	41.69	34.31
Model Soups2025.08		8.7	36.2	37.1	47.8	36.5	33.3
Task Arithmetic2025.08		8.7	62.2	47.5	54.8	53.5	45.3
RegMean2025.08		8.3	35.5	39.2	47.3	39.8	34
Llama-3.2-3B2026.02		7.58	28.51	27.44	40.72	31.41	27.18
RegMean++2025.08		5.9	32.1	38.1	47.1	36.3	31.9