Multi-task Language Understanding Suite (PAWS, QASC, QuaRTz, StoryCloze, WikiQA, Winogrande, WSC)
95.5PAWS AccuracyTM-TIES + SiM
Evaluation Results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| TM-TIES + SiMBackbone=T5-large, Merging Strategy=Dynamic model merging2026.06 | 95.5 | 97 | 83.5 | 89.2 | 94.2 | 91.1 | 49 | 85.6 | |
| EMR + SiMBackbone=T5-large, Merging Strategy=Dynamic model merging2026.06 | 95 | 95.8 | 83.5 | 87.8 | 92.4 | 80.6 | 59 | 84.9 | |
| TSV-C + SiMBackbone=T5-large, Merging Strategy=Dynamic model merging2026.06 | 95 | 97.6 | 84.5 | 89.7 | 94.4 | 74.5 | 55 | 84.4 | |
| MoW-MergingBackbone=T5-large, Merging Strategy=Dynamic model merging2026.06 | 94.7 | 90.2 | 79 | 83.1 | 56.6 | 93.8 | 51 | 78.3 | |
| Fine-tunedBackbone=T5-large, Method Category=Task-specific baselines2026.06 | 94.4 | 98.9 | 87.8 | 90.8 | 96 | 74.7 | 79.2 | 88.8 | |
| WEMoEBackbone=T5-large, Merging Strategy=Dynamic model merging2026.06 | 93.4 | 95.4 | 74.5 | 79.6 | 94.1 | 90.4 | 57 | 83.4 | |
| TWIN-MergingBackbone=T5-large, Merging Strategy=Dynamic model merging2026.06 | 93.2 | 96 | 87.1 | 85.6 | 77.1 | 70.8 | 69.7 | 82.8 | |
| TM-TA + SiMBackbone=T5-large, Merging Strategy=Dynamic model merging2026.06 | 88.3 | 95.6 | 81.5 | 88 | 95.2 | 87.2 | 53 | 84.1 | |
| DaWinBackbone=T5-large, Merging Strategy=Dynamic model merging2026.06 | 85.6 | 96.1 | 81.4 | 73.2 | 70.2 | 68.7 | 57.9 | 76.2 | |
| TSV-MBackbone=T5-large, Merging Strategy=Static model merging2026.06 | 85 | 92.4 | 66 | 81.1 | 88.4 | 87 | 57 | 79.6 | |
| TIES-MergingBackbone=T5-large, Merging Strategy=Static model merging2026.06 | 81.5 | 96.2 | 80.1 | 83.6 | 64.9 | 66.5 | 65.3 | 76.9 | |
| Task ArithmeticBackbone=T5-large, Merging Strategy=Static model merging2026.06 | 77.8 | 96 | 78.6 | 86.4 | 59.1 | 62.3 | 52.8 | 73.3 | |
| Iso-CBackbone=T5-large, Merging Strategy=Static model merging2026.06 | 72.9 | 94.8 | 50.5 | 80.7 | 92.5 | 75.7 | 54 | 74.4 |