Natural Language Understanding on SuperGLUE (MultiRC, BoolQ, WiC, WSC, CB subset)
75.9MultiRC ScoreAdapter
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Adapter#Param/Task=1.9M, Learning Setup=Single-Task Learning, Backbone=T5-base2025.02 | 75.9 | 82.5 | 67.1 | 67.3 | 85.7 | 75.7 | |
| MPT#Param/Task=77.6K, Learning Setup=Transfer Learning, Backbone=T5-base2025.02 | 74.8 | 79.6 | 69 | 67.3 | 79.8 | 74.1 | |
| MPT#Param/Task=10.5K, Learning Setup=Multi-Task & Transfer Learning, Backbone=T5-base2025.02 | 74.8 | 79.6 | 70.2 | 67.3 | 89.3 | 76.1 | |
| BitFit#Param/Task=280K, Learning Setup=Single-Task Learning, Backbone=T5-base2025.02 | 74.5 | 79.6 | 70 | 59.6 | 78.6 | 72.5 | |
| DPT#Param/Task=9.0K, Learning Setup=Single-Task Learning, Backbone=T5-base, Rank (r)=102025.02 | 74.5 | 78.7 | 66.8 | 67.3 | 71.4 | 71.7 | |
| Fine-tuning#Param/Task=28M, Learning Setup=Multi-Task Learning, Backbone=T5-base2025.02 | 74.4 | 81.1 | 70 | 71.2 | 85.7 | 76.1 | |
| ATTEMPT#Param/Task=232K, Learning Setup=Transfer Learning, Backbone=T5-base2025.02 | 74.4 | 78.8 | 66.8 | 53.8 | 78.6 | 70.5 | |
| ATTEMPT#Param/Task=96K, Learning Setup=Multi-Task & Transfer Learning, Backbone=T5-base2025.02 | 74.4 | 78.5 | 66.5 | 69.2 | 82.1 | 74.1 | |
| DePT#Param/Task=76.8K, Learning Setup=Single-Task Learning, Backbone=T5-base2025.02 | 74.3 | 79.3 | 68.7 | 67.3 | 92.9 | 76.5 | |
| ULPT#Param/Task=27.1K, Learning Setup=Single-Task Learning, Backbone=T5-base, Rank (r)=2562025.02 | 74.3 | 79.9 | 63.3 | 67.3 | 89.3 | 74.8 | |
| SPoT#Param/Task=76.8K, Learning Setup=Transfer Learning, Backbone=T5-base2025.02 | 74 | 77.2 | 67 | 50 | 46.4 | 62.9 | |
| ULPT#Param/Task=1.7K, Learning Setup=Single-Task Learning, Backbone=T5-base, Rank (r)=22025.02 | 73.4 | 76.7 | 67.4 | 67.3 | 71.4 | 71.2 | |
| DPT#Param/Task=55.6K, Learning Setup=Single-Task Learning, Backbone=T5-base, Rank (r)=642025.02 | 73.2 | 80.1 | 63 | 67.3 | 85.7 | 73.9 | |
| ULPT#Param/Task=7.9K, Learning Setup=Single-Task Learning, Backbone=T5-base, Rank (r)=642025.02 | 73.1 | 78.2 | 69 | 67.3 | 96.4 | 76.8 | |
| AdapterDrop#Param/Task=1.1M, Learning Setup=Single-Task Learning, Backbone=T5-base2025.02 | 72.9 | 82.3 | 68.3 | 67.3 | 85.7 | 75.3 | |
| HyperFormer#Param/Task=638K, Learning Setup=Multi-Task Learning, Backbone=T5-base2025.02 | 72.9 | 82.5 | 69 | 67.3 | 85.7 | 75.4 | |
| Fine-tuning#Param/Task=220M, Learning Setup=Single-Task Learning, Backbone=T5-base2025.02 | 72.8 | 81.1 | 70.2 | 59.6 | 85.7 | 73.9 | |
| LoRA#Param/Task=3.8M, Learning Setup=Single-Task Learning, Backbone=T5-base2025.02 | 72.6 | 81.3 | 68.3 | 67.3 | 92.9 | 76.5 | |
| ULPT#Param/Task=3.1K, Learning Setup=Single-Task Learning, Backbone=T5-base, Rank (r)=162025.02 | 72.6 | 77.7 | 66.1 | 67.3 | 89.3 | 74.6 | |
| Adapter#Param/Task=1.8M, Learning Setup=Multi-Task Learning, Backbone=T5-base2025.02 | 72.6 | 82.3 | 66.5 | 67.3 | 89.3 | 75.6 | |
| PT#Param/Task=76.8K, Learning Setup=Single-Task Learning, Backbone=T5-base2025.02 | 72.3 | 80.4 | 67.7 | 67.3 | 78.6 | 73.3 | |
| HyperDecoder#Param/Task=1.8M, Learning Setup=Multi-Task Learning, Backbone=T5-base2025.02 | 70.4 | 78.8 | 67.1 | 61.5 | 82.1 | 72 |