Natural Language Understanding on GLUE & SuperGLUE (test)
83RTE AccuracyPEMT
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PEMTBackbone=T5-Base, Tuning Mode=Knowledge Transfer PEFT2024.02 | 83 | 91.1 | 88.7 | 67 | 75.5 | 82.6 | 68.7 | 67.3 | 94.1 | 79.8 | — | — | |
| MPTBackbone=T5-Base, Tuning Mode=Prompt-based tuning2024.02 | 79.4 | 90.4 | 89.1 | 62.4 | 74.8 | 79.6 | 69 | 67.3 | 79.8 | 76.9 | — | — | |
| FOMethod Type=First-Order Fine-tuning, Model=Pangu-1B2026.01 | 78 | — | — | — | — | 75.1 | 65.7 | — | 69.6 | — | 82.2 | 80 | |
| ConMeZOModel=RoBERTa-Large, Seeds=5 seeds, Hyperparameter Selection=default configuration2025.11 | 75.1 | — | — | — | — | — | — | — | — | 76.3 | 93.5 | — | |
| LoRABackbone=T5-Base, Tuning Mode=Parameter-efficient tuning2024.02 | 74.1 | 91.1 | 86.8 | 61.5 | 75.2 | 81.8 | 69.2 | 65.4 | 85.7 | 76.7 | — | — | |
| AGZOMethod Type=Zeroth-Order Fine-tuning, Model=Pangu-1B2026.01 | 73.6 | — | — | — | — | 73 | 57.5 | — | 73.2 | — | 77.8 | 77 | |
| ATTEMPTBackbone=T5-Base, Tuning Mode=Prompt-based tuning2024.02 | 73.4 | 89.7 | 85.7 | 57.4 | 74.4 | 77.1 | 66.8 | 53.8 | 78.6 | 73 | — | — | |
| MEZOMethod Type=Zeroth-Order Fine-tuning, Model=Pangu-1B2026.01 | 72.9 | — | — | — | — | 69.9 | 56.7 | — | 73.2 | — | 76.4 | 75 | |
| FTBackbone=T5-Base, Tuning Mode=Full fine-tuning2024.02 | 71.9 | 89.7 | 89.1 | 61.8 | 72.8 | 81.1 | 70.2 | 59.6 | 85.7 | 75.8 | — | — | |
| AdapterBackbone=T5-Base, Tuning Mode=Vanilla adapter-based tuning2024.02 | 71.9 | 90.7 | 85.3 | 64 | 75.9 | 82.5 | 67.1 | 67.3 | 85.7 | 76.7 | — | — | |
| AdaMixBackbone=T5-Base, Tuning Mode=Adapter-based tuning2024.02 | 70.5 | 91 | 88.2 | 58.7 | 72.9 | 80.2 | 63.6 | 51.9 | 85.7 | 73.6 | — | — | |
| SPoTBackbone=T5-Base, Tuning Mode=Prompt-based tuning2024.02 | 69.8 | 90 | 79.7 | 57.1 | 74 | 77.2 | 48.9 | 51.9 | 67.9 | 68.5 | — | — | |
| LOZOMethod Type=Zeroth-Order Fine-tuning, Model=Pangu-1B2026.01 | 69.7 | — | — | — | — | 69.6 | 56.3 | — | 67.9 | — | 72 | 77 | |
| ICLEvaluation Protocol=In-context learning, Model=Pangu-1B2026.01 | 68.2 | — | — | — | — | 73.5 | 51.1 | — | 44.6 | — | 71.7 | 75 | |
| BitFitBackbone=T5-Base, Tuning Mode=Parameter-efficient tuning2024.02 | 67.6 | 90.9 | 86.8 | 58.2 | 74.5 | 79.6 | 70 | 59.6 | 78.6 | 74 | — | — | |
| MixDABackbone=T5-Base, Tuning Mode=Adapter-based tuning2024.02 | 66.9 | 90.8 | 88.2 | 60.8 | 59.2 | 61.7 | 48.9 | 50 | 78.6 | 67.2 | — | — | |
| ZeroEvaluation Protocol=Zero-shot prompting, Model=Pangu-1B2026.01 | 58.1 | — | — | — | — | 69.5 | 46.6 | — | 50 | — | 56.8 | 76 | |
| PTBackbone=T5-Base, Tuning Mode=Vanilla prompt tuning2024.02 | 54.7 | 89.5 | 68.1 | 10.6 | 58.7 | 61.7 | 48.9 | 51.9 | 67.9 | 56.9 | — | — | |
| HiZOOModel=RoBERTa-Large, Seeds=5 seeds, Hyperparameter Selection=best across learning rates {10^-5, 10^-6, 10^-7}2025.11 | 54.2 | — | — | — | — | — | — | — | — | 69.6 | 83.4 | — | |
| ConMeZOModel=OPT-1.3B, Seeds=3 seeds, Hyperparameter Selection=default configuration2025.11 | — | — | — | — | — | 64.2 | 58.2 | — | — | 76.3 | 90.6 | — | |
| HiZOOModel=OPT-1.3B, Seeds=3 seeds, Hyperparameter Selection=best across learning rates {10^-5, 10^-6, 10^-7}2025.11 | — | — | — | — | — | 63 | 59 | — | — | 69.6 | 88.5 | — |