Multitask Evaluation on Aggregate (MMLU, GSM8K, SQuAD V2, Amazon, 20News)
77.11Average ScoreShadowPEFT w/ pretrained shadow
Evaluation Results
| Method | Links | |
|---|---|---|
| ShadowPEFT w/ pretrained shadowTrainable Params=454,991,872, Backbone Model=Qwen3 8B + 0.5B Shadow, Inference Mode=Shadow-attached, Shadow Model Initialization=pretrained2026.04 | 77.11 | |
| ShadowPEFTTrainable Params=29,118,720, Backbone Model=Qwen3 8B, Inference Mode=Shadow-attached2026.04 | 76.92 | |
| ShadowPEFT w/ random shadowTrainable Params=454,991,872, Backbone Model=Qwen3 8B + 0.5B Shadow, Inference Mode=Shadow-attached, Shadow Model Initialization=random2026.04 | 76.6 | |
| LoRATrainable Params=30,670,848, Backbone Model=Qwen3 8B2026.04 | 76.51 | |
| DoRATrainable Params=31,039,488, Backbone Model=Qwen3 8B2026.04 | 75.99 | |
| ShadowPEFTTrainable Params=23,442,176, Backbone Model=Qwen3 4B, Inference Mode=Shadow-attached2026.04 | 75.43 | |
| DoRATrainable Params=23,906,304, Backbone Model=Qwen3 4B2026.04 | 74.85 | |
| LoRATrainable Params=23,592,960, Backbone Model=Qwen3 4B2026.04 | 74.55 | |
| ShadowPEFTTrainable Params=9,073,920, Backbone Model=Qwen3 0.6B, Inference Mode=Shadow-attached2026.04 | 62.27 | |
| Detached Shadow Only (pretrained)Trainable Params=454,991,872, Backbone Model=Qwen3 8B + 0.5B Shadow, Inference Mode=Detached, Shadow Model Initialization=pretrained2026.04 | 62.11 | |
| DoRATrainable Params=9,318,400, Backbone Model=Qwen3 0.6B2026.04 | 62.08 | |
| LoRATrainable Params=9,175,040, Backbone Model=Qwen3 0.6B2026.04 | 61.81 | |
| Detached Shadow OnlyTrainable Params=23,442,176, Backbone Model=Qwen3 4B, Inference Mode=Detached2026.04 | 38.4 | |
| Detached Shadow OnlyTrainable Params=9,073,920, Backbone Model=Qwen3 0.6B, Inference Mode=Detached2026.04 | 36.79 | |
| Detached Shadow OnlyTrainable Params=29,118,720, Backbone Model=Qwen3 8B, Inference Mode=Detached2026.04 | 36.09 | |
| Detached Shadow Only (random)Trainable Params=454,991,872, Backbone Model=Qwen3 8B + 0.5B Shadow, Inference Mode=Detached, Shadow Model Initialization=random2026.04 | 35.11 |