Language Modeling on MMLU (Final Performance & Critical Bits)
76.82MMLU Final PerformanceShadowPEFT w/ random shadow
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| ShadowPEFT w/ random shadowTrainable Params=454,991,872, Backbone Model=Qwen3 8B + 0.5B Shadow, Inference Mode=Shadow-attached, Shadow Model Initialization=random2026.04 | 76.82 | — | |
| ShadowPEFT w/ pretrained shadowTrainable Params=454,991,872, Backbone Model=Qwen3 8B + 0.5B Shadow, Inference Mode=Shadow-attached, Shadow Model Initialization=pretrained2026.04 | 76.54 | — | |
| ShadowPEFTTrainable Params=29,118,720, Backbone Model=Qwen3 8B, Inference Mode=Shadow-attached2026.04 | 76.51 | — | |
| LoRATrainable Params=30,670,848, Backbone Model=Qwen3 8B2026.04 | 76.46 | — | |
| DoRATrainable Params=31,039,488, Backbone Model=Qwen3 8B2026.04 | 75.79 | — | |
| ShadowPEFTTrainable Params=23,442,176, Backbone Model=Qwen3 4B, Inference Mode=Shadow-attached2026.04 | 72.91 | — | |
| DoRATrainable Params=23,906,304, Backbone Model=Qwen3 4B2026.04 | 72.56 | — | |
| LoRATrainable Params=23,592,960, Backbone Model=Qwen3 4B2026.04 | 72.37 | — | |
| Qwen3 8BTrainable Params=0, Backbone Model=Qwen3 8B2026.04 | 71.92 | — | |
| Random FlipsTarget Model=DeepSeek-V2, Parameters=7B (MoE)2025.12 | 69 | 10,000 | |
| Donor2026.03 | 68.32 | — | |
| CNTTransfer Rate (TR)=0.0232026.03 | 68.25 | — | |
| GradientTransfer Rate (TR)=0.0232026.03 | 67.76 | — | |
| RandomTransfer Rate (TR)=0.0232026.03 | 67.67 | — | |
| Qwen3 4BTrainable Params=0, Backbone Model=Qwen3 4B2026.04 | 67.54 | — | |
| Random FlipsTarget Model=LLaMA 3.1 8B, Parameters=8B2025.12 | 67.5 | 10,000 | |
| Biased Model2026.03 | 67.5 | — | |
| ShadowPEFTTrainable Params=9,073,920, Backbone Model=Qwen3 0.6B, Inference Mode=Shadow-attached2026.04 | 50.63 | — | |
| DoRATrainable Params=9,318,400, Backbone Model=Qwen3 0.6B2026.04 | 50.34 | — | |
| Detached Shadow Only (pretrained)Trainable Params=454,991,872, Backbone Model=Qwen3 8B + 0.5B Shadow, Inference Mode=Detached, Shadow Model Initialization=pretrained2026.04 | 50.03 | — | |
| LoRATrainable Params=9,175,040, Backbone Model=Qwen3 0.6B2026.04 | 49.91 | — | |
| Base modelRuntime (min)=N/A2026.04 | 46.5 | — | |
| MuonRuntime (min)=12.02026.04 | 46.2 | — | |
| SIFTRuntime (min)=26.42026.04 | 46.2 | — | |
| AdamWRuntime (min)=8.12026.04 | 46.1 | — | |
| POMERuntime (min)=10.62026.04 | 46 | — | |
| BLURRuntime (min)=8.62026.04 | 46 | — | |
| Qwen3 0.6BTrainable Params=0, Backbone Model=Qwen3 0.6B2026.04 | 37.34 | — | |
| Random FlipsTarget Model=GPT-2 Large, Parameters=774M2025.12 | 29.5 | 10,000 | |
| Detached Shadow OnlyTrainable Params=23,442,176, Backbone Model=Qwen3 4B, Inference Mode=Detached2026.04 | 27.01 | — | |
| Detached Shadow Only (random)Trainable Params=454,991,872, Backbone Model=Qwen3 8B + 0.5B Shadow, Inference Mode=Detached, Shadow Model Initialization=random2026.04 | 25.35 | — | |
| Detached Shadow OnlyTrainable Params=9,073,920, Backbone Model=Qwen3 0.6B, Inference Mode=Detached2026.04 | 24.62 | — | |
| Detached Shadow OnlyTrainable Params=29,118,720, Backbone Model=Qwen3 8B, Inference Mode=Detached2026.04 | 24.03 | — | |
| Gradient-BasedTarget Model=GPT-2 Large, Parameters=774M2025.12 | 1.5 | 670 | |
| Gradient-BasedTarget Model=DeepSeek-V2, Parameters=7B (MoE)2025.12 | 1.1 | 920 | |
| Gradient-BasedTarget Model=LLaMA 3.1 8B, Parameters=8B2025.12 | 0.95 | 850 | |
| GenBFATarget Model=GPT-2 Large, Parameters=774M2025.12 | 0.38 | 5 | |
| FlipLLMTarget Model=GPT-2 Large, Parameters=774M2025.12 | 0.35 | 5 | |
| GenBFATarget Model=LLaMA 3.1 8B, Parameters=8B2025.12 | 0.2 | 5 | |
| FlipLLMTarget Model=LLaMA 3.1 8B, Parameters=8B2025.12 | 0.18 | 5 | |
| GenBFATarget Model=DeepSeek-V2, Parameters=7B (MoE)2025.12 | 0.17 | 6 | |
| FlipLLMTarget Model=DeepSeek-V2, Parameters=7B (MoE)2025.12 | 0.15 | 6 |