List your company
ResearchFormerly WizWand

Language Modeling on MMLU (Final Performance & Critical Bits)

76.82MMLU Final PerformanceShadowPEFT w/ random shadow
-2.916817.784138.48559.1859Dec 10, 2025Jan 1, 2026Jan 23, 2026Feb 14, 2026Mar 8, 2026Mar 30, 2026Apr 21, 2026
Updated 6mo ago

Evaluation results

MethodLinks
ShadowPEFT w/ random shadowTrainable Params=454,991,872, Backbone Model=Qwen3 8B + 0.5B Shadow, Inference Mode=Shadow-attached, Shadow Model Initialization=random2026.0476.82—
ShadowPEFT w/ pretrained shadowTrainable Params=454,991,872, Backbone Model=Qwen3 8B + 0.5B Shadow, Inference Mode=Shadow-attached, Shadow Model Initialization=pretrained2026.0476.54—
ShadowPEFTTrainable Params=29,118,720, Backbone Model=Qwen3 8B, Inference Mode=Shadow-attached2026.0476.51—
LoRATrainable Params=30,670,848, Backbone Model=Qwen3 8B2026.0476.46—
DoRATrainable Params=31,039,488, Backbone Model=Qwen3 8B2026.0475.79—
ShadowPEFTTrainable Params=23,442,176, Backbone Model=Qwen3 4B, Inference Mode=Shadow-attached2026.0472.91—
DoRATrainable Params=23,906,304, Backbone Model=Qwen3 4B2026.0472.56—
LoRATrainable Params=23,592,960, Backbone Model=Qwen3 4B2026.0472.37—
Qwen3 8BTrainable Params=0, Backbone Model=Qwen3 8B2026.0471.92—
Random FlipsTarget Model=DeepSeek-V2, Parameters=7B (MoE)2025.126910,000
Donor2026.0368.32—
CNTTransfer Rate (TR)=0.0232026.0368.25—
GradientTransfer Rate (TR)=0.0232026.0367.76—
RandomTransfer Rate (TR)=0.0232026.0367.67—
Qwen3 4BTrainable Params=0, Backbone Model=Qwen3 4B2026.0467.54—
Random FlipsTarget Model=LLaMA 3.1 8B, Parameters=8B2025.1267.510,000
Biased Model2026.0367.5—
ShadowPEFTTrainable Params=9,073,920, Backbone Model=Qwen3 0.6B, Inference Mode=Shadow-attached2026.0450.63—
DoRATrainable Params=9,318,400, Backbone Model=Qwen3 0.6B2026.0450.34—
Detached Shadow Only (pretrained)Trainable Params=454,991,872, Backbone Model=Qwen3 8B + 0.5B Shadow, Inference Mode=Detached, Shadow Model Initialization=pretrained2026.0450.03—
LoRATrainable Params=9,175,040, Backbone Model=Qwen3 0.6B2026.0449.91—
Base modelRuntime (min)=N/A2026.0446.5—
MuonRuntime (min)=12.02026.0446.2—
SIFTRuntime (min)=26.42026.0446.2—
AdamWRuntime (min)=8.12026.0446.1—
POMERuntime (min)=10.62026.0446—
BLURRuntime (min)=8.62026.0446—
Qwen3 0.6BTrainable Params=0, Backbone Model=Qwen3 0.6B2026.0437.34—
Random FlipsTarget Model=GPT-2 Large, Parameters=774M2025.1229.510,000
Detached Shadow OnlyTrainable Params=23,442,176, Backbone Model=Qwen3 4B, Inference Mode=Detached2026.0427.01—
Detached Shadow Only (random)Trainable Params=454,991,872, Backbone Model=Qwen3 8B + 0.5B Shadow, Inference Mode=Detached, Shadow Model Initialization=random2026.0425.35—
Detached Shadow OnlyTrainable Params=9,073,920, Backbone Model=Qwen3 0.6B, Inference Mode=Detached2026.0424.62—
Detached Shadow OnlyTrainable Params=29,118,720, Backbone Model=Qwen3 8B, Inference Mode=Detached2026.0424.03—
Gradient-BasedTarget Model=GPT-2 Large, Parameters=774M2025.121.5670
Gradient-BasedTarget Model=DeepSeek-V2, Parameters=7B (MoE)2025.121.1920
Gradient-BasedTarget Model=LLaMA 3.1 8B, Parameters=8B2025.120.95850
GenBFATarget Model=GPT-2 Large, Parameters=774M2025.120.385
FlipLLMTarget Model=GPT-2 Large, Parameters=774M2025.120.355
GenBFATarget Model=LLaMA 3.1 8B, Parameters=8B2025.120.25
FlipLLMTarget Model=LLaMA 3.1 8B, Parameters=8B2025.120.185
GenBFATarget Model=DeepSeek-V2, Parameters=7B (MoE)2025.120.176
FlipLLMTarget Model=DeepSeek-V2, Parameters=7B (MoE)2025.120.156