Language Modeling on Qwen3-0.6B (val)
31.45Validation PerplexityDelta Block
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Delta BlockL (Layers)=28, N (Routing layers)=28, Steps=10K, Learning Rate (lr)=6×10−42026.05 | 31.45 | 117 | 25.2 | |
| BaselineL (Layers)=28, Steps=10K, Learning Rate (lr)=6×10−42026.05 | 32.22 | 173 | 13.8 | |
| AttnResL (Layers)=28, N (Routing layers)=28, Steps=10K, Learning Rate (lr)=6×10−42026.05 | 32.38 | 111 | 28.4 |