General NLP Evaluation on The Pile Downstream Evaluation Suite
29.7HellaSwag AccuracyDOGE
Evaluation Results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| DOGEBackbone=LLaMA-3.2-3B, Training Dataset=The Pile2026.04 | 29.7 | 56.9 | 29.2 | 64 | 25.5 | 50.6 | 61.7 | 40.6 | 11.9 | 41.1 | |
| UniformBackbone=LLaMA-3.2-3B, Training Dataset=The Pile2026.04 | 29.6 | 58.8 | 29.4 | 66 | 25.9 | 51.1 | 61 | 39.1 | 12.6 | 41.5 | |
| DoReMiBackbone=LLaMA-3.2-3B, Training Dataset=The Pile2026.04 | 29.6 | 58.4 | 29.8 | 66 | 24.9 | 51.4 | 61.1 | 40.5 | 12.8 | 41.6 | |
| DoGraphBackbone=LLaMA-3.2-3B, Training Dataset=The Pile2026.04 | 29.6 | 60.5 | 29 | 67 | 29.7 | 51.4 | 65.2 | 40.2 | 15.1 | 43.1 | |
| RegMixBackbone=LLaMA-3.2-3B, Training Dataset=The Pile2026.04 | 29.4 | 59.5 | 29.4 | 66.5 | 25.1 | 53.6 | 62.5 | 41.2 | 12.3 | 42.2 | |
| Dynamic Loss-BasedBackbone=LLaMA-3.2-3B, Training Dataset=The Pile2026.04 | 29.3 | 58.1 | 29.6 | 66.1 | 25 | 52.5 | 62.7 | 39.9 | 12.2 | 41.7 | |
| Data Mixing LawBackbone=LLaMA-3.2-3B, Training Dataset=The Pile2026.04 | 29.3 | 58.4 | 30.2 | 65.9 | 25.6 | 51.3 | 61.3 | 40.2 | 12.1 | 41.6 |