Commonsense Reasoning (LM Evaluation Harness) Zero-Shot
11.86LAMBADA PerplexityPKDA
Evaluation results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PKDAParameters=1B, Training Tokens=50B, Batch size tokens=0.5M, Training Dataset=SlimPajama2026.04 | 11.86 | 17.82 | 57.24 | 26.96 | 49.05 | 48.52 | 70.62 | 55.49 | 58.8 | 86.4 | 56.64 | |
| PGDNParameters=1B, Training Tokens=50B, Batch size tokens=0.5M, Training Dataset=SlimPajama2026.04 | 13.09 | 18.01 | 56.66 | 26.96 | 49.66 | 46.49 | 70.73 | 54.54 | 59.9 | 86.1 | 56.38 | |
| GDNParameters=1B, Training Tokens=50B, Batch size tokens=0.5M, Training Dataset=SlimPajama2026.04 | 13.46 | 18.05 | 56.1 | 27.3 | 48.91 | 46.42 | 70.08 | 54.22 | 55.8 | 84.7 | 55.44 | |
| KDAParameters=1B, Training Tokens=50B, Batch size tokens=0.5M, Training Dataset=SlimPajama2026.04 | 14.52 | 17.95 | 54.55 | 26.88 | 48.95 | 45.16 | 70.29 | 54.22 | 57.6 | 85.8 | 55.43 | |
| PKDAParameters=355M, Training Tokens=15B, Batch size tokens=0.5M, Training Dataset=SlimPajama2026.04 | 25.33 | 25.98 | 46.97 | 22.95 | 37.01 | 37.22 | 65.68 | 51.46 | 59.1 | 78.7 | 49.89 | |
| PGDNParameters=340M, Training Tokens=15B, Batch size tokens=0.5M, Training Dataset=SlimPajama2026.04 | 28.43 | 26.86 | 47.81 | 24.21 | 36.19 | 35.55 | 64.83 | 52.97 | 60.1 | 78 | 49.96 | |
| KDAParameters=355M, Training Tokens=15B, Batch size tokens=0.5M, Training Dataset=SlimPajama2026.04 | 31.37 | 26.18 | 45.45 | 22.7 | 36.06 | 34.04 | 66 | 52.25 | 57 | 79.5 | 49.12 | |
| GDNParameters=340M, Training Tokens=15B, Batch size tokens=0.5M, Training Dataset=SlimPajama2026.04 | 31.39 | 27.08 | 44.49 | 24.32 | 35.96 | 34.5 | 65.83 | 51.3 | 56.3 | 78.2 | 48.86 | |
| PDNParameters=340M, Training Tokens=15B, Batch size tokens=0.5M, Training Dataset=SlimPajama2026.04 | 43.05 | 29.05 | 44.7 | 23.52 | 33.83 | 30.48 | 65.14 | 52.41 | 58.5 | 75.1 | 47.96 | |
| DNParameters=340M, Training Tokens=15B, Batch size tokens=0.5M, Training Dataset=SlimPajama2026.04 | 45.76 | 29.04 | 44.02 | 23.55 | 34.08 | 29.58 | 65.07 | 50.91 | 56.8 | 75.3 | 47.41 |