Zero-shot Commonsense Reasoning Suite (PIQA, HellaSwag, WinoGrande, ARC, SocialIQA, BoolQ)
74.15PIQA AccuracyDNH-HOPE
Evaluation Results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| DNH-HOPEModel Parameters=1.3B, Training Tokens=100B, Evaluation Protocol=Zero-shot2025.11 | 74.15 | 57.46 | 60.72 | 73.51 | 41.96 | 43.18 | 62.05 | 57.84 | |
| Titans (LMM)Model Parameters=1.3B, Training Tokens=100B, Evaluation Protocol=Zero-shot2025.11 | 73.09 | 56.31 | 59.81 | 72.43 | 40.82 | 42.05 | 60.97 | 56.82 | |
| DeltaNetModel Parameters=1.3B, Training Tokens=100B, Evaluation Protocol=Zero-shot2025.11 | 70.72 | 50.93 | 53.35 | 68.47 | 35.66 | 40.22 | 55.29 | 52.14 | |
| DNH-HOPEModel Parameters=760M, Training Tokens=30B, Evaluation Protocol=Zero-shot2025.11 | 70.45 | 49.72 | 53.14 | 67.28 | 36.91 | 41.05 | 63.84 | 52.71 | |
| HOPEModel Parameters=1.3B, Training Tokens=100B, Evaluation Protocol=Zero-shot2025.11 | 70.13 | 49.21 | 52.7 | 66.89 | 36.05 | 40.71 | 63.29 | 52.26 | |
| RetNetModel Parameters=1.3B, Training Tokens=100B, Evaluation Protocol=Zero-shot2025.11 | 70.07 | 49.16 | 54.14 | 67.34 | 33.78 | 40.78 | 60.39 | 52.02 | |
| Transformer++Model Parameters=1.3B, Training Tokens=100B, Evaluation Protocol=Zero-shot2025.11 | 70.02 | 50.23 | 53.51 | 68.83 | 35.1 | 40.66 | 57.09 | 52.25 | |
| Titans (LMM)Model Parameters=760M, Training Tokens=30B, Evaluation Protocol=Zero-shot2025.11 | 69.28 | 48.46 | 52.27 | 66.31 | 35.84 | 40.13 | 62.76 | 51.56 | |
| RetNetModel Parameters=760M, Training Tokens=30B, Evaluation Protocol=Zero-shot2025.11 | 67.19 | 41.63 | 52.09 | 63.17 | 32.78 | 38.36 | 57.92 | 48.46 | |
| DeltaNetModel Parameters=760M, Training Tokens=30B, Evaluation Protocol=Zero-shot2025.11 | 66.93 | 41.98 | 50.65 | 64.87 | 31.39 | 39.88 | 59.02 | 48.97 | |
| Transformer++Model Parameters=760M, Training Tokens=30B, Evaluation Protocol=Zero-shot2025.11 | 66.92 | 42.19 | 51.95 | 60.38 | 32.46 | 39.51 | 60.37 | 48.69 | |
| HOPEModel Parameters=760M, Training Tokens=30B, Evaluation Protocol=Zero-shot2025.11 | 64.62 | 40.11 | 51.19 | 56.92 | 28.49 | 38.33 | 60.12 | 46.9 |