Multiple Choice Evaluation on ARC, HellaSwag, PIQA, and Winogrande (Standard LLM Eval)
56.98Average AccuracyL3 Transformer
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| L3 TransformerActive Parameters / Token=2.6B, L3 Layers=2, Total Parameters=7B, Evaluation Mode=0-Shot2026.01 | 56.98 | 38.21 | 71.21 | 44.95 | 71.71 | 58.8 | |
| Dense TransformerActive Parameters / Token=2.6B, L3 Layers=0, Total Parameters=2.6B, Evaluation Mode=0-Shot2026.01 | 55.59 | 36.35 | 68.86 | 44.2 | 71.22 | 57.3 | |
| L3 TransformerActive Parameters / Token=1.5B, L3 Layers=2, Total Parameters=4.6B, Evaluation Mode=0-Shot2026.01 | 53.84 | 33.11 | 67.63 | 41.93 | 70.67 | 55.88 | |
| L3 TransformerActive Parameters / Token=1.5B, L3 Layers=1, Total Parameters=3.1B, Evaluation Mode=0-Shot2026.01 | 53.09 | 31.91 | 67.34 | 41.04 | 69.37 | 55.81 | |
| Dense TransformerActive Parameters / Token=1.5B, L3 Layers=0, Total Parameters=1.5B, Evaluation Mode=0-Shot2026.01 | 51.93 | 29.18 | 66.93 | 40.23 | 67.96 | 55.37 | |
| L3 TransformerActive Parameters / Token=818M, L3 Layers=3*, Total Parameters=5.2B, Evaluation Mode=0-Shot, Note=Uses a wider L3 layer2026.01 | 50.25 | 29.01 | 63.64 | 36.32 | 68.23 | 54.06 | |
| L3 TransformerActive Parameters / Token=803M, L3 Layers=2, Total Parameters=3.1B, Evaluation Mode=0-Shot2026.01 | 49.45 | 28.67 | 62.92 | 36.09 | 67.83 | 51.72 | |
| L3 TransformerActive Parameters / Token=795M, L3 Layers=1, Total Parameters=1.8B, Evaluation Mode=0-Shot2026.01 | 49.1 | 28.21 | 62.03 | 35.58 | 67.52 | 52.17 | |
| Dense TransformerActive Parameters / Token=809M, L3 Layers=0, Total Parameters=809M, Evaluation Mode=0-Shot2026.01 | 48.28 | 27.56 | 60.52 | 34.32 | 67.13 | 51.85 |