Commonsense Reasoning and Knowledge Question Answering on General Ability Suite (test)
36.4ARC-C AccuracyFEM-SM
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| FEM-SMParameters=1.3B, Tokens=100B, Architecture Components=SMAttn+C,L,T,G2026.02 | 36.4 | 63.6 | 51.9 | 71.3 | 62.4 | 53.4 | 74 | 38.2 | 80.7 | |
| FEM-GLAParameters=1.3B, Tokens=100B, Architecture Components=GLA+C,L,T,G2026.02 | 31 | 56.4 | 48.2 | 70.8 | 60.2 | 52.9 | 74 | 35.8 | 78.2 | |
| FEM-SM(-G)Parameters=340M, Tokens=15B, Architecture Components=SMAttn+C,L,T2026.02 | 29.3 | 53.1 | 41.2 | 66.8 | 59.3 | 51.9 | 71 | 33.8 | 71.6 | |
| FEM-SM(-G,T)Parameters=340M, Tokens=15B, Architecture Components=SMAttn+C,L2026.02 | 29 | 51.8 | 40.7 | 65.7 | 59.5 | 51.1 | 69 | 34.2 | 73.1 | |
| DiffTransParameters=340M, Tokens=15B2026.02 | 28.9 | 53.1 | 40.8 | 66.8 | 60.3 | 53.4 | 69 | 33 | 73.4 | |
| GSAParameters=1.3B, Tokens=100B2026.02 | 28.7 | 52.9 | 51 | 71.2 | 54.1 | 53.6 | 76 | 33 | 77.3 | |
| FEM-SMParameters=340M, Tokens=15B, Architecture Components=SM-Attn+C,L,T,G2026.02 | 28.4 | 54.2 | 40.9 | 67.6 | 60.9 | 52.3 | 73 | 34.2 | 73.5 | |
| HGRN2Parameters=1.3B, Tokens=100B2026.02 | 28.2 | 50.4 | 31.7 | 67.1 | 41.6 | 52.2 | 77 | 32.8 | 37.8 | |
| Transformer (SMAttn)Parameters=1.3B, Tokens=100B2026.02 | 28 | 49.2 | 49.2 | 70.5 | 62.1 | 55.2 | 76 | 31.8 | 76.9 | |
| GLAParameters=1.3B, Tokens=100B2026.02 | 27.7 | 48.2 | 48.8 | 70.2 | 57.4 | 54.1 | 69 | 32.6 | 72.1 | |
| FEM-SM(-G,T,L)Parameters=340M, Tokens=15B, Architecture Components=SMAttn+C2026.02 | 27.7 | 50.7 | 40.3 | 66.4 | 58.3 | 51.5 | 67 | 32 | 72.8 | |
| GatedDeltaNetParameters=340M, Tokens=15B2026.02 | 27.6 | 52.7 | 39.6 | 66.2 | 58.8 | 52.7 | 71 | 33.8 | 73.5 | |
| FEM-SM(-G,T,L,C)Parameters=340M, Tokens=15B, Backbone=SMAttn2026.02 | 27.3 | 50.6 | 39.6 | 65 | 56.9 | 49.9 | 72 | 32.4 | 72.7 | |
| RetNetParameters=1.3B, Tokens=100B2026.02 | 27.1 | 48.9 | 48 | 70.1 | 58.3 | 53.3 | 71 | 32.4 | 73.6 | |
| HGRNParameters=1.3B, Tokens=100B2026.02 | 27.1 | 51.8 | 48.1 | 70.7 | 58.4 | 51.5 | 70 | 32.6 | 69.5 | |
| FEM-GLAParameters=340M, Tokens=15B, Architecture Components=GLA+C,L,T,G2026.02 | 27.1 | 49.3 | 39.7 | 64.4 | 59.2 | 51 | 68 | 33.1 | 68.3 | |
| DeltaNetParameters=340M, Tokens=15B2026.02 | 26.9 | 50.2 | 40.5 | 65.3 | 51.9 | 50.4 | 69 | 31.6 | 71.7 | |
| DeltaNetParameters=1.3B, Tokens=100B2026.02 | 26.6 | 52.2 | 50.2 | 70.4 | 61.1 | 54.1 | 74 | 31.8 | 76.1 | |
| GLAParameters=340M, Tokens=15B2026.02 | 24.7 | 47.8 | 36.6 | 63.7 | 54.7 | 48.9 | 64 | 29.4 | 64.9 |