Question Answering on ARC-Challenge (Normalized Accuracy)
46.93Normalized AccuracyCALDERA
Evaluation results
| Method | Links | |
|---|---|---|
| CALDERABackbone=Mistral 7B, Rank=256, Avg Bits=2.382024.05 | 46.93 | |
| CALDERABackbone=Mistral 7B, Rank=128, Avg Bits=2.192024.05 | 46.59 | |
| QuIP#Backbone=Mistral 7B, Rank=0, Avg Bits=22024.05 | 44.8 | |
| SpanNormParam=A2.4B-16B, Tokens=200B, Architecture=MoE2026.01 | 38.6 | |
| PreNormParam=A2.4B-16B, Tokens=200B, Architecture=MoE2026.01 | 37 | |
| SpanNormParam=5B, Tokens=200B, Architecture=Dense2026.01 | 36.3 | |
| PreNormParam=5B, Tokens=200B, Architecture=Dense2026.01 | 34.4 | |
| SpanNormParam=1.3B, Tokens=100B, Architecture=Dense2026.01 | 28.8 | |
| PreNormParam=1.3B, Tokens=100B, Architecture=Dense2026.01 | 26.8 | |
| PreNormParam=740M, Tokens=30B, Architecture=Dense2026.01 | 25.3 | |
| SpanNormParam=740M, Tokens=30B, Architecture=Dense2026.01 | 25.1 | |
| AutoregressiveAlpha (α)=1, Data repetitions=12025.12 | 5.9 | |
| DualAlpha (α)=63/64, Data repetitions=12025.12 | 5.7 | |
| AutoregressiveAlpha (α)=1, Data repetitions=322025.12 | 5 | |
| DualAlpha (α)=3/4, Data repetitions=322025.12 | 3.3 | |
| DualAlpha (α)=1/8, Data repetitions=1282025.12 | 1.7 | |
| AutoregressiveAlpha (α)=1, Data repetitions=1282025.12 | -1 |