In-context Learning on Fineweb-Edu 16.8B tokens
36.86ARC-c AccuracySpectra-AdEMAMix
Evaluation Results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Spectra-AdEMAMixModel size=820M, Protocol=5-shot, Pre-training tokens=16.8B, Harness=lm-evaluation-harness2026.03 | 36.86 | 68.81 | 53.94 | 69.48 | 47.09 | 53.43 | 36.4 | 28.26 | 89.4 | 26.23 | 50.99 | |
| D-MuonModel size=820M, Protocol=5-shot, Pre-training tokens=16.8B, Harness=lm-evaluation-harness2026.03 | 36.6 | 67.47 | 53.73 | 69.31 | 46.69 | 52.72 | 34.6 | 28.84 | 90.9 | 25.08 | 50.59 | |
| Spectra-AdamWModel size=820M, Protocol=5-shot, Pre-training tokens=16.8B, Harness=lm-evaluation-harness2026.03 | 35.67 | 66.96 | 55.41 | 68.82 | 46.57 | 52.09 | 36.4 | 26.74 | 89.2 | 25.14 | 50.3 | |
| Spectra-SignumModel size=820M, Protocol=5-shot, Pre-training tokens=16.8B, Harness=lm-evaluation-harness2026.03 | 35.58 | 68.81 | 58.1 | 70.46 | 47.04 | 54.46 | 35.6 | 29.36 | 90.3 | 26.44 | 51.62 | |
| AdEMAMixModel size=820M, Protocol=5-shot, Pre-training tokens=16.8B, Harness=lm-evaluation-harness2026.03 | 35.41 | 65.95 | 43.43 | 70.08 | 46.43 | 53.2 | 34.4 | 28.51 | 88.3 | 24.84 | 49.05 | |
| MarsModel size=820M, Protocol=5-shot, Pre-training tokens=16.8B, Harness=lm-evaluation-harness2026.03 | 34.98 | 67.8 | 57.25 | 70.08 | 46.87 | 52.17 | 34.2 | 29.38 | 89.6 | 25.37 | 50.77 | |
| SignumModel size=820M, Protocol=5-shot, Pre-training tokens=16.8B, Harness=lm-evaluation-harness2026.03 | 34.56 | 65.66 | 56.02 | 69.64 | 45.01 | 52.25 | 34.2 | 28.04 | 87.3 | 25.75 | 49.84 | |
| AdamWModel size=820M, Protocol=5-shot, Pre-training tokens=16.8B, Harness=lm-evaluation-harness2026.03 | 33.45 | 66.84 | 56.09 | 69.21 | 45.65 | 50.99 | 35.2 | 25.54 | 90.8 | 23.86 | 49.76 |