Zero-shot Evaluation on AI2 OLMES zero-shot
35.96ScienceQAGEM
Evaluation results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GEMData Mixing Strategy=Perf, Evaluation Protocol=zero-shot, Model Architecture=LLaMA-style Transformer, Parameters=1.1B, Training Tokens=25B2026.04 | 35.96 | — | — | — | — | — | — | — | — | — | — | 40.43 | 57.98 | 44.79 | |
| WebOrganizer FormatData Mixing Strategy=Perf, Evaluation Protocol=zero-shot, Model Architecture=LLaMA-style Transformer, Parameters=1.1B, Training Tokens=25B2026.04 | 35.06 | — | — | — | — | — | — | — | — | — | — | 39.73 | 57.97 | 44.25 | |
| WebOrganizer TopicData Mixing Strategy=Perf, Evaluation Protocol=zero-shot, Model Architecture=LLaMA-style Transformer, Parameters=1.1B, Training Tokens=25B2026.04 | 35.05 | — | — | — | — | — | — | — | — | — | — | 39.73 | 57.9 | 44.23 | |
| GEMData Mixing Strategy=DoReMi, Evaluation Protocol=zero-shot, Model Architecture=LLaMA-style Transformer, Parameters=1.1B, Training Tokens=25B2026.04 | 34.79 | — | — | — | — | — | — | — | — | — | — | 39.96 | 57.11 | 43.95 | |
| WebOrganizer TopicData Mixing Strategy=DoReMi, Evaluation Protocol=zero-shot, Model Architecture=LLaMA-style Transformer, Parameters=1.1B, Training Tokens=25B2026.04 | 34.68 | — | — | — | — | — | — | — | — | — | — | 38.26 | 55.35 | 42.76 | |
| WebOrganizer FormatData Mixing Strategy=DoReMi, Evaluation Protocol=zero-shot, Model Architecture=LLaMA-style Transformer, Parameters=1.1B, Training Tokens=25B2026.04 | 34.44 | — | — | — | — | — | — | — | — | — | — | 38.73 | 55.19 | 42.79 | |
| WebOrganizer FormatData Mixing Strategy=RegMix, Evaluation Protocol=zero-shot, Model Architecture=LLaMA-style Transformer, Parameters=1.1B, Training Tokens=25B2026.04 | 34.12 | — | — | — | — | — | — | — | — | — | — | 33.94 | 54.26 | 40.77 | |
| GEMData Mixing Strategy=RegMix, Evaluation Protocol=zero-shot, Model Architecture=LLaMA-style Transformer, Parameters=1.1B, Training Tokens=25B2026.04 | 34.07 | — | — | — | — | — | — | — | — | — | — | 35.3 | 54.97 | 41.45 | |
| WebOrganizer TopicData Mixing Strategy=RegMix, Evaluation Protocol=zero-shot, Model Architecture=LLaMA-style Transformer, Parameters=1.1B, Training Tokens=25B2026.04 | 33.9 | — | — | — | — | — | — | — | — | — | — | 33.83 | 52.5 | 40.08 | |
| TOSData Mixing Strategy=Perf, Evaluation Protocol=zero-shot, Model Architecture=LLaMA-style Transformer, Parameters=1.1B, Training Tokens=25B2026.04 | 32.54 | — | — | — | — | — | — | — | — | — | — | 34.51 | 53.29 | 40.11 | |
| TOSData Mixing Strategy=DoReMi, Evaluation Protocol=zero-shot, Model Architecture=LLaMA-style Transformer, Parameters=1.1B, Training Tokens=25B2026.04 | 32.49 | — | — | — | — | — | — | — | — | — | — | 34.29 | 53.98 | 40.25 | |
| K-MeansData Mixing Strategy=Perf, Evaluation Protocol=zero-shot, Model Architecture=LLaMA-style Transformer, Parameters=1.1B, Training Tokens=25B2026.04 | 32.48 | — | — | — | — | — | — | — | — | — | — | 35.11 | 55.52 | 41.04 | |
| K-MeansData Mixing Strategy=DoReMi, Evaluation Protocol=zero-shot, Model Architecture=LLaMA-style Transformer, Parameters=1.1B, Training Tokens=25B2026.04 | 32.18 | — | — | — | — | — | — | — | — | — | — | 34.21 | 53.43 | 39.94 | |
| TOSData Mixing Strategy=RegMix, Evaluation Protocol=zero-shot, Model Architecture=LLaMA-style Transformer, Parameters=1.1B, Training Tokens=25B2026.04 | 32.18 | — | — | — | — | — | — | — | — | — | — | 34.35 | 52.4 | 39.64 | |
| K-MeansData Mixing Strategy=RegMix, Evaluation Protocol=zero-shot, Model Architecture=LLaMA-style Transformer, Parameters=1.1B, Training Tokens=25B2026.04 | 31.63 | — | — | — | — | — | — | — | — | — | — | 34.86 | 55.67 | 40.72 | |
| BaseParams=246M, Compute Budget (FLOPs)=10^18, Evaluation Protocol=Zero-shot2026.05 | — | 39.3 | 25.3 | 51.4 | 29.3 | 26.2 | 27.6 | 57.8 | 40.5 | 50.5 | 38.7 | — | — | — | |
| LoopedParams=168M, Compute Budget (FLOPs)=10^18, Evaluation Protocol=Zero-shot2026.05 | — | 39.1 | 24.6 | 49.6 | 27.1 | 25.6 | 24.4 | 57.1 | 38.2 | 51.2 | 37.4 | — | — | — | |
| Looped-MoEParams=216M, Compute Budget (FLOPs)=10^18, Evaluation Protocol=Zero-shot2026.05 | — | 40.4 | 24.3 | 63.9 | 30.9 | 25.4 | 26.8 | 55.2 | 38 | 51.7 | 39.6 | — | — | — |