Quantitative Reasoning on MMLU STEM (test)
75Accuracy (STEM Test)Minerva
Evaluation results
| Method | Links | |
|---|---|---|
| MinervaModel Parameters=540B, Majority Voting (maj1@k)=true, k samples=16, Chain-of-thought prompt=true2022.06 | 75 | |
| MinervaModel Parameters=540B, Few-shot (5-shot)=true2022.06 | 63.9 | |
| MinervaModel Parameters=62B, Majority Voting (maj1@k)=true, k samples=16, Chain-of-thought prompt=true2022.06 | 63.5 | |
| PaLMModel Parameters=540B, Few-shot (5-shot)=true2022.06 | 58.7 | |
| Chinchilla (Published SOTA)2022.06 | 54.9 | |
| MinervaModel Parameters=62B, Few-shot (5-shot)=true2022.06 | 53.9 | |
| MinervaModel Parameters=8B, Majority Voting (maj1@k)=true, k samples=16, Chain-of-thought prompt=true2022.06 | 43.4 | |
| PaLMModel Parameters=62B, Few-shot (5-shot)=true2022.06 | 39.1 | |
| MinervaModel Parameters=8B, Few-shot (5-shot)=true2022.06 | 35.6 | |
| PaLMModel Parameters=8B, Few-shot (5-shot)=true2022.06 | 22 |