Multiple Choice Question Answering on MMLU, ARC-Easy, HellaSwag, PIQA, OpenBookQA, WinoGrande
24.3MMLU AccuracyBaseline
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| BaselineDepth=6, Hidden width=7682026.05 | 24.3 | 33.1 | 27.6 | 58.9 | 25.2 | 52.4 | |
| 4-viumDepth=24, Hidden width=768, beta=0.152026.05 | 23.1 | 37 | 32.3 | 63.9 | 28.4 | 53.5 | |
| 4-viumDepth=6, Hidden width=768, beta=0.152026.05 | 23 | 32.8 | 28.5 | 59.1 | 23.2 | 50.7 | |
| BaselineDepth=48, Hidden width=7682026.05 | 23 | 37.4 | 35.3 | 65.3 | 28.8 | 50.1 | |
| BaselineDepth=12, Hidden width=7682026.05 | 22.9 | 35.1 | 28.8 | 61.8 | 24.4 | 51.9 | |
| 4-viumDepth=12, Hidden width=768, beta=0.152026.05 | 22.9 | 33.5 | 29.2 | 61.4 | 28 | 48.2 | |
| BaselineDepth=24, Hidden width=7682026.05 | 22.9 | 35.8 | 31.3 | 63.9 | 27.8 | 50.6 | |
| BaselineDepth=36, Hidden width=7682026.05 | 22.9 | 38.3 | 33.8 | 63.3 | 28.2 | 51.4 | |
| 4-viumDepth=36, Hidden width=768, beta=0.152026.05 | 22.9 | 39.3 | 35.5 | 64.9 | 28.4 | 51.7 | |
| 4-viumDepth=48, Hidden width=768, beta=0.152026.05 | 22.9 | 39.7 | 37 | 66.6 | 29.6 | 53.7 |