MCQA Suite Subset (HellaSwag, PIQA, ARC-E, ARC-C) on Test Set
38.8HellaSwag AccuracyUncompressed
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| UncompressedSize=1B, P=4, Evaluation Mode=Zero-shot2026.06 | 38.8 | 66.3 | 46.8 | 24.5 | 44.1 | |
| UncompressedSize=1B, P=8, Evaluation Mode=Zero-shot2026.06 | 38.8 | 66.3 | 46.8 | 24.5 | 44.1 | |
| MAPLSize=1B, P=4, Evaluation Mode=Zero-shot2026.06 | 37.1 | 65.7 | 45.7 | 24.5 | 43.3 | |
| MAPLSize=1B, P=8, Evaluation Mode=Zero-shot2026.06 | 36.2 | 64.8 | 45.2 | 24.1 | 42.6 | |
| UncompressedSize=500M, P=4, Evaluation Mode=Zero-shot2026.06 | 35.7 | 64.4 | 43.2 | 24.8 | 42 | |
| UncompressedSize=500M, P=8, Evaluation Mode=Zero-shot2026.06 | 35.7 | 64.4 | 43.2 | 24.8 | 42 | |
| MAPLSize=500M, P=4, Evaluation Mode=Zero-shot2026.06 | 35.1 | 64 | 43.6 | 24.3 | 41.8 | |
| MAPLSize=500M, P=8, Evaluation Mode=Zero-shot2026.06 | 34.5 | 64.7 | 42.4 | 24.7 | 41.6 | |
| SSNSize=500M, P=4, Evaluation Mode=Zero-shot2026.06 | 29.1 | 59.8 | 38.5 | 22.3 | 37.4 | |
| SSNSize=1B, P=4, Evaluation Mode=Zero-shot2026.06 | 29.1 | 61.4 | 37.1 | 22.7 | 37.6 | |
| SSNSize=1B, P=8, Evaluation Mode=Zero-shot2026.06 | 28.7 | 60.6 | 37.2 | 21.8 | 37.1 | |
| SSNSize=500M, P=8, Evaluation Mode=Zero-shot2026.06 | 28.5 | 59.1 | 37.2 | 22.2 | 36.7 | |
| MAPL + VQSize=150M, P=4, Evaluation Mode=Zero-shot2026.06 | 28.4 | 52.8 | 31.7 | 22.7 | 33.9 | |
| UncompressedSize=150M, P=4, Evaluation Mode=Zero-shot2026.06 | 28.1 | 60.1 | 38.2 | 22.8 | 37.3 | |
| UncompressedSize=150M, P=8, Evaluation Mode=Zero-shot2026.06 | 28.1 | 60.1 | 38.2 | 22.8 | 37.3 | |
| MAPLSize=150M, P=4, Evaluation Mode=Zero-shot2026.06 | 28 | 59.4 | 37 | 22.6 | 36.7 | |
| MAPLSize=150M, P=8, Evaluation Mode=Zero-shot2026.06 | 27.8 | 59.3 | 37.8 | 21.5 | 36.6 | |
| SSNSize=150M, P=4, Evaluation Mode=Zero-shot2026.06 | 27.6 | 56.7 | 34.4 | 20.9 | 34.9 | |
| SSN (AdamW version)Size=1B, P=4, Evaluation Mode=Zero-shot2026.06 | 27.5 | 57.9 | 35.1 | 20.8 | 35.3 | |
| SSN (AdamW version)Size=500M, P=8, Evaluation Mode=Zero-shot2026.06 | 27.4 | 57.9 | 34.6 | 22.1 | 35.5 | |
| SSN (AdamW version)Size=1B, P=8, Evaluation Mode=Zero-shot2026.06 | 27.4 | 57.6 | 35.6 | 20.7 | 35.3 | |
| SSNSize=150M, P=8, Evaluation Mode=Zero-shot2026.06 | 27 | 56.3 | 33.8 | 21.7 | 34.7 | |
| SSN (AdamW version)Size=500M, P=4, Evaluation Mode=Zero-shot2026.06 | 27 | 59.5 | 35.6 | 21.2 | 35.8 | |
| MAPL + VQSize=500M, P=8, Evaluation Mode=Zero-shot2026.06 | 26.9 | 51.2 | 27.2 | 24.3 | 32.4 | |
| SSN (AdamW version)Size=150M, P=8, Evaluation Mode=Zero-shot2026.06 | 26.8 | 56 | 32.1 | 20.6 | 33.9 | |
| SSN (AdamW version)Size=150M, P=4, Evaluation Mode=Zero-shot2026.06 | 26.5 | 56.1 | 32.5 | 20.6 | 33.9 | |
| MAPL + VQSize=150M, P=8, Evaluation Mode=Zero-shot2026.06 | 26.4 | 50.5 | 29.1 | 25.9 | 33 | |
| MAPL + VQSize=500M, P=4, Evaluation Mode=Zero-shot2026.06 | 25.9 | 50.8 | 26.6 | 26.7 | 32.5 | |
| MAPL + VQSize=1B, P=4, Evaluation Mode=Zero-shot2026.06 | 25.9 | 49.6 | 26.6 | 27.6 | 32.4 | |
| MAPL + VQSize=1B, P=8, Evaluation Mode=Zero-shot2026.06 | 25.8 | 49.8 | 27.1 | 28 | 32.7 |