Zero-shot Learning on Reasoning Suite (ARC-e, ARC-c, WG, BQ, PIQA, HS, OBQA, HQA)
49.7ARC-e AccuracyOmniQuant
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| OmniQuantParams (B)=1.3, Quantization=W4A42026.04 | 49.7 | 25.5 | 58.1 | 58.3 | 66 | 41.7 | 26.2 | 25.7 | 43.9 | |
| TriLMParams (B)=1.5, Precision=1.58-bit2026.04 | 49.1 | 25.2 | 57.3 | 57.2 | 64.5 | 41.1 | 25.6 | 25.3 | 43.16 | |
| BitNetParams (B)=1.3, Precision=1.58-bit2026.04 | 48.7 | 24.1 | 56.8 | 57.4 | 64.2 | 40.6 | 24.6 | 24.9 | 42.66 | |
| BiSpikCLM-LlamaParams (B)=1.2, Time steps (T)=42026.04 | 46.5 | 24.4 | 55.9 | 56.6 | 63.6 | 41.8 | 25.4 | 24.4 | 42.33 | |
| BiSpikCLMParams (B)=1.3, Time steps (T)=42026.04 | 46.3 | 24.3 | 55.6 | 56.8 | 63.4 | 41.7 | 25.2 | 24.3 | 42.19 | |
| TriLMParams (B)=1.1, Precision=1.58-bit2026.04 | 46.2 | 23.9 | 55.5 | 56.5 | 64.1 | 39.2 | 24.7 | 24.5 | 41.83 | |
| BiSpikCLM-LlamaParams (B)=1.2, Time steps (T)=22026.04 | 45.8 | 23.9 | 54.7 | 56.1 | 63 | 40.6 | 24.7 | 24.1 | 41.61 | |
| BiSpikCLMParams (B)=1.3, Time steps (T)=22026.04 | 45.7 | 23.5 | 54.2 | 56.3 | 62.3 | 40.2 | 24.5 | 24 | 41.33 | |
| SmoothQuantParams (B)=1.3, Quantization=W4A42026.04 | 44.3 | 23.5 | 54.2 | 55.6 | 62.3 | 40 | 23.9 | 23.2 | 40.88 |