Zero-shot Reasoning on HellaSwag (200 items subset)
68AccuracyHQMQ s192 r6 + Med3x
Evaluation Results
| Method | Links | |
|---|---|---|
| HQMQ s192 r6 + Med3xbits=5.152026.05 | 68 | |
| HQMQ s96 r6 + Med3xbits=4.912026.05 | 65 | |
| HQMQ s24 r6 + Med3xbits=4.422026.05 | 64 | |
| fp16bits=16.002026.05 | 63 | |
| fp16 baselineCompression=1×, Base Model=TinyLlama-1.1B2026.05 | 49.5 | |
| FIBQUANTb (bits/coord)=3.50, Compression=4.6×, k=2, N=128, Base Model=TinyLlama-1.1B2026.05 | 48 | |
| INTb (bits/coord)=4.00, Compression=4×, Base Model=TinyLlama-1.1B2026.05 | 47.5 | |
| FIBQUANTb (bits/coord)=4.00, Compression=4×, k=2, N=256, Base Model=TinyLlama-1.1B2026.05 | 47.5 | |
| FIBQUANTb (bits/coord)=3.00, Compression=5.3×, k=2, N=64, Base Model=TinyLlama-1.1B2026.05 | 47.5 | |
| TURBOQUANTb (bits/coord)=3.00, Compression=5.3×, Base Model=TinyLlama-1.1B2026.05 | 46.5 | |
| TURBOQUANTb (bits/coord)=4.00, Compression=4×, Base Model=TinyLlama-1.1B2026.05 | 45.5 | |
| FIBQUANTb (bits/coord)=2.50, Compression=6.4×, k=4, N=1024, Base Model=TinyLlama-1.1B2026.05 | 44 | |
| FIBQUANTb (bits/coord)=2.00, Compression=8×, k=4, N=256, Base Model=TinyLlama-1.1B2026.05 | 41.5 | |
| INTb (bits/coord)=3.00, Compression=5.3×, Base Model=TinyLlama-1.1B2026.05 | 37 | |
| TURBOQUANTb (bits/coord)=2.00, Compression=8×, Base Model=TinyLlama-1.1B2026.05 | 33.5 | |
| naive int4bits=4.002026.05 | 30 | |
| INTb (bits/coord)=2.00, Compression=8×, Base Model=TinyLlama-1.1B2026.05 | 22.5 |