Zero-shot Classification on 5 zero-shot tasks
79.48AccuracyFP16 (Ref)
Evaluation results
| Method | Links | |
|---|---|---|
| FP16 (Ref)BITS=16.02026.02 | 79.48 | |
| FP16Bits=16, Model Architecture=LLAMA-3 8B2026.02 | 72.97 | |
| FP16Bits=16, Model Architecture=LLAMA-2 13B2026.02 | 71.87 | |
| FP16Bits=16, Model Architecture=LLAMA-2 7B2026.02 | 68.88 | |
| EFFICIENTQATBits=2, Model Architecture=LLAMA-2 13B2026.02 | 68.06 | |
| EFFICIENTQATBits=2, Model Architecture=LLAMA-3 8B2026.02 | 63.49 | |
| EFFICIENTQATBits=2, Model Architecture=LLAMA-2 7B2026.02 | 62.93 | |
| NeUQISetting=W4A4, Backbone=LLaMA 2 7B, Hadamard Transform=true2025.05 | 62.77 | |
| MagRSetting=W4A4, Backbone=LLaMA 2 7B, Hadamard Transform=true2025.05 | 62.69 | |
| GPTAQSetting=W4A4, Backbone=LLaMA 2 7B, Hadamard Transform=true2025.05 | 61.93 | |
| GPTQSetting=W4A4, Backbone=LLaMA 2 7B, Hadamard Transform=true2025.05 | 61.67 | |
| LittleBit-2Bits=1.0, Model Architecture=LLAMA-2 13B2026.02 | 60.53 | |
| ONEBITBits=1.0, Model Architecture=LLAMA-2 13B2026.02 | 58.47 | |
| ARB-LLMBits=1.1, Model Architecture=LLAMA-2 13B2026.02 | 57.98 | |
| LittleBit-2Bits=1.0, Model Architecture=LLAMA-3 8B2026.02 | 57.33 | |
| LittleBit-2Bits=1.0, Model Architecture=LLAMA-2 7B2026.02 | 55.79 | |
| LittleBit-2Bits=0.55, Model Architecture=LLAMA-2 13B2026.02 | 55.64 | |
| ONEBITBits=1.0, Model Architecture=LLAMA-2 7B2026.02 | 54.72 | |
| NeUQISetting=W2A16, Backbone=LLaMA 2 7B, Hadamard Transform=true2025.05 | 52.91 | |
| LITTLEBITBits=1.0, Model Architecture=LLAMA-2 7B2026.02 | 52.81 | |
| ARB-LLMBits=1.1, Model Architecture=LLAMA-3 8B2026.02 | 52.75 | |
| ARB-LLMBits=1.1, Model Architecture=LLAMA-2 7B2026.02 | 52.32 | |
| ONEBITBits=1.0, Model Architecture=LLAMA-3 8B2026.02 | 52.23 | |
| LITTLEBITBits=0.55, Model Architecture=LLAMA-2 13B2026.02 | 51.76 | |
| LITTLEBITBits=1.0, Model Architecture=LLAMA-2 13B2026.02 | 51.49 | |
| LittleBit-2Bits=0.55, Model Architecture=LLAMA-3 8B2026.02 | 51.33 | |
| LittleBit-2Bits=0.55, Model Architecture=LLAMA-2 7B2026.02 | 50.97 | |
| NeUQISetting=W2A4, Backbone=LLaMA 2 7B, Hadamard Transform=true2025.05 | 50.33 | |
| BILLMBits=1.1, Model Architecture=LLAMA-2 13B2026.02 | 49.12 | |
| LITTLEBITBits=0.55, Model Architecture=LLAMA-2 7B2026.02 | 48.49 | |
| MagRSetting=W2A16, Backbone=LLaMA 2 7B, Hadamard Transform=true2025.05 | 48.01 | |
| LITTLEBITBits=1.0, Model Architecture=LLAMA-3 8B2026.02 | 47.11 | |
| LITTLEBIT-2 (OURS)BITS=0.12026.02 | 47.06 | |
| STBLLMBits=0.55, Model Architecture=LLAMA-2 13B2026.02 | 45.62 | |
| LITTLEBITBITS=0.12026.02 | 45.52 | |
| LITTLEBITBits=0.55, Model Architecture=LLAMA-3 8B2026.02 | 44.59 | |
| GPTQBits=2, Model Architecture=LLAMA-2 13B2026.02 | 44.34 | |
| STBLLMBits=0.55, Model Architecture=LLAMA-2 7B2026.02 | 43.4 | |
| MagRSetting=W2A4, Backbone=LLaMA 2 7B, Hadamard Transform=true2025.05 | 42.95 | |
| LittleBit-2Bits=0.1, Model Architecture=LLAMA-2 13B2026.02 | 42.18 | |
| LittleBit-2Bits=0.1, Model Architecture=LLAMA-2 7B2026.02 | 42.06 | |
| LITTLEBITBits=0.1, Model Architecture=LLAMA-3 8B2026.02 | 41.8 | |
| LittleBit-2Bits=0.1, Model Architecture=LLAMA-3 8B2026.02 | 41.64 | |
| LITTLEBITBits=0.1, Model Architecture=LLAMA-2 7B2026.02 | 41.33 | |
| LITTLEBITBits=0.1, Model Architecture=LLAMA-2 13B2026.02 | 40.99 | |
| BILLMBits=1.1, Model Architecture=LLAMA-2 7B2026.02 | 40.81 | |
| FP (TINY-RANK)BITS=0.12026.02 | 39.15 | |
| GPTQBits=2, Model Architecture=LLAMA-2 7B2026.02 | 38.79 | |
| BILLMBits=1.1, Model Architecture=LLAMA-3 8B2026.02 | 37.73 | |
| GPTAQSetting=W2A16, Backbone=LLaMA 2 7B, Hadamard Transform=true2025.05 | 37.14 | |
| GPTAQSetting=W2A4, Backbone=LLaMA 2 7B, Hadamard Transform=true2025.05 | 36.78 | |
| STBLLMBits=0.55, Model Architecture=LLAMA-3 8B2026.02 | 36.76 | |
| GPTQBits=2, Model Architecture=LLAMA-3 8B2026.02 | 36.01 | |
| GPTQSetting=W2A16, Backbone=LLaMA 2 7B, Hadamard Transform=true2025.05 | 34.95 | |
| GPTQSetting=W2A4, Backbone=LLaMA 2 7B, Hadamard Transform=true2025.05 | 34.88 |