Large Language Model Inference on Prefill Phase SeqLen=2k
0.025Prefill Time (s)BWTA (Bitnet-b1.58-2B)
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| BWTA (Bitnet-b1.58-2B)Wbit/Abit=1.35/6.05 (30% layers), Storage (GB)=1.796, BS=12026.04 | 0.025 | 81,529 | 10.1 | |
| Bitnet-b1.58-2BWbit/Abit=1.5/8, Storage (GB)=1.835, BS=12026.04 | 0.026 | 77,253 | 10.56 | |
| BWTA (Bitnet-b1.58-2B)Wbit/Abit=1.45/7.35 (10% layers), Storage (GB)=1.819, BS=12026.04 | 0.026 | 77,225 | 10.28 | |
| Gemma-2BWbit/Abit=16/16, Storage (GB)=5.012, BS=12026.04 | 0.063 | 32,492 | 20.75 | |
| Llama-3.2-3BWbit/Abit=4/16 (bnb), Storage (GB)=2.240, BS=12026.04 | 0.072 | 28,267 | 15.59 | |
| Gemma-2BWbit/Abit=4/16 (gptq), Storage (GB)=3.130, BS=12026.04 | 0.107 | 19,133 | 22.53 | |
| Gemma-2BWbit/Abit=16/16, Storage (GB)=5.012, BS=42026.04 | 0.116 | 17,694 | 64.51 | |
| Llama-3.2-3BWbit/Abit=4/16 (bnb), Storage (GB)=2.240, BS=42026.04 | 0.172 | 11,924 | 54.94 | |
| Gemma-2BWbit/Abit=4/16 (gptq), Storage (GB)=3.130, BS=42026.04 | 0.173 | 11,858 | 57.71 | |
| BWTA (Bitnet-b1.58-2B)Wbit/Abit=1.35/6.05 (30% layers), Storage (GB)=1.796, BS=82026.04 | 0.179 | 11,466 | 32.02 | |
| BWTA (Bitnet-b1.58-2B)Wbit/Abit=1.45/7.35 (10% layers), Storage (GB)=1.819, BS=82026.04 | 0.185 | 11,096 | 31.48 | |
| Bitnet-b1.58-2BWbit/Abit=1.5/8, Storage (GB)=1.835, BS=82026.04 | 0.189 | 10,814 | 31.67 | |
| BWTA (Bitnet-b1.58-2B)Wbit/Abit=1.35/6.05 (30% layers), Storage (GB)=1.796, BS=162026.04 | 0.356 | 5,745 | 56.94 | |
| BWTA (Bitnet-b1.58-2B)Wbit/Abit=1.45/7.35 (10% layers), Storage (GB)=1.819, BS=162026.04 | 0.369 | 5,544 | 55.41 | |
| Bitnet-b1.58-2BWbit/Abit=1.5/8, Storage (GB)=1.835, BS=162026.04 | 0.378 | 5,415 | 55.6 |