Large Language Model Inference on Decode Phase BS=1
0.152Latency (s)BWTA (Bitnet-b1.58-2B)
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| BWTA (Bitnet-b1.58-2B)Wbit/Abit=1.35/6.05 (30% layers), Storage (GB)=1.796, G. Len.=502026.04 | 0.152 | 375.4 | |
| BWTA (Bitnet-b1.58-2B)Wbit/Abit=1.45/7.35 (10% layers), Storage (GB)=1.819, G. Len.=502026.04 | 0.156 | 365.9 | |
| Bitnet-b1.58-2BWbit/Abit=1.5/8, Storage (GB)=1.835, G. Len.=502026.04 | 0.158 | 360.2 | |
| BWTA (Bitnet-b1.58-2B)Wbit/Abit=1.35/6.05 (30% layers), Storage (GB)=1.796, G. Len.=1002026.04 | 0.285 | 344.4 | |
| BWTA (Bitnet-b1.58-2B)Wbit/Abit=1.45/7.35 (10% layers), Storage (GB)=1.819, G. Len.=1002026.04 | 0.29 | 338 | |
| Bitnet-b1.58-2BWbit/Abit=1.5/8, Storage (GB)=1.835, G. Len.=1002026.04 | 0.295 | 331 | |
| BWTA (Bitnet-b1.58-2B)Wbit/Abit=1.35/6.05 (30% layers), Storage (GB)=1.796, G. Len.=1502026.04 | 0.443 | 345.8 | |
| BWTA (Bitnet-b1.58-2B)Wbit/Abit=1.45/7.35 (10% layers), Storage (GB)=1.819, G. Len.=1502026.04 | 0.449 | 338.4 | |
| Bitnet-b1.58-2BWbit/Abit=1.5/8, Storage (GB)=1.835, G. Len.=1502026.04 | 0.458 | 333.9 | |
| Gemma-2BWbit/Abit=16/16, Storage (GB)=5.012, G. Len.=502026.04 | 0.512 | 111.6 | |
| Gemma-2BWbit/Abit=4/16 (gptq), Storage (GB)=3.130, G. Len.=502026.04 | 0.989 | 50.8 | |
| Gemma-2BWbit/Abit=16/16, Storage (GB)=5.012, G. Len.=1002026.04 | 1.146 | 87.3 | |
| Llama-3.2-3BWbit/Abit=4/16 (bnb), Storage (GB)=2.240, G. Len.=502026.04 | 1.317 | 43.4 | |
| Gemma-2BWbit/Abit=16/16, Storage (GB)=5.012, G. Len.=1502026.04 | 1.86 | 83.3 | |
| Gemma-2BWbit/Abit=4/16 (gptq), Storage (GB)=3.130, G. Len.=1002026.04 | 1.959 | 51 | |
| Llama-3.2-3BWbit/Abit=4/16 (bnb), Storage (GB)=2.240, G. Len.=1002026.04 | 2.563 | 39 | |
| Gemma-2BWbit/Abit=4/16 (gptq), Storage (GB)=3.130, G. Len.=1502026.04 | 2.932 | 52.9 | |
| Llama-3.2-3BWbit/Abit=4/16 (bnb), Storage (GB)=2.240, G. Len.=1502026.04 | 3.507 | 44.2 |