Mathematical Reasoning on GSM8K (Str., Flex.)
48.64String Accuracy (Str.)LLaMA3-8B
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| LLaMA3-8BSpike=✗, Bit=-, T x D=-, R=-, FLOPs (T)=7.97, Power (J)=36.672026.04 | 48.64 | 49.55 | |
| SDLLMBackbone=LLaMA3-8B, Spike=✓, Bit=6-1.58, T x D=1 × 32, R=0.213, FLOPs (T)=5.09, Power (J)=4.582026.04 | 46.55 | 47.16 | |
| SDLLMBackbone=LLaMA3-8B, Spike=✓, Bit=4-1.58, T x D=1 × 16, R=0.211, FLOPs (T)=1.68, Power (J)=1.512026.04 | 31.84 | 32.6 | |
| LLaMA2-13BSpike=✗, Bit=-, T x D=-, R=-, FLOPs (T)=13.42, Power (J)=61.742026.04 | 22.73 | 22.88 | |
| QuaRotBackbone=LLaMA2-13B, Spike=✗, Bit=6-6, T x D=-, R=-, FLOPs (T)=3.77, Power (J)=17.362026.04 | 21.82 | 21.97 | |
| SDLLMBackbone=LLaMA2-13B, Spike=✓, Bit=6-1.58, T x D=1 × 32, R=0.217, FLOPs (T)=8.74, Power (J)=7.862026.04 | 21.52 | 21.82 | |
| SDLLMBackbone=LLaMA2-13B, Spike=✓, Bit=4-1.58, T x D=1 × 16, R=0.215, FLOPs (T)=2.89, Power (J)=2.602026.04 | 21.06 | 21.36 | |
| SDLLMBackbone=LLaMA3-8B, Spike=✓, Bit=4-1.58, T x D=1 × 8, R=0.210, FLOPs (T)=0.84, Power (J)=0.752026.04 | 18.8 | 19.41 | |
| SDLLMBackbone=LLaMA2-13B, Spike=✓, Bit=4-1.58, T x D=1 × 8, R=0.209, FLOPs (T)=1.40, Power (J)=1.262026.04 | 15.91 | 16.21 | |
| LLaMA2-7BSpike=✗, Bit=-, T x D=-, R=-, FLOPs (T)=6.91, Power (J)=31.772026.04 | 15.3 | 15.3 | |
| SDLLMBackbone=LLaMA2-7B, Spike=✓, Bit=6-1.58, T x D=1 × 32, R=0.222, FLOPs (T)=4.60, Power (J)=4.142026.04 | 14.85 | 15.11 | |
| QuaRotBackbone=LLaMA2-7B, Spike=✗, Bit=6-6, T x D=-, R=-, FLOPs (T)=1.94, Power (J)=8.942026.04 | 13.48 | 13.79 | |
| QuaRotBackbone=LLaMA2-13B, Spike=✗, Bit=4-4, T x D=-, R=-, FLOPs (T)=1.68, Power (J)=7.722026.04 | 12.42 | 12.42 | |
| SDLLMBackbone=LLaMA2-7B, Spike=✓, Bit=4-1.58, T x D=1 × 16, R=0.221, FLOPs (T)=1.53, Power (J)=1.372026.04 | 10 | 10.15 | |
| SDLLMBackbone=LLaMA2-7B, Spike=✓, Bit=4-1.58, T x D=1 × 8, R=0.216, FLOPs (T)=0.75, Power (J)=0.672026.04 | 6.36 | 6.67 | |
| QuaRotBackbone=LLaMA2-7B, Spike=✗, Bit=4-4, T x D=-, R=-, FLOPs (T)=0.86, Power (J)=3.972026.04 | 1.97 | 2.73 |