Reading Comprehension on SQuAD (EM, F1, HA-EM, HA-F1, NA-F1)
28.32EMSDLLM
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| SDLLMBackbone=LLaMA3-8B, Spike=✓, Bit=6-1.58, T x D=1 × 32, R=0.213, FLOPs (T)=5.09, Power (J)=4.582026.04 | 28.32 | 34.74 | 54.31 | 66.61 | 0.05 | |
| LLaMA3-8BSpike=✗, Bit=-, T x D=-, R=-, FLOPs (T)=7.97, Power (J)=36.672026.04 | 26.71 | 32.58 | 52.69 | 64.3 | 0.07 | |
| SDLLMBackbone=LLaMA3-8B, Spike=✓, Bit=4-1.58, T x D=1 × 16, R=0.211, FLOPs (T)=1.68, Power (J)=1.512026.04 | 24.55 | 30.89 | 48.37 | 60.89 | 0.14 | |
| QuaRotBackbone=LLaMA2-13B, Spike=✗, Bit=6-6, T x D=-, R=-, FLOPs (T)=3.77, Power (J)=17.362026.04 | 23.85 | 30.42 | 41.25 | 54.24 | 6 | |
| LLaMA2-13BSpike=✗, Bit=-, T x D=-, R=-, FLOPs (T)=13.42, Power (J)=61.742026.04 | 22.77 | 29.67 | 37.19 | 50.82 | 7.98 | |
| SDLLMBackbone=LLaMA2-13B, Spike=✓, Bit=6-1.58, T x D=1 × 32, R=0.217, FLOPs (T)=8.74, Power (J)=7.862026.04 | 21.99 | 29.06 | 34.46 | 48.43 | 9.21 | |
| QuaRotBackbone=LLaMA2-13B, Spike=✗, Bit=4-4, T x D=-, R=-, FLOPs (T)=1.68, Power (J)=7.722026.04 | 21.96 | 29.62 | 41.78 | 56.9 | 1.64 | |
| SDLLMBackbone=LLaMA2-13B, Spike=✓, Bit=4-1.58, T x D=1 × 8, R=0.209, FLOPs (T)=1.40, Power (J)=1.262026.04 | 21.29 | 28.49 | 37.52 | 51.75 | 4.64 | |
| SDLLMBackbone=LLaMA2-13B, Spike=✓, Bit=4-1.58, T x D=1 × 16, R=0.215, FLOPs (T)=2.89, Power (J)=2.602026.04 | 21.15 | 28.82 | 33.07 | 48.21 | 8.94 | |
| SDLLMBackbone=LLaMA3-8B, Spike=✓, Bit=4-1.58, T x D=1 × 8, R=0.210, FLOPs (T)=0.84, Power (J)=0.752026.04 | 19.65 | 27.43 | 37.66 | 52.59 | 0.05 | |
| SDLLMBackbone=LLaMA2-7B, Spike=✓, Bit=4-1.58, T x D=1 × 8, R=0.216, FLOPs (T)=0.75, Power (J)=0.672026.04 | 19.16 | 25.52 | 18.96 | 31.52 | 19.37 | |
| SDLLMBackbone=LLaMA2-7B, Spike=✓, Bit=6-1.58, T x D=1 × 32, R=0.222, FLOPs (T)=4.60, Power (J)=4.142026.04 | 18.05 | 25.12 | 24.82 | 38.77 | 11.12 | |
| QuaRotBackbone=LLaMA2-7B, Spike=✗, Bit=4-4, T x D=-, R=-, FLOPs (T)=0.86, Power (J)=3.972026.04 | 17.04 | 24.76 | 29.01 | 44.26 | 4.77 | |
| LLaMA2-7BSpike=✗, Bit=-, T x D=-, R=-, FLOPs (T)=6.91, Power (J)=31.772026.04 | 16.77 | 24.21 | 24.68 | 39.37 | 8.66 | |
| QuaRotBackbone=LLaMA2-7B, Spike=✗, Bit=6-6, T x D=-, R=-, FLOPs (T)=1.94, Power (J)=8.942026.04 | 16.5 | 24.37 | 22.22 | 37.77 | 10.64 | |
| SDLLMBackbone=LLaMA2-7B, Spike=✓, Bit=4-1.58, T x D=1 × 16, R=0.221, FLOPs (T)=1.53, Power (J)=1.372026.04 | 13.88 | 22.53 | 20.76 | 37.86 | 6.82 |