Natural Language Understanding on PIQA (accuracy)
83.9PIQA AccuracyNLS
Evaluation results
| Method | Links | |
|---|---|---|
| NLSModel=Llama 3 70B, Precision=W8A8, Time Steps (T)=12026.05 | 83.9 | |
| SNNModel=Llama 2 7B, Precision=W8A8, Time Steps (T)=42026.05 | 78.24 | |
| SpaceByte + SPSequence Reduction=6.3, FLOPs/byte Reduction=2.72026.05 | 73.8 | |
| H-NetSequence Reduction=4.8, FLOPs/byte Reduction=3.42026.05 | 73.7 | |
| Tokenizer-basedSequence Reduction=3.7, FLOPs/byte Reduction=4.12026.05 | 73.6 | |
| Fixed (p = 8) + SPSequence Reduction=8.0, FLOPs/byte Reduction=2.62026.05 | 73 | |
| Fixed (p = 4) + SPSequence Reduction=4.0, FLOPs/byte Reduction=2.12026.05 | 72.9 | |
| Byte-levelSequence Reduction=1.0, FLOPs/byte Reduction=1.02026.05 | 72.8 | |
| SpaceByteSequence Reduction=6.3, FLOPs/byte Reduction=4.02026.05 | 72.8 | |
| Entropy-based + SPSequence Reduction=6.3†, FLOPs/byte Reduction=2.72026.05 | 72.8 | |
| Fixed (p = 16) + SPSequence Reduction=16.0, FLOPs/byte Reduction=2.92026.05 | 72.5 | |
| Entropy-basedSequence Reduction=6.4, FLOPs/byte Reduction=4.02026.05 | 72.4 | |
| H-Net + SPSequence Reduction=5.3†, FLOPs/byte Reduction=2.42026.05 | 72.1 | |
| Fixed (p = 8)Sequence Reduction=8.0, FLOPs/byte Reduction=4.52026.05 | 71 | |
| Fixed (p = 4)Sequence Reduction=4.0, FLOPs/byte Reduction=3.12026.05 | 70.4 | |
| Fixed (p = 16)Sequence Reduction=16.0, FLOPs/byte Reduction=5.72026.05 | 67.7 |