Zero-shot Evaluation on PiQA, LAMBADA, HellaSwag, and ARC (Easy/Challenge)
62.47PiQA AccuracyDense (full)
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Dense (full)Model=Qwen3-0.6B, Latency (128k context)=77.652026.03 | 62.47 | 34.82 | 45.41 | 20.35 | 33.95 | 39.4 | |
| SFA (k = 16)Model=Qwen3-0.6B, Latency (128k context)=34.202026.03 | 61.73 | 34.05 | 45.62 | 19.27 | 34.03 | 38.94 | |
| Dense (full)Model=GPT2-350M, Latency (128k context)=46.782026.03 | 59.79 | 24.74 | 30.19 | 15.78 | 22.04 | 30.51 | |
| Dense (d = 64)Model=Qwen3-0.6B, Latency (128k context)=30.842026.03 | 58.43 | 31.27 | 41.58 | 15.83 | 28.29 | 36.68 | |
| SFA (k = 8)Model=GPT2-350M, Latency (128k context)=23.672026.03 | 58.02 | 23.83 | 30.22 | 13.66 | 22.13 | 29.57 | |
| Dense (full)Model=GPT2-124M, Latency (128k context)=16.862026.03 | 56.34 | 22.78 | 28.35 | 14.32 | 19.61 | 28.28 | |
| Dense (d = 32)Model=GPT2-350M, Latency (128k context)=20.582026.03 | 55.17 | 19.96 | 28.15 | 11.83 | 18.43 | 26.71 | |
| SFA (k = 8)Model=GPT2-124M, Latency (128k context)=9.412026.03 | 54.92 | 21.03 | 28.41 | 7.39 | 19.26 | 27.4 | |
| Dense (d = 32)Model=GPT2-124M, Latency (128k context)=7.862026.03 | 51.3 | 19.39 | 25.72 | 12.47 | 14.26 | 24.63 |