Commonsense Reasoning on LAMBADA (ppl)
7.02PerplexitySWA-CPT
Evaluation Results
| Method | Links | |
|---|---|---|
| SWA-CPTAttention Mechanism=Sliding Window Attention, Training/Evaluation Protocol=Continual pretraining2026.03 | 7.02 | |
| SWA-ZSAttention Mechanism=Sliding Window Attention, Training/Evaluation Protocol=Zero-shot2026.03 | 7.16 | |
| FullAttnAttention Mechanism=Full attention2026.03 | 7.16 | |
| SwiAttnAttention Mechanism=Switch Attention2026.03 | 7.28 | |
| StaticHybridAttention Mechanism=Static hybrid of full attention and SWA, Ratio (Full:SWA)=1:32026.03 | 7.32 |