Language Modeling on WikiText-103 (fine-tuning) - Efficiency
12.4Training Time (s)π-Attention
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| π-AttentionPhase=Fine-tuning2025.11 | 12.4 | 36.7 | 8.8 | 262 | 55.4 | — | |
| LongformerPhase=Fine-tuning2025.11 | 14.2 | 40.1 | 9.5 | 330 | 51.2 | — | |
| RingAttentionPhase=Fine-tuning2025.11 | 14.6 | 44.3 | 9.2 | 345 | 51.7 | — | |
| BigBirdPhase=Fine-tuning2025.11 | 15.3 | 42.8 | 10.2 | 320 | 48.7 | — | |
| Full AttentionPhase=Fine-tuning2025.11 | 45.2 | 125.4 | 32.1 | 345 | 23.4 | — | |
| GoSGDArchitecture=GPT-2 XL, Evaluation Protocol=Finetuning2024.10 | 7,652.33 | — | — | — | — | 9.82 | |
| LayUpArchitecture=GPT-2 XL, Evaluation Protocol=Finetuning2024.10 | 7,688.33 | — | — | — | — | 9.75 | |
| AD-PSGDArchitecture=GPT-2 XL, Evaluation Protocol=Finetuning2024.10 | 7,860 | — | — | — | — | 10.52 | |
| CO2Architecture=GPT-2 XL, Evaluation Protocol=Finetuning2024.10 | 9,789.33 | — | — | — | — | 9.91 | |
| SlowMoArchitecture=GPT-2 XL, Evaluation Protocol=Finetuning2024.10 | 9,791.33 | — | — | — | — | 9.88 | |
| DDPArchitecture=GPT-2 XL, Evaluation Protocol=Finetuning2024.10 | 11,283.33 | — | — | — | — | 9.91 |