Language Modeling on ArXiv OOD from The Pile
10.01PerplexityASEntmax
Evaluation Results
| Method | Links | |
|---|---|---|
| ASEntmaxContext Length=16K2025.06 | 10.01 | |
| EntmaxContext Length=16K2025.06 | 10.07 | |
| ASEntmaxContext Length=8K2025.06 | 10.89 | |
| EntmaxContext Length=8K2025.06 | 11.04 | |
| SSMaxContext Length=8K2025.06 | 12.29 | |
| SSMaxContext Length=16K2025.06 | 12.31 | |
| SoftmaxContext Length=8K2025.06 | 12.46 | |
| SoftmaxContext Length=16K2025.06 | 12.71 | |
| ASEntmaxContext Length=4K2025.06 | 13.31 | |
| EntmaxContext Length=4K2025.06 | 13.36 | |
| SSMaxContext Length=4K2025.06 | 13.74 | |
| SoftmaxContext Length=4K2025.06 | 13.87 |