Language Modeling on FineWeb-Edu T=4096
41.45Perplexityfib / stagger W=12
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| fib / stagger W=12Training Length=10242026.06 | 41.45 | 1.3 | |
| fib / stagger W=8Training Length=10242026.06 | 41.98 | 1.3 | |
| fib / learned W=8Training Length=10242026.06 | 43.15 | 1.3 | |
| fib / coprime W=12Training Length=10242026.06 | 43.25 | 2.8 | |
| fib / fixed W=8Training Length=10242026.06 | 43.56 | 1.2 | |
| pow2 / fixed W=8Training Length=10242026.06 | 49.86 | 11.3 | |
| denseTraining Length=10242026.06 | 100.36 | 201 |