Long-context language modeling on OpenWebText (val)
3.93NLL (Context=1024)Tensor Cache
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Tensor CacheParameters=130M, Training iterations=50K, Training block size=1024, Window size (W)=5122026.05 | 3.93 | 4 | 4.14 | 4.29 | 4.55 | 5.14 | 0.8 | |
| Full KVParameters=130M, Training iterations=50K, Training block size=1024, Window size (W)=5122026.05 | 4.05 | 4.46 | 4.49 | 5.13 | 5.4 | 6 | 1.94 | |
| StreamingLLMParameters=130M, Training iterations=50K, Training block size=1024, Window size (W)=5122026.05 | 4.11 | 4.66 | 4.74 | 4.85 | 5.35 | 5.65 | 0.76 | |
| Infini-attentionParameters=130M, Training iterations=50K, Training block size=1024, Window size (W)=5122026.05 | 4.14 | 4.05 | 4.49 | 4.57 | 5.01 | 5.42 | 0.76 | |
| Window KVParameters=130M, Training iterations=50K, Training block size=1024, Window size (W)=5122026.05 | 4.16 | 3.98 | 4.86 | 4.78 | 5.08 | 5.44 | 0.75 |