Online Language Modeling on PG-19 (Whole Book)
18.87PPL @ 50KTTT w/ Locas-MLP + NL-SVD
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| TTT w/ Locas-MLP + NL-SVDBackbone=0.6B MLP-FFN LM, Compression=NL-SVD, Extra #Params=4.2M, Relative Time=12.7x2026.02 | 18.87 | 18.46 | 18.23 | 17.89 | |
| TTT w/ Locas-MLP + BPBackbone=0.6B MLP-FFN LM, Optimization=Back-propagation (BP), Extra #Params=4.2M, Relative Time=1.7x2026.02 | 18.89 | 18.44 | 18.22 | 17.9 | |
| TTT w/ TempLoRABackbone=0.6B MLP-FFN LM, Extra #Params=41.9M, Relative Time=4.7x2026.02 | 18.94 | 18.47 | 18.25 | 17.93 | |
| Cxt. Trunc. (4K)Backbone=0.6B MLP-FFN LM, Extra #Params=0, Relative Time=1.4x2026.02 | 18.98 | 18.64 | 18.54 | 18.31 | |
| Long Cxt. Attn. (16K)Backbone=Qwen3-1.7B-Base, Extra #Params=0, Relative Time=18.6x2026.02 | 18.98 | 18.7 | 18.53 | 18.58 | |
| Cxt. Trunc. (2K)Backbone=0.6B MLP-FFN LM, Extra #Params=0, Relative Time=1x2026.02 | 19.35 | 19.03 | 18.92 | 18.72 | |
| TTT w/ Locas-GLUBackbone=Qwen3-1.7B-Base, Extra #Params=11.0M, Relative Time=1.8x2026.02 | 20 | 19.49 | 19.12 | 19.04 | |
| TTT w/ TempLoRABackbone=Qwen3-1.7B-Base, Extra #Params=73.4M, Relative Time=4.7x2026.02 | 20.03 | 19.55 | 19.2 | 19.13 | |
| Cxt. Trunc. (2K)Backbone=Qwen3-1.7B-Base, Extra #Params=0, Relative Time=1x2026.02 | 20.6 | 20.38 | 20.25 | 20.5 | |
| TTT w/ Locas-GLUBackbone=Qwen3-1.7B(-Instruct), Extra #Params=11.0M, Relative Time=1.8x2026.02 | 23.65 | 22.55 | 21.92 | 21.65 | |
| TTT w/ TempLoRABackbone=Qwen3-1.7B(-Instruct), Extra #Params=73.4M, Relative Time=4.7x2026.02 | 23.74 | 22.57 | 21.92 | 21.65 | |
| Long Cxt. Attn. (16K)Backbone=Qwen3-0.6B-Base, Extra #Params=0, Relative Time=13.9x2026.02 | 25.24 | 24.79 | 24.55 | 24.57 | |
| TTT w/ Locas-GLUBackbone=Qwen3-0.6B-Base, Extra #Params=5.5M, Relative Time=1.4x2026.02 | 26.58 | 25.76 | 25.21 | 25 | |
| TTT w/ TempLoRABackbone=Qwen3-0.6B-Base, Extra #Params=36.7M, Relative Time=4.3x2026.02 | 26.71 | 25.94 | 25.42 | 25.22 | |
| Cxt. Trunc. (2K)Backbone=Qwen3-0.6B-Base, Extra #Params=0, Relative Time=1x2026.02 | 27.49 | 27.13 | 26.94 | 27.28 | |
| Long Cxt. Attn. (16K)Backbone=Qwen3-1.7B(-Instruct), Extra #Params=0, Relative Time=18.6x2026.02 | 31.45 | 31.18 | 30.97 | 31.03 | |
| Cxt. Trunc. (2K)Backbone=Qwen3-1.7B(-Instruct), Extra #Params=0, Relative Time=1x2026.02 | 31.59 | 31.27 | 31.09 | 31.49 |