Language Modeling on C4 en (val)
14.79PerplexityCR-Net
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| CR-NetModel size=1B, Training tokens (B)=29.52025.09 | 14.79 | — | — | — | — | — | — | — | |
| GaLoreModel size=1B, Training tokens (B)=29.52025.09 | 15.03 | — | — | — | — | — | — | — | |
| FedACTModel=Llama2-250M2026.07 | 16.8 | — | — | — | — | — | — | — | |
| FedAdamWModel=Llama2-250M2026.07 | 17.95 | — | — | — | — | — | — | — | |
| CR-NetModel size=350M, Training tokens (B)=8.02025.09 | 18.86 | — | — | — | — | — | — | — | |
| GaLoreModel size=350M, Training tokens (B)=8.02025.09 | 18.87 | — | — | — | — | — | — | — | |
| LocalAdamWModel=Llama2-250M2026.07 | 19.3 | — | — | — | — | — | — | — | |
| SCAFFOLDModel=Llama2-250M2026.07 | 19.71 | — | — | — | — | — | — | — | |
| FedACTModel=Llama2-130M2026.07 | 21.48 | — | — | — | — | — | — | — | |
| FedAdamWModel=Llama2-130M2026.07 | 22.82 | — | — | — | — | — | — | — | |
| CR-NetModel size=130M, Training tokens (B)=2.92025.09 | 23.73 | — | — | — | — | — | — | — | |
| FedLADAModel=Llama2-130M2026.07 | 25.06 | — | — | — | — | — | — | — | |
| GaLoreModel size=130M, Training tokens (B)=2.92025.09 | 25.14 | — | — | — | — | — | — | — | |
| LocalAdamWModel=Llama2-130M2026.07 | 26.33 | — | — | — | — | — | — | — | |
| FedLADAModel=Llama2-250M2026.07 | 27.36 | — | — | — | — | — | — | — | |
| SCAFFOLDModel=Llama2-130M2026.07 | 27.44 | — | — | — | — | — | — | — | |
| FedACTModel=Llama2-60M2026.07 | 32.57 | — | — | — | — | — | — | — | |
| FedAdamWModel=Llama2-60M2026.07 | 35.64 | — | — | — | — | — | — | — | |
| FedProxModel=Llama2-250M2026.07 | 35.98 | — | — | — | — | — | — | — | |
| LocalAdamWModel=Llama2-60M2026.07 | 37.09 | — | — | — | — | — | — | — | |
| FedLADAModel=Llama2-60M2026.07 | 37.87 | — | — | — | — | — | — | — | |
| SCAFFOLDModel=Llama2-60M2026.07 | 41.53 | — | — | — | — | — | — | — | |
| FedProxModel=Llama2-130M2026.07 | 48.38 | — | — | — | — | — | — | — | |
| Partially Reverse BDLM Mamba-HLR=8×10−32026.07 | 61.59 | — | — | — | — | — | — | 1.3 | |
| FedProxModel=Llama2-60M2026.07 | 69.66 | — | — | — | — | — | — | — | |
| BDLM attentionLR=4×10−32026.07 | 76.46 | — | — | — | — | — | — | 1.37 | |
| Full-sequence DiffuMamba-HLR=4×10−32026.07 | 83.73 | — | — | — | — | — | — | 1.39 | |
| Full-sequence attentionLR=8×10−32026.07 | 87.64 | — | — | — | — | — | — | 1.41 | |
| CR-NetBackbone=LLaMA-3 8B, low-rank coefficient rr=4482025.09 | — | 18.29 | 16.05 | 15.7 | 15.65 | — | — | — | |
| GaLoreBackbone=LLaMA-3 8B, low-rank coefficient rr=5122025.09 | — | 19.29 | 16.89 | 16.47 | 16.4 | — | — | — | |
| HyperQuantnom. bits=4, corr.=none, res. win.=–2026.06 | — | — | — | — | — | 1 | 3.7 | — | |
| HyperQuantnom. bits=4, corr.=qjl, res. win.=–2026.06 | — | — | — | — | — | 1 | 3.6 | — | |
| HyperQuantnom. bits=4, corr.=none, res. win.=322026.06 | — | — | — | — | — | 0.2 | 3.6 | — | |
| HyperQuantnom. bits=4, corr.=qjl, res. win.=322026.06 | — | — | — | — | — | 0.3 | 3.5 | — | |
| HyperQuantnom. bits=3, corr.=none, res. win.=–2026.06 | — | — | — | — | — | 5.7 | 4.8 | — | |
| HyperQuantnom. bits=3, corr.=qjl, res. win.=–2026.06 | — | — | — | — | — | 6.5 | 4.6 | — | |
| HyperQuantnom. bits=3, corr.=none, res. win.=322026.06 | — | — | — | — | — | 1.4 | 4.6 | — | |
| HyperQuantnom. bits=3, corr.=qjl, res. win.=322026.06 | — | — | — | — | — | 1.5 | 4.5 | — | |
| HyperQuantnom. bits=2, corr.=none, res. win.=–2026.06 | — | — | — | — | — | 54.3 | 6.6 | — | |
| HyperQuantnom. bits=2, corr.=qjl, res. win.=–2026.06 | — | — | — | — | — | 53.7 | 6.4 | — | |
| HyperQuantnom. bits=2, corr.=none, res. win.=322026.06 | — | — | — | — | — | 8.1 | 6.4 | — | |
| HyperQuantnom. bits=2, corr.=qjl, res. win.=322026.06 | — | — | — | — | — | 15.2 | 6.1 | — | |
| HyperQuantnom. bits=1.7, corr.=none, res. win.=322026.06 | — | — | — | — | — | 33.7 | 7.1 | — | |
| OCTOPUSnom. bits=4, corr.=none, res. win.=322026.06 | — | — | — | — | — | 1.5 | 2.2 | — | |
| OCTOPUSnom. bits=3, corr.=none, res. win.=322026.06 | — | — | — | — | — | 5.9 | 2.5 | — | |
| OCTOPUSnom. bits=2, corr.=none, res. win.=322026.06 | — | — | — | — | — | 41.5 | 2.9 | — | |
| OCTOPUS-QJLnom. bits=4, corr.=qjl, res. win.=322026.06 | — | — | — | — | — | 1.5 | 2 | — | |
| OCTOPUS-QJLnom. bits=3, corr.=qjl, res. win.=322026.06 | — | — | — | — | — | 6.1 | 2.3 | — | |
| OCTOPUS-QJLnom. bits=2, corr.=qjl, res. win.=322026.06 | — | — | — | — | — | 41.4 | 2.6 | — | |
| TurboQuant-MSEnom. bits=4, corr.=none, res. win.=322026.06 | — | — | — | — | — | 1.7 | 2.2 | — | |
| TurboQuant-MSEnom. bits=3, corr.=none, res. win.=322026.06 | — | — | — | — | — | 8.3 | 2.6 | — | |
| TurboQuant-MSEnom. bits=2, corr.=none, res. win.=322026.06 | — | — | — | — | — | 77.4 | 3 | — | |
| TurboQuant-QJLnom. bits=4, corr.=qjl, res. win.=322026.06 | — | — | — | — | — | 7.9 | 2.2 | — | |
| TurboQuant-QJLnom. bits=3, corr.=qjl, res. win.=322026.06 | — | — | — | — | — | 59.9 | 2.5 | — | |
| TurboQuant-QJLnom. bits=2, corr.=qjl, res. win.=322026.06 | — | — | — | — | — | 1,349 | 3 | — |