Language Modeling on Language Modeling Dataset
8.31PPLO-FANG
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| O-FANGModel=LLaMA-3.1-8B, Sparsity=20%2025.12 | 8.31 | — | |
| FP16Bits=16.02026.07 | 8.74 | — | |
| TASA b4.0Bits=4.02026.07 | 9.02 | — | |
| SpQR†Bits=4.12 (W3)2026.07 | 9.06 | — | |
| GPTQBits=4.02026.07 | 9.1 | — | |
| AWQBits=4.02026.07 | 9.13 | — | |
| OBCModel=LLaMA-3.1-8B, Sparsity=20%2025.12 | 9.17 | — | |
| HQQBits=4.02026.07 | 9.23 | — | |
| RTNBits=4.02026.07 | 9.75 | — | |
| TASA b3.0Bits=3.02026.07 | 9.96 | — | |
| OWQBits=3.02026.07 | 10.16 | — | |
| GPTQBits=3.02026.07 | 10.69 | — | |
| HQQBits=3.02026.07 | 11.65 | — | |
| O-FANGModel=Qwen2.5-7B, Sparsity=40%2025.12 | 11.72 | — | |
| DensePrune%=0%, Zero-shot=true, Backbone=LLaMA-2-7B2025.12 | 12.18 | — | |
| OBCModel=Qwen2.5-7B, Sparsity=40%2025.12 | 14.04 | — | |
| PPPrune%=20%, Zero-shot=true, Backbone=LLaMA-2-7B2025.12 | 15.31 | — | |
| FLAPPrune%=20%, Zero-shot=true, Backbone=LLaMA-2-7B2025.12 | 16.32 | — | |
| SlimGPT w/oPrune%=20%, Zero-shot=true, Backbone=LLaMA-2-7B2025.12 | 16.36 | — | |
| Token FilteringPrune%=20%, Zero-shot=true, Backbone=LLaMA-2-7B2025.12 | 16.65 | — | |
| GPT-2 XLParams=1.5B2026.05 | 17.48 | — | |
| CCT-306M (RC2)Params=306M2026.05 | 21.27 | — | |
| GPT-2 LargeParams=774M2026.05 | 22.05 | — | |
| Transformer-XLParams=257M2026.05 | 24 | — | |
| GPT-2 MediumParams=355M2026.05 | 26.37 | — | |
| GPT-2 SmallParams=124M2026.05 | 37.5 | — | |
| PPPrune%=50%, Zero-shot=true, Backbone=LLaMA-2-7B2025.12 | 39.4 | — | |
| SlimGPT w/oPrune%=50%, Zero-shot=true, Backbone=LLaMA-2-7B2025.12 | 40.83 | — | |
| FLAPPrune%=50%, Zero-shot=true, Backbone=LLaMA-2-7B2025.12 | 43.11 | — | |
| Token FilteringPrune%=50%, Zero-shot=true, Backbone=LLaMA-2-7B2025.12 | 54.59 | — | |
| RTNBits=3.02026.07 | 106.75 | — | |
| Adam-DecayModel size=7 B, Inner steps=240, Staleness (τ)=0, Number of seeds (n)=22026.05 | — | 11.28 | |
| Adam-DecayModel size=7 B, Inner steps=240, Staleness (τ)=8, Number of seeds (n)=22026.05 | — | 11.3 | |
| Adam-DecayModel size=7 B, Inner steps=240, Staleness (τ)=16, Number of seeds (n)=22026.05 | — | 11.69 | |
| CGADModel size=25 M, Staleness (τ)=0, Number of seeds (n)=3–82026.05 | — | 6.79 | |
| CGADModel size=25 M, Staleness (τ)=8, Number of seeds (n)=3–82026.05 | — | 8.46 | |
| CGADModel size=25 M, Staleness (τ)=16, Number of seeds (n)=3–82026.05 | — | 8.96 | |
| CGADModel size=1 B, Staleness (τ)=0, Number of seeds (n)=42026.05 | — | 7.48 | |
| CGADModel size=1 B, Staleness (τ)=8, Number of seeds (n)=32026.05 | — | 9.94 | |
| CGADModel size=1 B, Staleness (τ)=16, Number of seeds (n)=12026.05 | — | 7.44 | |
| CGADModel size=7 B, Inner steps=240, Staleness (τ)=0, Number of seeds (n)=42026.05 | — | 10.48 | |
| CGADModel size=7 B, Inner steps=240, Staleness (τ)=8, Number of seeds (n)=32026.05 | — | 10.9 | |
| CGADModel size=7 B, Inner steps=240, Staleness (τ)=16, Number of seeds (n)=22026.05 | — | 11.7 | |
| DelayedNesterovModel size=7 B, Inner steps=240, Staleness (τ)=0, Number of seeds (n)=22026.05 | — | 11.1 | |
| DelayedNesterovModel size=7 B, Inner steps=240, Staleness (τ)=8, Number of seeds (n)=22026.05 | — | 10.39 | |
| DelayedNesterovModel size=7 B, Inner steps=240, Staleness (τ)=16, Number of seeds (n)=22026.05 | — | 11.69 | |
| NesterovModel size=25 M, Staleness (τ)=0, Number of seeds (n)=3–52026.05 | — | 8.04 | |
| NesterovModel size=25 M, Staleness (τ)=8, Number of seeds (n)=3–52026.05 | — | 170.9 | |
| NesterovModel size=25 M, Staleness (τ)=16, Number of seeds (n)=3–52026.05 | — | 487.6 | |
| NesterovModel size=1 B, Staleness (τ)=0, Number of seeds (n)=52026.05 | — | 9.7 | |
| NesterovModel size=1 B, Staleness (τ)=8, Number of seeds (n)=32026.05 | — | 241.2 | |
| NesterovModel size=1 B, Staleness (τ)=16, Number of seeds (n)=22026.05 | — | 319 | |
| NesterovModel size=7 B, Inner steps=240, Staleness (τ)=0, Number of seeds (n)=42026.05 | — | 46.69 | |
| NesterovModel size=7 B, Inner steps=240, Staleness (τ)=8, Number of seeds (n)=12026.05 | — | 65.31 | |
| SDMModel size=7 B, Inner steps=240, Staleness (τ)=0, Number of seeds (n)=22026.05 | — | 11.3 | |
| SDMModel size=7 B, Inner steps=240, Staleness (τ)=8, Number of seeds (n)=22026.05 | — | 11.44 | |
| SDMModel size=7 B, Inner steps=240, Staleness (τ)=16, Number of seeds (n)=22026.05 | — | 11.69 |