Language Modeling Evaluation
1.71Perplexity (PPL)Gated Subspace Inference
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Gated Subspace InferenceModel=GPT-2 124M (d = 768, L = 12), k/ε=256/0.102026.05 | 1.71 | 0.986 | 99 | 100 | 3 | |
| Gated Subspace InferenceModel=GPT-2 124M (d = 768, L = 12), k/ε=128/0.052026.05 | 1.72 | 0.995 | 99.4 | 41.1 | 1.5 | |
| Gated Subspace InferenceModel=GPT-2 124M (d = 768, L = 12), k/ε=256/0.052026.05 | 1.72 | 0.991 | 99.4 | 91.4 | 2.6 | |
| Gated Subspace InferenceModel=GPT-J 6B (d = 4096, L = 28), k/ε=256/0.102026.05 | 1.72 | 0.991 | 98.6 | 99.8 | 15.6 | |
| Gated Subspace InferenceModel=GPT-J 6B (d = 4096, L = 28), k/ε=256/0.152026.05 | 1.72 | 0.992 | 98.6 | 100 | 16 | |
| Baseline GPT-2 124MModel=GPT-2 124M (d = 768, L = 12)2026.05 | 1.73 | — | — | — | — | |
| Baseline GPT-J 6BModel=GPT-J 6B (d = 4096, L = 28)2026.05 | 1.73 | — | — | — | — | |
| Gated Subspace InferenceModel=GPT-J 6B (d = 4096, L = 28), k/ε=128/0.052026.05 | 1.73 | 0.999 | 99.4 | 21.1 | 1.3 | |
| Gated Subspace InferenceModel=GPT-J 6B (d = 4096, L = 28), k/ε=256/0.052026.05 | 1.73 | 0.997 | 99.2 | 77.2 | 3.6 | |
| Gated Subspace InferenceModel=GPT-J 6B (d = 4096, L = 28), k/ε=128/0.102026.05 | 1.74 | 1.002 | 99 | 63.6 | 2.6 | |
| Gated Subspace InferenceModel=GPT-J 6B (d = 4096, L = 28), k/ε=128/0.152026.05 | 1.74 | 1.005 | 96.5 | 81.9 | 4.8 | |
| Gated Subspace InferenceModel=OPT 6.7B (d = 4096, L = 32), k/ε=256/0.152026.05 | 1.8 | 0.986 | 99.2 | 97.3 | 11.4 | |
| Gated Subspace InferenceModel=OPT 6.7B (d = 4096, L = 32), k/ε=128/0.052026.05 | 1.81 | 0.989 | 98.2 | 45.7 | 1.8 | |
| Gated Subspace InferenceModel=OPT 6.7B (d = 4096, L = 32), k/ε=256/0.102026.05 | 1.81 | 0.989 | 98.8 | 96.5 | 10.5 | |
| Gated Subspace InferenceModel=OPT 6.7B (d = 4096, L = 32), k/ε=256/0.052026.05 | 1.82 | 0.995 | 98.6 | 77.4 | 3.7 | |
| Baseline OPT 6.7BModel=OPT 6.7B (d = 4096, L = 32)2026.05 | 1.83 | — | — | — | — | |
| FP16Model=LLaMA-2-7B, Setting=FP162026.06 | 5.471 | — | — | — | — | |
| FP16Model=LLaMA-3.1-8B, Setting=FP162026.06 | 6.277 | — | — | — | — | |
| SYM-INTModel=LLaMA-2-7B, Setting=W3A4 (GPTQ)2026.06 | 6.897 | — | — | — | — | |
| Qift-MNZModel=LLaMA-2-7B, Setting=L=16 (GPTAQ)2026.06 | 7.122 | — | — | — | — | |
| Qift-MNZModel=LLaMA-2-7B, Setting=Pure W2A4 (GPTAQ)2026.06 | 9.294 | — | — | — | — | |
| SYM-INTModel=LLaMA-3.1-8B, Setting=W3A4 (GPTQ)2026.06 | 10.954 | — | — | — | — | |
| Qift-MNZModel=LLaMA-3.1-8B, Setting=L=16 (GPTAQ)2026.06 | 11.936 | — | — | — | — | |
| SYM-INTModel=LLaMA-2-7B, Setting=Pure W2A4 (GPTAQ)2026.06 | 12.118 | — | — | — | — | |
| Qift-MNZModel=LLaMA-3.1-8B, Setting=Pure W2A4 (GPTAQ)2026.06 | 19.515 | — | — | — | — | |
| SYM-INTModel=LLaMA-3.1-8B, Setting=Pure W2A4 (GPTAQ)2026.06 | 29.695 | — | — | — | — |