Language Modeling on Wikitext-103 (PPL)
14.8Perplexity (PPL)GPT2 - M
Evaluation Results
| Method | Links | |
|---|---|---|
| GPT2 - MModel=GPT2 - M, Evaluation Protocol=Fine-Tuned, Bits per Token=7864322025.05 | 14.8 | |
| ASTRAModel=GPT2 - M, #Groups=32, Evaluation Protocol=Fine-Tuned, Bits per Token=7680, Compression Ratio=102.42025.05 | 16.84 | |
| ASTRAModel=GPT2 - M, #Groups=16, Evaluation Protocol=Fine-Tuned, Bits per Token=3840, Compression Ratio=204.82025.05 | 18.03 | |
| GPT2 - SModel=GPT2 - S, Evaluation Protocol=Fine-Tuned, Bits per Token=2949122025.05 | 18.96 | |
| ASTRAModel=GPT2 - S, #Groups=32, Evaluation Protocol=Fine-Tuned, Bits per Token=3840, Compression Ratio=76.82025.05 | 20.95 | |
| ASTRAModel=GPT2 - M, #Groups=1, Evaluation Protocol=Fine-Tuned, Bits per Token=240, Compression Ratio=3276.82025.05 | 21.97 | |
| ASTRAModel=GPT2 - S, #Groups=16, Evaluation Protocol=Fine-Tuned, Bits per Token=1920, Compression Ratio=153.62025.05 | 23.22 | |
| ASTRAModel=GPT2 - S, #Groups=1, Evaluation Protocol=Fine-Tuned, Bits per Token=120, Compression Ratio=2457.62025.05 | 25.98 | |
| BLIND-VALM-MModel=BLIND-VALM-M2024.09 | 33.95 | |
| Manta-LMZero-shot=true, Protocol=ELBO-based proxy, Model parameters=101M2026.05 | 35.35 | |
| MD4Zero-shot=true, Protocol=ELBO-based proxy2026.05 | 35.9 | |
| RADDZero-shot=true, Protocol=ELBO-based proxy2026.05 | 37.98 | |
| SEDDZero-shot=true, Protocol=ELBO-based proxy2026.05 | 40.62 | |
| BLIND-VALM+Model=BLIND-VALM+2024.09 | 40.95 | |
| AO-GPTZero-shot=true2026.05 | 41.51 | |
| GPT-2Zero-shot=true2026.05 | 41.6 | |
| GPT2 - MModel=GPT2 - M, Evaluation Protocol=Zero-Shot, Bits per Token=7864322025.05 | 43.22 | |
| BLIND-VALMModel=BLIND-VALM2024.09 | 43.46 | |
| VALMModel=VALM2024.09 | 43.68 | |
| PLAIDZero-shot=true, Protocol=ELBO-based proxy2026.05 | 50.86 | |
| GPT2 - SModel=GPT2 - S, Evaluation Protocol=Zero-Shot, Bits per Token=2949122025.05 | 58.91 | |
| ASTRAModel=GPT2 - M, #Groups=32, Evaluation Protocol=Zero-Shot, Bits per Token=7680, Compression Ratio=102.42025.05 | 62.29 | |
| D3PMZero-shot=true, Protocol=ELBO-based proxy2026.05 | 75.16 | |
| ASTRAModel=GPT2 - M, #Groups=16, Evaluation Protocol=Zero-Shot, Bits per Token=3840, Compression Ratio=204.82025.05 | 75.29 | |
| ASTRAModel=GPT2 - S, #Groups=32, Evaluation Protocol=Zero-Shot, Bits per Token=3840, Compression Ratio=76.82025.05 | 76.24 | |
| ASTRAModel=GPT2 - S, #Groups=16, Evaluation Protocol=Zero-Shot, Bits per Token=1920, Compression Ratio=153.62025.05 | 94.8 | |
| ASTRAModel=GPT2 - M, #Groups=1, Evaluation Protocol=Zero-Shot, Bits per Token=240, Compression Ratio=3276.82025.05 | 96.99 | |
| ASTRAModel=GPT2 - S, #Groups=1, Evaluation Protocol=Zero-Shot, Bits per Token=120, Compression Ratio=2457.62025.05 | 120.7 |