Next Token Prediction on C4 (held-out)
21.5Perplexity (PPL)Concepts
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| ConceptsModel=Llama 3.1 8B, lambda=0.752026.03 | 21.5 | 39.11 | |
| ConceptsModel=Llama 3.1 8B, lambda=0.52026.03 | 21.88 | 38.39 | |
| ConceptsModel=Llama 3.1 8B, lambda=1.02026.03 | 22.03 | 39.02 | |
| ConceptsModel=Llama 3.1 8B, lambda=0.252026.03 | 22.95 | 37.5 | |
| BaselineModel=Llama 3.1 8B, lambda=Base2026.03 | 25.1 | 36.01 | |
| BaselineModel=Llama 3.1 8B, lambda=0.02026.03 | 25.27 | 35.86 | |
| ConceptsModel=Llama 3.2 3B, lambda=0.752026.03 | 25.7 | 37.08 | |
| ConceptsModel=Llama 3.2 3B, lambda=1.02026.03 | 25.78 | 37.17 | |
| ConceptsModel=Llama 3.2 3B, lambda=0.52026.03 | 26.46 | 36.48 | |
| ConceptsModel=Llama 3.2 3B, lambda=0.252026.03 | 27.93 | 35.39 | |
| BaselineModel=Llama 3.2 3B, lambda=Base2026.03 | 30.1 | 33.91 | |
| BaselineModel=Llama 3.2 3B, lambda=0.02026.03 | 30.91 | 33.72 | |
| ConceptsModel=Llama 3.2 1B, lambda=1.02026.03 | 33.11 | 34.19 | |
| ConceptsModel=Llama 3.2 1B, lambda=0.752026.03 | 33.2 | 33.88 | |
| ConceptsModel=Llama 3.2 1B, lambda=0.52026.03 | 34.38 | 33.22 | |
| ConceptsModel=Llama 3.2 1B, lambda=0.252026.03 | 36.48 | 32.24 | |
| BaselineModel=Llama 3.2 1B, lambda=0.02026.03 | 40.49 | 30.57 | |
| BaselineModel=Llama 3.2 1B, lambda=Base2026.03 | 40.61 | 30.61 | |
| NoiseModel=Llama 3.1 8B, lambda=0.252026.03 | 43.65 | 32.86 | |
| NoiseModel=Llama 3.2 3B, lambda=0.252026.03 | 52.71 | 30.22 | |
| NoiseModel=Llama 3.2 1B, lambda=0.252026.03 | 67.82 | 26.39 | |
| NoiseModel=Llama 3.1 8B, lambda=0.52026.03 | 85.58 | 29.5 | |
| NoiseModel=Llama 3.2 3B, lambda=0.52026.03 | 108.5 | 25.89 | |
| NoiseModel=Llama 3.2 1B, lambda=0.52026.03 | 139.55 | 21.59 | |
| NoiseModel=Llama 3.1 8B, lambda=0.752026.03 | 223.01 | 22.17 | |
| NoiseModel=Llama 3.2 3B, lambda=0.752026.03 | 299.11 | 17.71 | |
| NoiseModel=Llama 3.2 1B, lambda=0.752026.03 | 401.52 | 14.05 | |
| NoiseModel=Llama 3.2 3B, lambda=1.02026.03 | 19,518.8 | 0.28 | |
| NoiseModel=Llama 3.2 1B, lambda=1.02026.03 | 19,670.63 | 0.25 | |
| NoiseModel=Llama 3.1 8B, lambda=1.02026.03 | 23,849.59 | 0.24 |