Next Token Prediction on OpenWebText (held-out)
18.53ID PPLConcepts
Evaluation results
| Method | Links | ||
|---|---|---|---|
| ConceptsModel=Llama 3.1 8B, λ=0.752026.03 | 18.53 | 40.13 | |
| ConceptsModel=Llama 3.1 8B, λ=0.52026.03 | 18.89 | 39.45 | |
| ConceptsModel=Llama 3.1 8B, λ=1.02026.03 | 18.92 | 39.69 | |
| ConceptsModel=Llama 3.1 8B, λ=0.252026.03 | 19.84 | 38.31 | |
| BaselineModel=Llama 3.1 8B, λ=0.02026.03 | 22.05 | 36.44 | |
| BaselineModel=Llama 3.1 8B, λ=Base2026.03 | 22.26 | 36.4 | |
| ConceptsModel=Llama 3.2 3B, λ=1.02026.03 | 22.77 | 37.64 | |
| ConceptsModel=Llama 3.2 3B, λ=0.752026.03 | 22.87 | 37.41 | |
| ConceptsModel=Llama 3.2 3B, λ=0.52026.03 | 23.63 | 36.58 | |
| ConceptsModel=Llama 3.2 3B, λ=0.252026.03 | 25.09 | 35.35 | |
| BaselineModel=Llama 3.2 3B, λ=0.02026.03 | 28.03 | 33.43 | |
| BaselineModel=Llama 3.2 3B, λ=Base2026.03 | 28.43 | 33.47 | |
| ConceptsModel=Llama 3.2 1B, λ=1.02026.03 | 30.3 | 34.28 | |
| ConceptsModel=Llama 3.2 1B, λ=0.752026.03 | 30.65 | 33.88 | |
| ConceptsModel=Llama 3.2 1B, λ=0.52026.03 | 31.85 | 33.13 | |
| ConceptsModel=Llama 3.2 1B, λ=0.252026.03 | 33.94 | 32 | |
| BaselineModel=Llama 3.2 1B, λ=0.02026.03 | 38.05 | 30.24 | |
| BaselineModel=Llama 3.2 1B, λ=Base2026.03 | 38.56 | 30.11 | |
| NoiseModel=Llama 3.1 8B, λ=0.252026.03 | 39.22 | 33.62 | |
| NoiseModel=Llama 3.2 3B, λ=0.252026.03 | 48.69 | 30.47 | |
| NoiseModel=Llama 3.2 1B, λ=0.252026.03 | 63.97 | 26.75 | |
| NoiseModel=Llama 3.1 8B, λ=0.52026.03 | 77.73 | 30.67 | |
| NoiseModel=Llama 3.2 3B, λ=0.52026.03 | 100.46 | 26.63 | |
| NoiseModel=Llama 3.2 1B, λ=0.52026.03 | 132.84 | 22.46 | |
| NoiseModel=Llama 3.1 8B, λ=0.752026.03 | 202.21 | 24.45 | |
| NoiseModel=Llama 3.2 3B, λ=0.752026.03 | 275.96 | 19.33 | |
| NoiseModel=Llama 3.2 1B, λ=0.752026.03 | 377.89 | 15.58 | |
| NoiseModel=Llama 3.2 1B, λ=1.02026.03 | 18,020.53 | 0.47 | |
| NoiseModel=Llama 3.2 3B, λ=1.02026.03 | 18,376.56 | 0.68 | |
| NoiseModel=Llama 3.1 8B, λ=1.02026.03 | 22,424.33 | 0.49 |