Next Token Prediction on C4
21.1OOD PerplexityConcepts
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| ConceptsModel=Llama 3.1 8B, λ=1.02026.03 | 21.1 | 39.38 | |
| ConceptsModel=Llama 3.1 8B, λ=0.752026.03 | 21.51 | 39.12 | |
| ConceptsModel=Llama 3.1 8B, λ=0.52026.03 | 22.21 | 38.29 | |
| ConceptsModel=Llama 3.1 8B, λ=0.252026.03 | 23.45 | 37.1 | |
| ConceptsModel=Llama 3.2 3B, λ=1.02026.03 | 25.08 | 37.34 | |
| BaselineModel=Llama 3.1 8B, λ=Base2026.03 | 25.1 | 36.01 | |
| BaselineModel=Llama 3.1 8B, λ=0.02026.03 | 25.8 | 35.52 | |
| ConceptsModel=Llama 3.2 3B, λ=0.752026.03 | 25.84 | 36.9 | |
| ConceptsModel=Llama 3.2 3B, λ=0.52026.03 | 26.9 | 36.17 | |
| ConceptsModel=Llama 3.2 3B, λ=0.252026.03 | 28.57 | 35.13 | |
| BaselineModel=Llama 3.2 3B, λ=Base2026.03 | 31 | 33.91 | |
| BaselineModel=Llama 3.2 3B, λ=0.02026.03 | 31.53 | 33.62 | |
| ConceptsModel=Llama 3.2 1B, λ=1.02026.03 | 32.91 | 34.08 | |
| ConceptsModel=Llama 3.2 1B, λ=0.752026.03 | 33.67 | 33.58 | |
| ConceptsModel=Llama 3.2 1B, λ=0.52026.03 | 35.13 | 32.75 | |
| ConceptsModel=Llama 3.2 1B, λ=0.252026.03 | 37.46 | 31.83 | |
| BaselineModel=Llama 3.2 1B, λ=Base2026.03 | 40.61 | 30.61 | |
| NoiseModel=Llama 3.1 8B, λ=0.252026.03 | 41.4 | 33.02 | |
| BaselineModel=Llama 3.2 1B, λ=0.02026.03 | 41.54 | 30.24 | |
| NoiseModel=Llama 3.2 3B, λ=0.252026.03 | 49.7 | 30.46 | |
| NoiseModel=Llama 3.2 1B, λ=0.252026.03 | 66.2 | 26.73 | |
| NoiseModel=Llama 3.1 8B, λ=0.52026.03 | 79.74 | 29.67 | |
| NoiseModel=Llama 3.2 3B, λ=0.52026.03 | 100.2 | 25.95 | |
| NoiseModel=Llama 3.2 1B, λ=0.52026.03 | 137.04 | 21.88 | |
| NoiseModel=Llama 3.1 8B, λ=0.752026.03 | 217.33 | 22.93 | |
| NoiseModel=Llama 3.2 3B, λ=0.752026.03 | 273.07 | 19.02 | |
| NoiseModel=Llama 3.2 1B, λ=0.752026.03 | 399.66 | 14.52 | |
| NoiseModel=Llama 3.2 3B, λ=1.02026.03 | 22,826.87 | 0.55 | |
| NoiseModel=Llama 3.2 1B, λ=1.02026.03 | 23,964.93 | 0.39 | |
| NoiseModel=Llama 3.1 8B, λ=1.02026.03 | 29,168.22 | 0.37 |