Next Token Prediction on OpenWebText (PPL, Accuracy)
18.68PPLConcepts
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| ConceptsModel=Llama 3.1 8B, lambda=0.752026.03 | 18.68 | 40.1 | |
| ConceptsModel=Llama 3.1 8B, lambda=1.02026.03 | 18.68 | 40.11 | |
| ConceptsModel=Llama 3.1 8B, lambda=0.52026.03 | 19.19 | 39.24 | |
| ConceptsModel=Llama 3.1 8B, lambda=0.252026.03 | 20.26 | 38.01 | |
| BaselineModel=Llama 3.1 8B, lambda=Base2026.03 | 22.26 | 36.4 | |
| BaselineModel=Llama 3.1 8B, lambda=0.02026.03 | 22.42 | 36.26 | |
| ConceptsModel=Llama 3.2 3B, lambda=1.02026.03 | 22.8 | 37.63 | |
| ConceptsModel=Llama 3.2 3B, lambda=0.752026.03 | 23.22 | 37.23 | |
| ConceptsModel=Llama 3.2 3B, lambda=0.52026.03 | 24.12 | 36.26 | |
| ConceptsModel=Llama 3.2 3B, lambda=0.252026.03 | 25.62 | 35.14 | |
| BaselineModel=Llama 3.2 3B, lambda=0.02026.03 | 28.39 | 33.41 | |
| BaselineModel=Llama 3.2 3B, lambda=Base2026.03 | 28.43 | 33.47 | |
| ConceptsModel=Llama 3.2 1B, lambda=1.02026.03 | 30.47 | 34.41 | |
| ConceptsModel=Llama 3.2 1B, lambda=0.752026.03 | 31.28 | 33.79 | |
| ConceptsModel=Llama 3.2 1B, lambda=0.52026.03 | 32.65 | 32.78 | |
| ConceptsModel=Llama 3.2 1B, lambda=0.252026.03 | 34.84 | 31.64 | |
| BaselineModel=Llama 3.2 1B, lambda=Base2026.03 | 38.56 | 30.11 | |
| BaselineModel=Llama 3.2 1B, lambda=0.02026.03 | 38.71 | 30.05 | |
| NoiseModel=Llama 3.1 8B, lambda=0.252026.03 | 39.49 | 33.37 | |
| NoiseModel=Llama 3.2 3B, lambda=0.252026.03 | 47.92 | 30.22 | |
| NoiseModel=Llama 3.2 1B, lambda=0.252026.03 | 62.75 | 26.51 | |
| NoiseModel=Llama 3.1 8B, lambda=0.52026.03 | 78.4 | 30.42 | |
| NoiseModel=Llama 3.2 3B, lambda=0.52026.03 | 99.13 | 26.32 | |
| NoiseModel=Llama 3.2 1B, lambda=0.52026.03 | 130.08 | 22.38 | |
| NoiseModel=Llama 3.1 8B, lambda=0.752026.03 | 203.15 | 23.68 | |
| NoiseModel=Llama 3.2 3B, lambda=0.752026.03 | 264.21 | 19.2 | |
| NoiseModel=Llama 3.2 1B, lambda=0.752026.03 | 374.05 | 15.22 | |
| NoiseModel=Llama 3.2 1B, lambda=1.02026.03 | 28,643.16 | 0.23 | |
| NoiseModel=Llama 3.2 3B, lambda=1.02026.03 | 28,898.47 | 0.23 | |
| NoiseModel=Llama 3.1 8B, lambda=1.02026.03 | 35,501.71 | 0.22 |