Language Modeling on RedPajama LLaMA pretraining corpus (evaluation slice)
0.62Perplexity (bits/byte)No intervention
Evaluation Results
| Method | Links | |
|---|---|---|
| No interventionModel Family=LLaMA, Model Size=30B2023.06 | 0.62 | |
| Random erasureModel Family=LLaMA, Model Size=30B2023.06 | 0.62 | |
| No interventionModel Family=LLaMA, Model Size=13B2023.06 | 0.66 | |
| Random erasureModel Family=LLaMA, Model Size=13B2023.06 | 0.66 | |
| No interventionModel Family=LLaMA, Model Size=7B2023.06 | 0.69 | |
| Random erasureModel Family=LLaMA, Model Size=7B2023.06 | 0.69 | |
| LEACEModel Family=LLaMA, Model Size=7B2023.06 | 1.73 | |
| LEACEModel Family=LLaMA, Model Size=13B2023.06 | 1.84 | |
| LEACEModel Family=LLaMA, Model Size=30B2023.06 | 1.96 | |
| unigram entropyModel Family=LLaMA, Model Size=7B2023.06 | 2.9 | |
| unigram entropyModel Family=LLaMA, Model Size=13B2023.06 | 2.9 | |
| unigram entropyModel Family=LLaMA, Model Size=30B2023.06 | 2.9 | |
| SALModel Family=LLaMA, Model Size=30B2023.06 | 3.16 | |
| SALModel Family=LLaMA, Model Size=7B2023.06 | 3.24 | |
| SALModel Family=LLaMA, Model Size=13B2023.06 | 3.26 |