List your company

Language Modeling on RedPajama LLaMA pretraining corpus (evaluation slice)

0.62Perplexity (bits/byte)No intervention
0.51441.22721.942.6528Jun 6, 2023

Evaluation results

MethodLinks
No interventionModel Family=LLaMA, Model Size=30B2023.060.62
Random erasureModel Family=LLaMA, Model Size=30B2023.060.62
No interventionModel Family=LLaMA, Model Size=13B2023.060.66
Random erasureModel Family=LLaMA, Model Size=13B2023.060.66
No interventionModel Family=LLaMA, Model Size=7B2023.060.69
Random erasureModel Family=LLaMA, Model Size=7B2023.060.69
LEACEModel Family=LLaMA, Model Size=7B2023.061.73
LEACEModel Family=LLaMA, Model Size=13B2023.061.84
LEACEModel Family=LLaMA, Model Size=30B2023.061.96
unigram entropyModel Family=LLaMA, Model Size=7B2023.062.9
unigram entropyModel Family=LLaMA, Model Size=13B2023.062.9
unigram entropyModel Family=LLaMA, Model Size=30B2023.062.9
SALModel Family=LLaMA, Model Size=30B2023.063.16
SALModel Family=LLaMA, Model Size=7B2023.063.24
SALModel Family=LLaMA, Model Size=13B2023.063.26