Language Modeling on ReRoPE (eval)
1.7622Overall LossHiROPE
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| HiROPEParameters=1.3B, Pre-training Length=4096, Model Variant=ShearedLLaMA2024.03 | 1.7622 | 5.8252 | 61.13 | 1.8332 | 6.2536 | 59.63 | |
| Self-ExtendParameters=1.3B, Pre-training Length=4096, Model Variant=ShearedLLaMA2024.03 | 1.7662 | 5.8486 | 61.05 | 1.8348 | 6.264 | 59.66 | |
| HiROPEParameters=1.1B, Pre-training Length=2048, Model Variant=TinyLlama2024.03 | 1.7829 | 5.9474 | 61.02 | 1.8717 | 6.4991 | 59.13 | |
| Self-ExtendParameters=1.1B, Pre-training Length=2048, Model Variant=TinyLlama2024.03 | 1.7904 | 5.9916 | 60.89 | 1.8749 | 6.5201 | 59.08 | |
| ReROPEParameters=1.3B, Pre-training Length=4096, Model Variant=ShearedLLaMA2024.03 | 1.7952 | 6.0205 | 60.5 | 1.8514 | 6.3687 | 59.32 | |
| ReROPEParameters=1.1B, Pre-training Length=2048, Model Variant=TinyLlama2024.03 | 1.8448 | 6.327 | 60.08 | 1.903 | 6.7062 | 58.67 | |
| NTKParameters=1.1B, Pre-training Length=2048, Model Variant=TinyLlama2024.03 | 2.1338 | 8.4468 | 55.53 | 2.2745 | 9.7234 | 53 | |
| NTKParameters=1.3B, Pre-training Length=4096, Model Variant=ShearedLLaMA2024.03 | 2.3596 | 10.5863 | 51.59 | 2.4056 | 11.0853 | 50.59 | |
| originParameters=1.3B, Pre-training Length=4096, Model Variant=ShearedLLaMA2024.03 | 8.5027 | — | 4.12 | 5.5237 | 250.5 | 27.4 | |
| originParameters=1.1B, Pre-training Length=2048, Model Variant=TinyLlama2024.03 | 9.3083 | — | 1.94 | 7.3486 | — | 14.66 |