Language Modeling on Book3
6.9PerplexityCREAM
Evaluation Results
| Method | Links | |
|---|---|---|
| CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=64K2024.06 | 6.9 | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=96K2024.06 | 7 | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=128K2024.06 | 7 | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=64K2024.06 | 7 | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=192K2024.06 | 7.1 | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=96K2024.06 | 7.1 | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=128K2024.06 | 7.1 | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=192K2024.06 | 7.1 | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=64K2024.06 | 7.1 | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=64K2024.06 | 7.2 | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=96K2024.06 | 7.2 | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=128K2024.06 | 7.2 | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=96K2024.06 | 7.3 | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=128K2024.06 | 7.3 | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=192K2024.06 | 7.3 | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=192K2024.06 | 7.3 | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=256K2024.06 | 7.3 | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=256K2024.06 | 7.4 | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=64K2024.06 | 7.6 | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=96K2024.06 | 7.7 | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=128K2024.06 | 7.8 | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=192K2024.06 | 7.8 | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=64K2024.06 | 10.2 | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=96K2024.06 | 10.3 | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=128K2024.06 | 10.5 | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=192K2024.06 | 10.7 | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=256K2024.06 | 10.8 | |
| PoSEBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=128K, Evaluation Context Length=64K2024.06 | 12.3 | |
| PoSEBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=128K, Evaluation Context Length=96K2024.06 | 13.07 | |
| PoSEBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=128K, Evaluation Context Length=128K2024.06 | 13.81 | |
| PoSEBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=128K, Evaluation Context Length=64K2024.06 | 16.04 | |
| PoSEBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=128K, Evaluation Context Length=96K2024.06 | 31.42 | |
| PoSEBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=128K, Evaluation Context Length=128K2024.06 | 37 | |
| PoSEBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=128K, Evaluation Context Length=64K2024.06 | 43.62 | |
| PoSEBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=128K, Evaluation Context Length=96K2024.06 | 57.08 | |
| PoSEBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=128K, Evaluation Context Length=128K2024.06 | 70.87 |