List your company
ResearchFormerly WizWand

Language Modeling on Book3

6.9PerplexityCREAM
4.341221.613138.88556.1569Jun 11, 2024
Updated 8mo ago

Evaluation results

MethodLinks
CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=64K2024.066.9
CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=96K2024.067
CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=128K2024.067
CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=64K2024.067
CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=192K2024.067.1
CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=96K2024.067.1
CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=128K2024.067.1
CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=192K2024.067.1
CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=64K2024.067.1
CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=64K2024.067.2
CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=96K2024.067.2
CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=128K2024.067.2
CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=96K2024.067.3
CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=128K2024.067.3
CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=192K2024.067.3
CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=192K2024.067.3
CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=256K2024.067.3
CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=256K2024.067.4
CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=64K2024.067.6
CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=96K2024.067.7
CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=128K2024.067.8
CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=192K2024.067.8
CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=64K2024.0610.2
CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=96K2024.0610.3
CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=128K2024.0610.5
CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=192K2024.0610.7
CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=256K2024.0610.8
PoSEBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=128K, Evaluation Context Length=64K2024.0612.3
PoSEBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=128K, Evaluation Context Length=96K2024.0613.07
PoSEBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=128K, Evaluation Context Length=128K2024.0613.81
PoSEBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=128K, Evaluation Context Length=64K2024.0616.04
PoSEBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=128K, Evaluation Context Length=96K2024.0631.42
PoSEBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=128K, Evaluation Context Length=128K2024.0637
PoSEBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=128K, Evaluation Context Length=64K2024.0643.62
PoSEBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=128K, Evaluation Context Length=96K2024.0657.08
PoSEBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=128K, Evaluation Context Length=128K2024.0670.87