List your company
ResearchFormerly WizWand

Language Modeling on GitHub (test)

2.42PerplexityKERPLE-log
2.07284.41646.769.1036May 20, 2022Jul 27, 2022Oct 4, 2022Dec 12, 2022Feb 18, 2023Apr 28, 2023Jul 6, 2023
Updated 8mo ago

Evaluation results

MethodLinks
KERPLE-logModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=81922022.052.42
KERPLE-logModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=20482022.052.44
KERPLE-logModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=81922022.052.44
KERPLE-logModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=40962022.052.46
T5 biasModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=20482022.052.47
KERPLE-logModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=163842022.052.48
T5 biasModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=40962022.052.49
KERPLE-logModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=40962022.052.53
T5 biasModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=40962022.052.54
T5 biasModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=81922022.052.57
ALiBiModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=81922022.052.57
T5 biasModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=81922022.052.57
ALiBiModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=20482022.052.58
KERPLE-logModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=10242022.052.6
KERPLE-logModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=163842022.052.6
ALiBiModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=163842022.052.61
T5L_ex=1024, Training length=512, Embedding Type=learnable2022.122.61
ALiBiModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=10242022.052.62
T5 biasModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=10242022.052.64
ALiBiModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=40962022.052.65
KERPLEL_ex=2048, Training length=512, Embedding Type=learnable2022.122.65
T5L_ex=2048, Training length=512, Embedding Type=learnable2022.122.65
KERPLEL_ex=1024, Training length=512, Embedding Type=learnable2022.122.67
T5 biasModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=20482022.052.68
SandwichL_ex=2048, Training length=512, Embedding Type=parameter-free2022.122.69
KERPLE-logModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=20482022.052.7
SmoothedL_ex=1024, Training length=512, Embedding Type=parameter-free2022.122.7
KERPLEL_ex=4096, Training length=512, Embedding Type=learnable2022.122.7
KERPLE-logExtrapolation Length=81922022.052.71
SandwichL_ex=1024, Training length=512, Embedding Type=parameter-free2022.122.71
SandwichL_ex=4096, Training length=512, Embedding Type=parameter-free2022.122.73
KERPLE-logExtrapolation Length=40962022.052.74
ALiBiModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=81922022.052.74
SmoothedL_ex=2048, Training length=512, Embedding Type=parameter-free2022.122.74
KERPLE-logExtrapolation Length=163842022.052.75
KERPLEL_ex=8192, Training length=512, Embedding Type=learnable2022.122.75
KERPLE-powerExtrapolation Length=81922022.052.76
KERPLE-powerExtrapolation Length=163842022.052.76
T5L_ex=512, Training length=512, Embedding Type=learnable2022.122.76
ALiBiModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=40962022.052.77
T5Extrapolation Length=40962022.052.78
SmoothedL_ex=4096, Training length=512, Embedding Type=parameter-free2022.122.78
KERPLE-powerExtrapolation Length=40962022.052.79
SandwichL_ex=8192, Training length=512, Embedding Type=parameter-free2022.122.79
ALiBiModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=163842022.052.8
T5 biasModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=10242022.052.81
KERPLEL_ex=512, Training length=512, Embedding Type=learnable2022.122.81
ALiBiModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=20482022.052.82
RotaryL_ex=512, Training length=512, Embedding Type=parameter-free2022.122.82
KERPLE-logModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=10242022.052.83
SmoothedL_ex=8192, Training length=512, Embedding Type=parameter-free2022.122.83
ALiBiL_ex=512, Training length=512, Embedding Type=parameter-free2022.122.83
T5Extrapolation Length=20482022.052.84
ALiBiModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=10242022.052.84
KERPLE-logExtrapolation Length=20482022.052.86
KERPLE-logModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=5122022.052.88
ALiBiModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=5122022.052.88
SandwichL_ex=512, Training length=512, Embedding Type=parameter-free2022.122.88
SmoothedL_ex=512, Training length=512, Embedding Type=parameter-free2022.122.88
KERPLE-powerExtrapolation Length=20482022.052.9
T5L_ex=4096, Training length=512, Embedding Type=learnable2022.122.91
T5 biasModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=5122022.052.93
T5Extrapolation Length=81922022.052.95
ALiBiL_ex=1024, Training length=512, Embedding Type=parameter-free2022.122.97
ALiBiL_ex=2048, Training length=512, Embedding Type=parameter-free2022.123.01
ALiBiL_ex=4096, Training length=512, Embedding Type=parameter-free2022.123.01
ALiBiL_ex=8192, Training length=512, Embedding Type=parameter-free2022.123.01
ALiBiExtrapolation Length=163842022.053.02
T5Extrapolation Length=10242022.053.02
KERPLE-logExtrapolation Length=10242022.053.04
ALiBiExtrapolation Length=40962022.053.04
ALiBiExtrapolation Length=81922022.053.04
KERPLE-powerExtrapolation Length=10242022.053.07
T5 biasModel size=162M, Train length=1024, Train steps=50k, Extrapolation length=163842022.053.1
ALiBiExtrapolation Length=20482022.053.13
ALiBiExtrapolation Length=10242022.053.15
T5 biasModel size=1.3B, Train length=512, Train steps=150k, Extrapolation length=163842022.053.16
T5Extrapolation Length=163842022.053.35
T5Extrapolation Length=5122022.053.38
KERPLE-logExtrapolation Length=5122022.053.4
KERPLE-powerExtrapolation Length=5122022.053.42
ALiBiExtrapolation Length=5122022.053.42
RotaryExtrapolation Length=5122022.053.44
T5L_ex=8192, Training length=512, Embedding Type=learnable2022.123.68
RotaryExtrapolation Length=10242022.053.86
RotaryL_ex=1024, Training length=512, Embedding Type=parameter-free2022.123.86
SinusoidalExtrapolation Length=5122022.054
SinusoidalL_ex=512, Training length=512, Embedding Type=parameter-free2022.124
FOTContext Length=64K, Zero-shot=true2023.075.32
FOTContext Length=16K, Zero-shot=true2023.075.43
FOTContext Length=4K, Zero-shot=true2023.075.88
RotaryExtrapolation Length=20482022.055.94
RotaryL_ex=2048, Training length=512, Embedding Type=parameter-free2022.125.94
FOTContext Length=2K, Zero-shot=true2023.076.72
Transformer-XLContext Length=2K, Zero-shot=true2023.076.85
Memorizing TransformerContext Length=64K, Zero-shot=true2023.077.26
Memorizing TransformerContext Length=16K, Zero-shot=true2023.077.27
Memorizing TransformerContext Length=4K, Zero-shot=true2023.077.55
Memorizing TransformerContext Length=2K, Zero-shot=true2023.078.1
RotaryExtrapolation Length=40962022.0511.1