List your company
ResearchFormerly WizWand

Language Modeling on arXiv

2.46PerplexitySHAREDLLM
-17.9796119.9877257.955395.9223Nov 15, 2022Jun 23, 2023Jan 29, 2024Sep 5, 2024Apr 13, 2025Nov 19, 2025Jun 28, 2026
Updated 4mo ago

Evaluation results

MethodLinks
SHAREDLLMContext Length=32K2026.032.46
CEPEContext Length=32K2026.032.51
YaRNContext Length=32K2026.032.58
LLaMA-3.1Context Length=32K2026.032.63
PIContext Length=32K2026.032.77
SHAREDLLMContext Length=128K2026.032.91
LLaMA-2-32KContext Length=32K2026.032.96
CEPEContext Length=128K2026.032.97
SHAREDLLMContext Length=8K2026.032.97
SHAREDLLMContext Length=4K2026.032.99
CEPEContext Length=8K2026.033.02
CEPEContext Length=4K2026.033.03
YaRNContext Length=8K2026.033.09
LLaMA-3.1Context Length=128K2026.033.12
LLaMA-3.1Context Length=4K2026.033.17
PIContext Length=8K2026.033.21
LLaMA-3.1Context Length=8K2026.033.26
LLaMA-2-32KContext Length=8K2026.033.34
YaRNContext Length=4K2026.033.35
PIContext Length=4K2026.033.49
LLaMA-2-32KContext Length=4K2026.033.58
(w+a)kNN-LMa (rescore)LMa=domain-adapted GPT-2, datastore=combined, rescore=true, context_representation=LN22022.1117.47
(a)kNN-LMa (rescore)LMa=domain-adapted GPT-2, datastore=adaptation, rescore=true, context_representation=LN22022.1117.49
(a)kNN-LMaLMa=domain-adapted GPT-2, datastore=adaptation, context_representation=LN22022.1117.81
(w+a)kNN-LMaLMa=domain-adapted GPT-2, datastore=combined, context_representation=LN22022.1117.85
ARMzero-shot=true, number of parameters=1B, training tokens=300B2026.0118.15
CARDzero-shot=true, number of parameters=1B, training tokens=300B2026.0120.34
MDLMzero-shot=true, number of parameters=1B, training tokens=300B2026.0123.58
(w)kNN-LMa (rescore)LMa=domain-adapted GPT-2, datastore=pretraining, rescore=true, context_representation=LN22022.1124.19
(a)kNN-LMLM=GPT-2, datastore=adaptation, context_representation=original2022.1124.38
(w)kNN-LMaLMa=domain-adapted GPT-2, datastore=pretraining, context_representation=LN22022.1124.42
LMa (only)LMa=domain-adapted GPT-22022.1124.97
ABDZero-shot=true, Block Size (BS)=42026.0637.11
XDLMSetting=Zero-shot2026.0237.232
MDLMModel Category=Diffusion (Absorbing-state), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.0437.37
MDLMSetting=Zero-shot2026.0237.457
SCMDMAddZero-shot=true, Training tokens=262B + 3.25B, Pre-training dataset=OWT2026.0437.67
SCMDMConcatZero-shot=true, Training tokens=262B + 3.25B, Pre-training dataset=OWT2026.0437.75
MDLMZero-shot=true2026.0637.89
LangFlowModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.0438.47
SEDD AbsorbModel Category=Diffusion (Absorbing-state), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.0438.48
GIDDSetting=Zero-shot2026.0239.019
ABDZero-shot=true, Block Size (BS)=12026.0639.11
MDLMZero-shot=true, Training tokens=262B + 3.25B, Pre-training dataset=OWT2026.0439.14
BD3LMZero-shot=true, Block Size (BS)=42026.0639.2
SEDDZero-shot=true2026.0640.03
DuoModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.0440.39
ARZero-shot=true2026.0641.22
Autoregressive TransformerModel Category=Autoregressive, Zero-shot=true, Training Data=OWT, Training Steps=1M2026.0441.73
UDLMSetting=Zero-shot2026.0242.671
UDLMModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.0444.08
BD3LMzero-shot=true, number of parameters=1B, training tokens=300B2026.0144.6
SEDD UniformModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.0450.86
(w)kNN-LMLM=GPT-2, datastore=pretraining, context_representation=LN22022.1151.89
(w)kNN-LMLM=GPT-2, datastore=pretraining, context_representation=original2022.1153.03
LM (only)LM=GPT-22022.1156.83
DID-FSize=Medium, Zero-shot=true, Alignment=FLOPs-aligned2026.0361.77
DID-SSize=Medium, Zero-shot=true, Alignment=Steps-aligned2026.0363.95
RADDSize=Medium, Zero-shot=true2026.0366.28
DID-FSize=Small, Zero-shot=true, Alignment=FLOPs-aligned2026.0378.38
DID-SSize=Small, Zero-shot=true, Alignment=Steps-aligned2026.0382.41
RADDSize=Small, Zero-shot=true2026.0385.95
(a)kNN (only)datastore=adaptation2022.1187.58
(w)kNN (only)datastore=pretraining2022.11513.45