List your company
ResearchFormerly WizWand

Language Modeling on WikiEn

10.2Perplexity (WikiEn)Pretraining with Memories
9.89611.9481416.052Sep 29, 2025
Updated 7mo ago

Evaluation results

MethodLinks
Pretraining with MemoriesAnchor model=1.4B, Initialization=C1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(768,256,16,0), Bank size=21.1B, Fetch size=153M, Retrieval Mode=Fetched2025.0910.2
Baseline Anchor ModelAnchor model=1.4B, Initialization=Scratch, Seen tokens=1.1T, Cotrain anchor=n/a, Memory config=(0,0,0,0), Bank size=0, Fetch size=0, Retrieval Mode=Generic2025.0910.8
Pretraining with MemoriesAnchor model=1.4B, Initialization=C1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(768,256,16,0), Bank size=21.1B, Fetch size=153M, Retrieval Mode=Generic2025.0911
Pretraining with MemoriesAnchor model=410M, Initialization=B1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(512,128,32,0), Bank size=12.7B, Fetch size=50M, Retrieval Mode=Fetched2025.0912.4
Pretraining with MemoriesAnchor model=410M, Initialization=B1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(512,128,32,0), Bank size=12.7B, Fetch size=50M, Retrieval Mode=Generic2025.0913.8
Baseline Anchor ModelAnchor model=410M, Initialization=Scratch, Seen tokens=2.2T, Cotrain anchor=n/a, Memory config=(0,0,0,0), Bank size=0, Fetch size=0, Retrieval Mode=Generic2025.0913.8
Baseline Anchor ModelAnchor model=410M, Initialization=Scratch, Seen tokens=1.1T, Cotrain anchor=n/a, Memory config=(0,0,0,0), Bank size=0, Fetch size=0, Retrieval Mode=Generic2025.0913.9
Pretraining with MemoriesAnchor model=160M, Initialization=A1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(256,64,16,0), Bank size=4.6B, Fetch size=18M, Retrieval Mode=Fetched2025.0914.2
Pretraining with MemoriesAnchor model=160M, Initialization=A1, Seen tokens=+1.1T, Cotrain anchor=No, Memory config=(256,64,16,0), Bank size=4.6B, Fetch size=18M, Retrieval Mode=Fetched2025.0915.2
Pretraining with MemoriesAnchor model=160M, Initialization=A1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(256,64,16,0), Bank size=4.6B, Fetch size=18M, Retrieval Mode=Generic2025.0916.7
Pretraining with MemoriesAnchor model=160M, Initialization=A1, Seen tokens=+1.1T, Cotrain anchor=No, Memory config=(256,64,16,0), Bank size=4.6B, Fetch size=18M, Retrieval Mode=Generic2025.0916.7
Baseline Anchor ModelAnchor model=160M, Initialization=Scratch, Seen tokens=1.1T, Cotrain anchor=n/a, Memory config=(0,0,0,0), Bank size=0, Fetch size=0, Retrieval Mode=Generic2025.0917.2
Baseline Anchor ModelAnchor model=160M, Initialization=Scratch, Seen tokens=2.2T, Cotrain anchor=n/a, Memory config=(0,0,0,0), Bank size=0, Fetch size=0, Retrieval Mode=Generic2025.0917.8