List your company
ResearchFormerly WizWand

Language Modeling on Specialized Knowledge (SK)

54.9Avg-SK ScorePretraining with Memories
32.95638.65344.3550.047Sep 29, 2025
Updated 7mo ago

Evaluation results

MethodLinks
Pretraining with MemoriesAnchor model=1.4B, Initialization=C1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(768,256,16,0), Bank size=21.1B, Fetch size=153M, Retrieval Mode=Fetched2025.0954.9
Pretraining with MemoriesAnchor model=1.4B, Initialization=C1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(768,256,16,0), Bank size=21.1B, Fetch size=153M, Retrieval Mode=Generic2025.0951.3
Baseline Anchor ModelAnchor model=1.4B, Initialization=Scratch, Seen tokens=1.1T, Cotrain anchor=n/a, Memory config=(0,0,0,0), Bank size=0, Fetch size=0, Retrieval Mode=Generic2025.0949.7
Pretraining with MemoriesAnchor model=410M, Initialization=B1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(512,128,32,0), Bank size=12.7B, Fetch size=50M, Retrieval Mode=Fetched2025.0945.9
Pretraining with MemoriesAnchor model=410M, Initialization=B1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(512,128,32,0), Bank size=12.7B, Fetch size=50M, Retrieval Mode=Generic2025.0941.8
Baseline Anchor ModelAnchor model=410M, Initialization=Scratch, Seen tokens=2.2T, Cotrain anchor=n/a, Memory config=(0,0,0,0), Bank size=0, Fetch size=0, Retrieval Mode=Generic2025.0941.1
Baseline Anchor ModelAnchor model=410M, Initialization=Scratch, Seen tokens=1.1T, Cotrain anchor=n/a, Memory config=(0,0,0,0), Bank size=0, Fetch size=0, Retrieval Mode=Generic2025.0940.9
Pretraining with MemoriesAnchor model=160M, Initialization=A1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(256,64,16,0), Bank size=4.6B, Fetch size=18M, Retrieval Mode=Fetched2025.0940.3
Pretraining with MemoriesAnchor model=160M, Initialization=A1, Seen tokens=+1.1T, Cotrain anchor=No, Memory config=(256,64,16,0), Bank size=4.6B, Fetch size=18M, Retrieval Mode=Fetched2025.0939.2
Pretraining with MemoriesAnchor model=160M, Initialization=A1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(256,64,16,0), Bank size=4.6B, Fetch size=18M, Retrieval Mode=Generic2025.0935.7
Pretraining with MemoriesAnchor model=160M, Initialization=A1, Seen tokens=+1.1T, Cotrain anchor=No, Memory config=(256,64,16,0), Bank size=4.6B, Fetch size=18M, Retrieval Mode=Generic2025.0934.7
Baseline Anchor ModelAnchor model=160M, Initialization=Scratch, Seen tokens=1.1T, Cotrain anchor=n/a, Memory config=(0,0,0,0), Bank size=0, Fetch size=0, Retrieval Mode=Generic2025.0934.1
Baseline Anchor ModelAnchor model=160M, Initialization=Scratch, Seen tokens=2.2T, Cotrain anchor=n/a, Memory config=(0,0,0,0), Bank size=0, Fetch size=0, Retrieval Mode=Generic2025.0933.8