Language Modeling on Common Knowledge (CK)
64.5Avg-CK ScorePretraining with Memories
Evaluation Results
| Method | Links | |
|---|---|---|
| Pretraining with MemoriesAnchor model=1.4B, Initialization=C1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(768,256,16,0), Bank size=21.1B, Fetch size=153M, Retrieval Mode=Fetched2025.09 | 64.5 | |
| Pretraining with MemoriesAnchor model=1.4B, Initialization=C1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(768,256,16,0), Bank size=21.1B, Fetch size=153M, Retrieval Mode=Generic2025.09 | 64.4 | |
| Baseline Anchor ModelAnchor model=1.4B, Initialization=Scratch, Seen tokens=1.1T, Cotrain anchor=n/a, Memory config=(0,0,0,0), Bank size=0, Fetch size=0, Retrieval Mode=Generic2025.09 | 61.2 | |
| Pretraining with MemoriesAnchor model=410M, Initialization=B1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(512,128,32,0), Bank size=12.7B, Fetch size=50M, Retrieval Mode=Fetched2025.09 | 56.1 | |
| Pretraining with MemoriesAnchor model=410M, Initialization=B1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(512,128,32,0), Bank size=12.7B, Fetch size=50M, Retrieval Mode=Generic2025.09 | 55.5 | |
| Baseline Anchor ModelAnchor model=410M, Initialization=Scratch, Seen tokens=2.2T, Cotrain anchor=n/a, Memory config=(0,0,0,0), Bank size=0, Fetch size=0, Retrieval Mode=Generic2025.09 | 53.2 | |
| Baseline Anchor ModelAnchor model=410M, Initialization=Scratch, Seen tokens=1.1T, Cotrain anchor=n/a, Memory config=(0,0,0,0), Bank size=0, Fetch size=0, Retrieval Mode=Generic2025.09 | 52.3 | |
| Pretraining with MemoriesAnchor model=160M, Initialization=A1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(256,64,16,0), Bank size=4.6B, Fetch size=18M, Retrieval Mode=Fetched2025.09 | 48.7 | |
| Pretraining with MemoriesAnchor model=160M, Initialization=A1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(256,64,16,0), Bank size=4.6B, Fetch size=18M, Retrieval Mode=Generic2025.09 | 47.9 | |
| Pretraining with MemoriesAnchor model=160M, Initialization=A1, Seen tokens=+1.1T, Cotrain anchor=No, Memory config=(256,64,16,0), Bank size=4.6B, Fetch size=18M, Retrieval Mode=Fetched2025.09 | 47.4 | |
| Pretraining with MemoriesAnchor model=160M, Initialization=A1, Seen tokens=+1.1T, Cotrain anchor=No, Memory config=(256,64,16,0), Bank size=4.6B, Fetch size=18M, Retrieval Mode=Generic2025.09 | 46.6 | |
| Baseline Anchor ModelAnchor model=160M, Initialization=Scratch, Seen tokens=2.2T, Cotrain anchor=n/a, Memory config=(0,0,0,0), Bank size=0, Fetch size=0, Retrieval Mode=Generic2025.09 | 46.6 | |
| Baseline Anchor ModelAnchor model=160M, Initialization=Scratch, Seen tokens=1.1T, Cotrain anchor=n/a, Memory config=(0,0,0,0), Bank size=0, Fetch size=0, Retrieval Mode=Generic2025.09 | 45.9 |