Language Modeling on Specialized Knowledge (SK)
54.9Avg-SK ScorePretraining with Memories
Evaluation Results
| Method | Links | |
|---|---|---|
| Pretraining with MemoriesAnchor model=1.4B, Initialization=C1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(768,256,16,0), Bank size=21.1B, Fetch size=153M, Retrieval Mode=Fetched2025.09 | 54.9 | |
| Pretraining with MemoriesAnchor model=1.4B, Initialization=C1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(768,256,16,0), Bank size=21.1B, Fetch size=153M, Retrieval Mode=Generic2025.09 | 51.3 | |
| Baseline Anchor ModelAnchor model=1.4B, Initialization=Scratch, Seen tokens=1.1T, Cotrain anchor=n/a, Memory config=(0,0,0,0), Bank size=0, Fetch size=0, Retrieval Mode=Generic2025.09 | 49.7 | |
| Pretraining with MemoriesAnchor model=410M, Initialization=B1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(512,128,32,0), Bank size=12.7B, Fetch size=50M, Retrieval Mode=Fetched2025.09 | 45.9 | |
| Pretraining with MemoriesAnchor model=410M, Initialization=B1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(512,128,32,0), Bank size=12.7B, Fetch size=50M, Retrieval Mode=Generic2025.09 | 41.8 | |
| Baseline Anchor ModelAnchor model=410M, Initialization=Scratch, Seen tokens=2.2T, Cotrain anchor=n/a, Memory config=(0,0,0,0), Bank size=0, Fetch size=0, Retrieval Mode=Generic2025.09 | 41.1 | |
| Baseline Anchor ModelAnchor model=410M, Initialization=Scratch, Seen tokens=1.1T, Cotrain anchor=n/a, Memory config=(0,0,0,0), Bank size=0, Fetch size=0, Retrieval Mode=Generic2025.09 | 40.9 | |
| Pretraining with MemoriesAnchor model=160M, Initialization=A1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(256,64,16,0), Bank size=4.6B, Fetch size=18M, Retrieval Mode=Fetched2025.09 | 40.3 | |
| Pretraining with MemoriesAnchor model=160M, Initialization=A1, Seen tokens=+1.1T, Cotrain anchor=No, Memory config=(256,64,16,0), Bank size=4.6B, Fetch size=18M, Retrieval Mode=Fetched2025.09 | 39.2 | |
| Pretraining with MemoriesAnchor model=160M, Initialization=A1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(256,64,16,0), Bank size=4.6B, Fetch size=18M, Retrieval Mode=Generic2025.09 | 35.7 | |
| Pretraining with MemoriesAnchor model=160M, Initialization=A1, Seen tokens=+1.1T, Cotrain anchor=No, Memory config=(256,64,16,0), Bank size=4.6B, Fetch size=18M, Retrieval Mode=Generic2025.09 | 34.7 | |
| Baseline Anchor ModelAnchor model=160M, Initialization=Scratch, Seen tokens=1.1T, Cotrain anchor=n/a, Memory config=(0,0,0,0), Bank size=0, Fetch size=0, Retrieval Mode=Generic2025.09 | 34.1 | |
| Baseline Anchor ModelAnchor model=160M, Initialization=Scratch, Seen tokens=2.2T, Cotrain anchor=n/a, Memory config=(0,0,0,0), Bank size=0, Fetch size=0, Retrieval Mode=Generic2025.09 | 33.8 |