Language Modeling on WikiEn
10.2Perplexity (WikiEn)Pretraining with Memories
Evaluation Results
| Method | Links | |
|---|---|---|
| Pretraining with MemoriesAnchor model=1.4B, Initialization=C1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(768,256,16,0), Bank size=21.1B, Fetch size=153M, Retrieval Mode=Fetched2025.09 | 10.2 | |
| Baseline Anchor ModelAnchor model=1.4B, Initialization=Scratch, Seen tokens=1.1T, Cotrain anchor=n/a, Memory config=(0,0,0,0), Bank size=0, Fetch size=0, Retrieval Mode=Generic2025.09 | 10.8 | |
| Pretraining with MemoriesAnchor model=1.4B, Initialization=C1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(768,256,16,0), Bank size=21.1B, Fetch size=153M, Retrieval Mode=Generic2025.09 | 11 | |
| Pretraining with MemoriesAnchor model=410M, Initialization=B1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(512,128,32,0), Bank size=12.7B, Fetch size=50M, Retrieval Mode=Fetched2025.09 | 12.4 | |
| Pretraining with MemoriesAnchor model=410M, Initialization=B1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(512,128,32,0), Bank size=12.7B, Fetch size=50M, Retrieval Mode=Generic2025.09 | 13.8 | |
| Baseline Anchor ModelAnchor model=410M, Initialization=Scratch, Seen tokens=2.2T, Cotrain anchor=n/a, Memory config=(0,0,0,0), Bank size=0, Fetch size=0, Retrieval Mode=Generic2025.09 | 13.8 | |
| Baseline Anchor ModelAnchor model=410M, Initialization=Scratch, Seen tokens=1.1T, Cotrain anchor=n/a, Memory config=(0,0,0,0), Bank size=0, Fetch size=0, Retrieval Mode=Generic2025.09 | 13.9 | |
| Pretraining with MemoriesAnchor model=160M, Initialization=A1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(256,64,16,0), Bank size=4.6B, Fetch size=18M, Retrieval Mode=Fetched2025.09 | 14.2 | |
| Pretraining with MemoriesAnchor model=160M, Initialization=A1, Seen tokens=+1.1T, Cotrain anchor=No, Memory config=(256,64,16,0), Bank size=4.6B, Fetch size=18M, Retrieval Mode=Fetched2025.09 | 15.2 | |
| Pretraining with MemoriesAnchor model=160M, Initialization=A1, Seen tokens=+1.1T, Cotrain anchor=Yes, Memory config=(256,64,16,0), Bank size=4.6B, Fetch size=18M, Retrieval Mode=Generic2025.09 | 16.7 | |
| Pretraining with MemoriesAnchor model=160M, Initialization=A1, Seen tokens=+1.1T, Cotrain anchor=No, Memory config=(256,64,16,0), Bank size=4.6B, Fetch size=18M, Retrieval Mode=Generic2025.09 | 16.7 | |
| Baseline Anchor ModelAnchor model=160M, Initialization=Scratch, Seen tokens=1.1T, Cotrain anchor=n/a, Memory config=(0,0,0,0), Bank size=0, Fetch size=0, Retrieval Mode=Generic2025.09 | 17.2 | |
| Baseline Anchor ModelAnchor model=160M, Initialization=Scratch, Seen tokens=2.2T, Cotrain anchor=n/a, Memory config=(0,0,0,0), Bank size=0, Fetch size=0, Retrieval Mode=Generic2025.09 | 17.8 |