Multi-query associative recall on MQAR seq=128, kv=8
66.8Mean AccuracyMLP-λ (softmax)
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| MLP-λ (softmax)Sequence length=128, Hidden dimension=64, Training steps=5000, Number of seeds=52026.05 | 66.8 | 99.3 | |
| MLP-λ (softplus)Sequence length=128, Hidden dimension=64, Training steps=5000, Number of seeds=52026.05 | 66.7 | 99.2 | |
| Baseline λSequence length=128, Hidden dimension=64, Training steps=5000, Number of seeds=52026.05 | 35.8 | 99.5 |