Multi-query associative recall on MQAR seq=128, kv=4
67Mean AccuracyMLP-λ (softplus)
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| MLP-λ (softplus)Sequence length=128, Hidden dimension=64, Training steps=5000, Number of seeds=52026.05 | 67 | 99.7 | |
| Baseline λSequence length=128, Hidden dimension=64, Training steps=5000, Number of seeds=52026.05 | 35.1 | 99.4 | |
| MLP-λ (softmax)Sequence length=128, Hidden dimension=64, Training steps=5000, Number of seeds=52026.05 | 2.7 | 3.8 |