Multi-query associative recall on MQAR seq=128, kv=16
67Mean AccuracyMLP-λ (softmax)
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| MLP-λ (softmax)Sequence length=128, Hidden dimension=64, Training steps=5000, Number of seeds=52026.05 | 67 | 99.6 | |
| MLP-λ (softplus)Sequence length=128, Hidden dimension=64, Training steps=5000, Number of seeds=52026.05 | 66.7 | 99.2 | |
| Baseline λSequence length=128, Hidden dimension=64, Training steps=5000, Number of seeds=52026.05 | 34.8 | 99.3 |