Multi-query associative recall on MQAR seq=128, kv=32
57.9Mean AccuracyMLP-λ (softplus)
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| MLP-λ (softplus)Sequence length=128, Hidden dimension=64, Training steps=5000, Number of seeds=52026.05 | 57.9 | 99.6 | |
| MLP-λ (softmax)Sequence length=128, Hidden dimension=64, Training steps=5000, Number of seeds=52026.05 | 35.2 | 99 | |
| Baseline λSequence length=128, Hidden dimension=64, Training steps=5000, Number of seeds=52026.05 | 2.9 | 99.3 |