Needle-In-a-Haystack Retrieval Success Rate on Gemini Prompt Test Set
57.2Success Rate @ 0% InsertionBlock Transformer
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Block TransformerN-Emb=1.2B2024.06 | 57.2 | 86.6 | 88.8 | 85.6 | 80.4 | 85.2 | 90.4 | 89.2 | 9,100 | 9,040 | 78.8 | 8,396 | |
| Vanilla TransformerN-Emb=300M2024.06 | 46.2 | 69 | 72.8 | 78.6 | 76.4 | 70.4 | 71.8 | 74.8 | 7,380 | 7,840 | 66.2 | 7,076 | |
| Block TransformerN-Emb=800M2024.06 | 35.8 | 74 | 76.4 | 78.4 | 69.8 | 77.4 | 76.4 | 79 | 7,520 | 7,280 | 53.6 | 6,989 | |
| Block TransformerN-Emb=300M2024.06 | 23.4 | 52.6 | 52.6 | 46.6 | 46 | 49.2 | 58.4 | 70.4 | 6,400 | 5,360 | 18.4 | 4,865 | |
| Vanilla TransformerN-Emb=85M2024.06 | 21 | 16.4 | 21.8 | 27.4 | 36.6 | 28 | 26.8 | 40.2 | 4,180 | 3,760 | 22.8 | 2,913 | |
| Block TransformerN-Emb=85M2024.06 | 5.6 | 2.4 | 0.8 | 0.8 | 0.2 | 1 | 0.8 | 1 | 260 | 180 | 6.4 | 213 | |
| Vanilla TransformerN-Emb=19M2024.06 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 20 | 80 | 6.4 | 67 |