Long-context Retrieval on NIAH 64k
49.3Single ScoreOriginal
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| OriginalModel Size=8B, Length Extrapolation=YaRN factor 22026.04 | 49.3 | 44.7 | 47 | |
| OriginalModel Size=30B, Length Extrapolation=YaRN factor 22026.04 | 49.3 | 45.7 | 47.5 | |
| OriginalModel Size=14B, Length Extrapolation=YaRN factor 22026.04 | 49.1 | 45.9 | 47.5 | |
| OriginalModel Size=1.7B, Length Extrapolation=YaRN factor 22026.04 | 47.9 | 23.5 | 35.7 | |
| BOSCHModel Size=14B, SWA ratio=0.75, Length Extrapolation=YaRN factor 22026.04 | 43.7 | 23.6 | 33.7 | |
| BOSCHModel Size=8B, SWA ratio=0.75, Length Extrapolation=YaRN factor 22026.04 | 41.3 | 13.9 | 27.6 | |
| BOSCHModel Size=30B, SWA ratio=0.75, Length Extrapolation=YaRN factor 22026.04 | 38.5 | 11.1 | 24.8 | |
| FISHERModel Size=8B, SWA ratio=0.75, Length Extrapolation=YaRN factor 22026.04 | 37 | 8.7 | 22.9 | |
| FISHERModel Size=14B, SWA ratio=0.75, Length Extrapolation=YaRN factor 22026.04 | 37 | 13.3 | 25.2 | |
| BOSCHModel Size=1.7B, SWA ratio=0.75, Length Extrapolation=YaRN factor 22026.04 | 32 | 15 | 23.5 | |
| FISHERModel Size=1.7B, SWA ratio=0.75, Length Extrapolation=YaRN factor 22026.04 | 27.7 | 7.5 | 17.6 | |
| FISHERModel Size=30B, SWA ratio=0.75, Length Extrapolation=YaRN factor 22026.04 | 23.9 | 5.7 | 14.8 | |
| B-layerModel Size=1.7B, SWA ratio=0.75, Length Extrapolation=YaRN factor 22026.04 | 23.7 | 8.2 | 16 | |
| B-layerModel Size=30B, SWA ratio=0.75, Length Extrapolation=YaRN factor 22026.04 | 19.5 | 3.8 | 11.7 | |
| B-layerModel Size=8B, SWA ratio=0.75, Length Extrapolation=YaRN factor 22026.04 | 18.1 | 1.6 | 9.9 | |
| INTRModel Size=30B, SWA ratio=0.75, Length Extrapolation=YaRN factor 22026.04 | 17.3 | 2.5 | 9.9 | |
| B-layerModel Size=14B, SWA ratio=0.75, Length Extrapolation=YaRN factor 22026.04 | 12.1 | 2.7 | 7.4 | |
| INTRModel Size=8B, SWA ratio=0.75, Length Extrapolation=YaRN factor 22026.04 | 2.9 | 1.8 | 2.4 | |
| INTRModel Size=1.7B, SWA ratio=0.75, Length Extrapolation=YaRN factor 22026.04 | 2.6 | 1.7 | 2.2 | |
| INTRModel Size=14B, SWA ratio=0.75, Length Extrapolation=YaRN factor 22026.04 | 2.6 | 1.3 | 2 |