Long-context Information Retrieval on Needle-In-a-Haystack Verbatim prompt (test)
0.996Accuracy (Depth 0%)Vanilla
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| VanillaModel Scale=300M, Prompt Type=Verbatim2024.06 | 0.996 | 1 | 1 | 0.998 | 1 | 1 | 0.998 | 1 | 1 | 1 | 0.998 | 0.9991 | |
| Block TransformerModel Scale=85M, Prompt Type=Verbatim2024.06 | 0.962 | 0.976 | 0.962 | 0.966 | 0.984 | 0.98 | 0.972 | 0.988 | 0.99 | 0.994 | 0.962 | 0.976 | |
| VanillaModel Scale=85M, Prompt Type=Verbatim2024.06 | 0.956 | 0.994 | 0.99 | 0.994 | 0.992 | 0.992 | 0.99 | 0.996 | 0.996 | 0.99 | 0.956 | 0.986 | |
| Block TransformerModel Scale=800M, Prompt Type=Verbatim2024.06 | 0.952 | 0.994 | 0.988 | 0.988 | 0.988 | 0.988 | 0.99 | 0.994 | 0.992 | 0.974 | 0.996 | 0.9858 | |
| Block TransformerModel Scale=1.2B, Prompt Type=Verbatim2024.06 | 0.926 | 0.984 | 0.994 | 0.988 | 0.996 | 0.996 | 0.988 | 0.998 | 0.998 | 0.992 | 0.98 | 0.9855 | |
| Block TransformerModel Scale=300M, Prompt Type=Verbatim2024.06 | 0.902 | 0.994 | 0.996 | 0.992 | 0.986 | 0.996 | 0.996 | 0.998 | 0.998 | 0.992 | 0.992 | 0.9856 | |
| VanillaModel Scale=19M, Prompt Type=Verbatim2024.06 | 0.082 | 0.014 | 0.03 | 0.068 | 0.078 | 0.126 | 0.454 | 0.658 | 0.634 | 0.846 | 0.994 | 0.3622 |