Long-context language understanding on L-Eval
58.28CourseraNTK
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| NTKBackbone=Llama3-8b-ins-8k, Target Context Window=32k2025.02 | 58.28 | — | — | — | 59.38 | 1.11 | 39.59 | |
| NTKBackbone=Llama3-8b-ins-4k, Target Context Window=16k2025.02 | 57.7 | 80 | 63.86 | 81.04 | 64.06 | 5.56 | 58.7 | |
| GALIBackbone=Llama3-8b-ins-4k, Target Context Window=16k2025.02 | 56.54 | 74 | 65.35 | 84.06 | 66.41 | 8.89 | 59.21 | |
| YaRNBackbone=Llama3-8b-ins-4k, Target Context Window=16k2025.02 | 56.4 | 81 | 59.4 | 79.18 | 64.06 | 5.56 | 57.6 | |
| YaRNBackbone=Llama3-8b-ins-8k, Target Context Window=16k2025.02 | 55.96 | — | — | — | 62.5 | 5.56 | 41.34 | |
| SelfExtendBackbone=Llama3-8b-ins-8k, Target Context Window=16k2025.02 | 55.38 | — | — | — | 64.06 | 5.56 | 41.67 | |
| SelfExtendBackbone=Llama3-8b-ins-4k, Target Context Window=16k2025.02 | 55.23 | 79 | 64.36 | 79.18 | 67.97 | 5.56 | 58.55 | |
| YaRNBackbone=Llama3-8b-ins-8k, Target Context Window=32k2025.02 | 55.23 | — | — | — | 67.19 | 7.78 | 43.4 | |
| GALIBackbone=Llama3-8b-ins-8k, Target Context Window=16k2025.02 | 54.65 | — | — | — | 65.63 | 6.67 | 42.32 | |
| SelfExtendBackbone=Llama3-8b-ins-4k, Target Context Window=32k2025.02 | 54.51 | 80 | 64.36 | 77.7 | 67.97 | 5.56 | 58.35 | |
| ChunkLlamaBackbone=Llama3-8b-ins-8k, Target Context Window=32k2025.02 | 54.36 | — | — | — | 64.06 | 5.56 | 41.33 | |
| Dyn-NTKBackbone=Llama3-8b-ins-8k, Target Context Window=32k2025.02 | 54.36 | — | — | — | 64.06 | 6.67 | 41.7 | |
| GALIBackbone=Llama3-8b-ins-4k, Target Context Window=32k2025.02 | 54.17 | 74 | 65.35 | 84.06 | 68.75 | 7.78 | 59.1 | |
| GALIBackbone=Llama3-8b-ins-8k, Target Context Window=32k2025.02 | 54.17 | — | — | — | 66.41 | 7.78 | 42.79 | |
| Dyn-NTKBackbone=Llama3-8b-ins-4k, Target Context Window=16k2025.02 | 54.07 | 75 | 64.36 | 82.16 | 67.97 | 1.11 | 57.44 | |
| SelfExtendBackbone=Llama3-8b-ins-8k, Target Context Window=32k2025.02 | 53.92 | — | — | — | 65.63 | 3.33 | 40.96 | |
| OriginalBackbone=Llama3-8b-ins-4k2025.02 | 53.34 | 75 | 59.41 | 81.41 | 60.94 | 4.44 | 55.76 | |
| ChunkLlamaBackbone=Llama3-8b-ins-8k, Target Context Window=16k2025.02 | 53.34 | — | — | — | 61.72 | 5.56 | 40.21 | |
| ChunkLlamaBackbone=Llama3-8b-ins-4k, Target Context Window=32k2025.02 | 53.2 | 75 | 63.37 | 81.04 | 63.28 | 2.22 | 56.35 | |
| OriginalBackbone=Llama3-8b-ins-8k2025.02 | 53.05 | — | — | — | 60.16 | 4.44 | 39.22 | |
| YaRNBackbone=Llama3-8b-ins-4k, Target Context Window=32k2025.02 | 53.05 | 73 | 59.41 | 79.55 | 68.75 | 5.56 | 56.55 | |
| NTKBackbone=Llama3-8b-ins-4k, Target Context Window=32k2025.02 | 52.91 | 82 | 61.39 | 79.93 | 67.19 | 2.22 | 57.6 | |
| ChunkLlamaBackbone=Llama3-8b-ins-4k, Target Context Window=16k2025.02 | 52.62 | 77 | 63.37 | 81.04 | 60.16 | 3.33 | 56.25 | |
| Dyn-NTKBackbone=Llama3-8b-ins-4k, Target Context Window=32k2025.02 | 52.33 | 76 | 63.86 | 82.16 | 71.88 | 3.33 | 58.26 | |
| NTKBackbone=Llama3-8b-ins-8k, Target Context Window=16k2025.02 | 52.03 | — | — | — | 42.97 | 0 | 31.67 | |
| Dyn-NTKBackbone=Llama3-8b-ins-8k, Target Context Window=16k2025.02 | 52.03 | — | — | — | 52.34 | 2.22 | 35.53 |