Long-context Language Modeling on LongBench (MultiFieldQA, MuSiQue, GovReport 2023 test)
32.18MultiFieldQA ScoreDroPE
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| DroPEBase Model=SmolLM-1.7B, recalibration_tokens=20B2025.12 | 32.18 | 753 | 24.77 | 21.49 | |
| YaRNBase Model=SmolLM-1.7B2025.12 | 27.6 | 390 | 17.19 | 16.23 | |
| RoPE-NTKBase Model=SmolLM-1.7B2025.12 | 27.58 | 337 | 24.65 | 18.53 | |
| DroPEBase Model=Llama2-7B, recalibration_tokens=20B2025.12 | 25.9 | 1,288 | 39.47 | 26.08 | |
| YaRNBase Model=Llama2-7B2025.12 | 23.13 | 765 | 26.65 | 19.14 | |
| RoPE-NTKBase Model=Llama2-7B2025.12 | 21.81 | 1,091 | 32.91 | 21.88 | |
| BaseBase Model=Llama2-7B2025.12 | 17.26 | 1,043 | 32.41 | 20.03 | |
| BaseBase Model=SmolLM-1.7B2025.12 | 4.12 | 50 | 4.7 | 3.11 |