Long-context reasoning on BABILong
14.1Err (2k Context)RoPE++EH
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| RoPE++EHModel size=376M, Training length=32k2025.12 | 14.1 | 15.6 | 12.2 | 9.9 | 8.3 | 9.7 | 11.6 | — | — | — | |
| RoPEModel size=376M, Training length=32k2025.12 | 17.7 | 16.1 | 9.1 | 9.4 | 5.9 | 7.8 | 11 | — | — | — | |
| RoPE++ECModel size=376M, Training length=32k2025.12 | 19.8 | 19.8 | 16.1 | 15.8 | 12.3 | 12.8 | 16.1 | — | — | — | |
| 0-shotModel=Qwen2.5-7B-Instruct2026.06 | 22.4 | 23.9 | 21 | — | — | — | 23.3 | — | 20.3 | 28.8 | |
| 0-shot + YaRNModel=Olmo3-7B-Instruct2026.06 | 23.4 | 26.2 | 21.6 | — | — | — | 25.8 | — | 31.2 | 26.5 | |
| 0-shotModel=Olmo3-7B-Instruct2026.06 | 23.7 | 25.6 | 21.6 | — | — | — | 25.6 | — | 31.2 | 26.1 | |
| 0-shot + YaRNModel=Qwen2.5-7B-Instruct2026.06 | 25 | 26.6 | 22 | — | — | — | 23.9 | — | 19.7 | 26.1 | |
| RoPE++EHModel size=776M, Training length=32k2025.12 | 31.9 | 26.5 | 18.6 | 16.2 | 11 | 12.2 | 19.4 | — | — | — | |
| RoPE++ECModel size=776M, Training length=32k2025.12 | 32.4 | 29.9 | 24.4 | 24.5 | 18.6 | 14.8 | 24.1 | — | — | — | |
| RoPEModel size=776M, Training length=32k2025.12 | 33.5 | 30.7 | 23.6 | 22 | 15.1 | 12.1 | 22.8 | — | — | — | |
| RPEModel=Qwen2.5-7B-Instruct2026.06 | 82.9 | 85.6 | 83.6 | — | — | — | 82.9 | — | 79.3 | 83.3 | |
| Randomized YaRNModel=Qwen2.5-7B-Instruct2026.06 | 90.8 | 92.5 | 93.8 | — | — | — | 91.4 | — | 89.5 | 90.2 | |
| Trained YaRNModel=Qwen2.5-7B-Instruct2026.06 | 92.4 | 91.5 | 89.5 | — | — | — | 90.5 | — | 87.2 | 91.7 | |
| LoRAModel=Qwen2.5-7B-Instruct2026.06 | 95.1 | 95.7 | 93.1 | — | — | — | 95.1 | — | 95.4 | 96.2 | |
| Randomized YaRNModel=Olmo3-7B-Instruct2026.06 | 95.1 | 98 | 98.7 | — | — | — | 97.4 | — | 96.4 | 98.9 | |
| Trained YaRNModel=Olmo3-7B-Instruct2026.06 | 98 | 96.7 | 95.7 | — | — | — | 97.5 | — | 97.4 | 99.6 | |
| CorDAFine-tuning dataset=MetaMathQA, Calibration data=model-generated pretraining-proxy2026.05 | — | — | — | — | — | — | — | 65.57 | — | — | |
| CorDAFine-tuning dataset=MetaMathQA, Calibration data=model-generated pretraining-proxy, Calibration model size=4B2026.05 | — | — | — | — | — | — | — | 66.06 | — | — | |
| FoLoRAFine-tuning dataset=MetaMathQA, Calibration data=NQ Open2026.05 | — | — | — | — | — | — | — | 66.34 | — | — | |
| FoLoRAFine-tuning dataset=MetaMathQA, Calibration data=NQ Open, Calibration model size=0.6B2026.05 | — | — | — | — | — | — | — | 66.09 | — | — | |
| FoLoRAFine-tuning dataset=MetaMathQA, Calibration data=NQ Open, Calibration model size=1.7B2026.05 | — | — | — | — | — | — | — | 66.5 | — | — | |
| FoLoRAFine-tuning dataset=MetaMathQA, Calibration data=NQ Open, Calibration model size=4B2026.05 | — | — | — | — | — | — | — | 66.17 | — | — | |
| LoRAFine-tuning dataset=MetaMathQA2026.05 | — | — | — | — | — | — | — | 65.74 | — | — | |
| LoRA-NullFine-tuning dataset=MetaMathQA, Calibration data=model-generated pretraining-proxy2026.05 | — | — | — | — | — | — | — | 66.07 | — | — | |
| LoRA-NullFine-tuning dataset=MetaMathQA, Calibration data=model-generated pretraining-proxy, Calibration model size=4B2026.05 | — | — | — | — | — | — | — | 65.52 | — | — | |
| MiLoRAFine-tuning dataset=MetaMathQA2026.05 | — | — | — | — | — | — | — | 65.98 | — | — | |
| OPLoRAFine-tuning dataset=MetaMathQA2026.05 | — | — | — | — | — | — | — | 65.23 | — | — | |
| Qwen3-1.7BType=Base Model2026.05 | — | — | — | — | — | — | — | 64.54 | — | — |