Long-context retrieval and reasoning on RULER 16k context 3k budget
100S-1 ScoreLLMLingua
Evaluation results
| Method | Links | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| LLMLinguaAvg Tokens=3,2302026.03 | 100 | 5 | 4 | 7 | 6 | 12 | 6 | 4.8 | 59.4 | 90.7 | 15 | 37 | 28.9 | |
| LongLLMLinguaAvg Tokens=3,1792026.03 | 100 | 6 | 4 | 9 | 2 | 3 | 7.8 | 7.3 | 60.4 | 91.3 | 19 | 36 | 28.8 | |
| BEAVERAvg Tokens=3,1982026.03 | 100 | 100 | 100 | 99 | 88 | 41 | 99.8 | 99.8 | 78.2 | 93.7 | 50 | 55 | 83.7 | |
| LLMLingua-2Avg Tokens=3,1322026.03 | 27 | 86 | 27 | 72 | 11 | 2 | 80.3 | 82.3 | 3.4 | 95.3 | 45 | 43 | 47.9 |