Zero-shot long-context reasoning on ZeroSCROLLS
33.5Average ScoreLLMLingua-2
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| LLMLingua-2Token constraint=3,0002026.03 | 33.5 | 3,206 | 3 | 3.5 | 3.5 | |
| LLMLingua-2Token constraint=2,0002026.03 | 33.4 | 1,898 | 5 | 3 | 4.1 | |
| LLMLingua-2-smallToken constraint=3,0002026.03 | 33.4 | 3,089 | 3 | 3 | 4.1 | |
| LLMLingua-2-smallToken constraint=2,0002026.03 | 33.3 | 1,862 | 5 | 2.6 | 4.7 | |
| LongLLMLinguaToken constraint=3,0002026.03 | 33 | 3,412 | 3 | 8.2 | 1.5 | |
| LongLLMLinguaToken constraint=2,0002026.03 | 32.7 | 1,753 | 6 | 5.2 | 2.3 | |
| Original PromptConfiguration=Reference Baseline2026.03 | 32.5 | 9,788 | — | 12.2 | — | |
| BEAVERToken constraint=3,0002026.03 | 32.4 | 3,319 | 3 | 2.9 | 4.2 | |
| BEAVERToken constraint=2,0002026.03 | 32 | 1,878 | 5 | 2.5 | 4.9 | |
| LLMLinguaToken constraint=3,0002026.03 | 30.7 | 3,366 | 3 | 7.4 | 1.7 | |
| LLMLinguaToken constraint=2,0002026.03 | 27.2 | 1,862 | 5 | 4.8 | 2.5 | |
| SBERTToken constraint=3,0002026.03 | 24 | 3,340 | 3 | 5.9 | 2.1 | |
| OpenAIToken constraint=3,0002026.03 | 22.4 | 3,362 | 3 | 11.7 | 1 | |
| Selective-ContextToken constraint=3,0002026.03 | 20.7 | 3,460 | 3 | 54.2 | 0.2 | |
| OpenAIToken constraint=2,0002026.03 | 20.6 | 1,784 | 5 | 9.9 | 1.2 | |
| SBERTToken constraint=2,0002026.03 | 20.5 | 1,773 | 6 | 4.1 | 3 | |
| Selective-ContextToken constraint=2,0002026.03 | 19.4 | 1,865 | 5 | 47.5 | 0.3 | |
| Zero-ShotConfiguration=Reference Baseline2026.03 | 12.8 | 32 | 306 | 1 | 12.2 |