Retrieval-Augmented Generation on SynthWiki 10 documents
94.44Average ScoreLongLLMLingua-rk + Cal.
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| LongLLMLingua-rk + Cal.Model=Tulu2024.06 | 94.44 | — | — | — | |
| Prompt reorderingModel=Tulu2024.06 | 94.04 | — | — | — | |
| LongLLMLingua-rkModel=Tulu2024.06 | 94.04 | — | — | — | |
| Attention sortingModel=Tulu2024.06 | 93.19 | 92.92 | 92.82 | 93.83 | |
| Calibrated attentionModel=Tulu2024.06 | 91.98 | 92.92 | 87.77 | 95.25 | |
| Vanilla attentionModel=Tulu2024.06 | 89.35 | 92.22 | 81.51 | 94.34 | |
| LongLLMLingua-rk + Cal.Model=Vicuna2024.06 | 73.43 | — | — | — | |
| LongLLMLingua-rkModel=Vicuna2024.06 | 70.5 | — | — | — | |
| Prompt reorderingModel=Vicuna2024.06 | 70.2 | — | — | — | |
| Attention sortingModel=Vicuna2024.06 | 66.36 | 67.37 | 64.14 | 67.57 | |
| Calibrated attentionModel=Vicuna2024.06 | 65.52 | 68.58 | 53.83 | 74.14 | |
| Vanilla attentionModel=Vicuna2024.06 | 60.8 | 65.15 | 48.68 | 68.58 |