Retrieval-Augmented Generation on SynthWiki 20 documents
95.75Mean ScoreLongLLMLingua-rk + Cal.
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| LongLLMLingua-rk + Cal.Model=Tulu2024.06 | 95.75 | — | — | — | |
| Prompt reorderingModel=Tulu2024.06 | 95.55 | — | — | — | |
| LongLLMLingua-rkModel=Tulu2024.06 | 95.45 | — | — | — | |
| Attention sortingModel=Tulu2024.06 | 94.2 | 94.04 | 93.53 | 95.05 | |
| Calibrated attentionModel=Tulu2024.06 | 84.5 | 82.22 | 75.15 | 96.14 | |
| Vanilla attentionModel=Tulu2024.06 | 78.81 | 80.4 | 60.3 | 95.75 | |
| LongLLMLingua-rk + Cal.Model=Vicuna2024.06 | 66.96 | — | — | — | |
| LongLLMLingua-rkModel=Vicuna2024.06 | 62.42 | — | — | — | |
| Prompt reorderingModel=Vicuna2024.06 | 62.22 | — | — | — | |
| Calibrated attentionModel=Vicuna2024.06 | 59.25 | 57.77 | 51.21 | 68.78 | |
| Attention sortingModel=Vicuna2024.06 | 57.82 | 60.6 | 51.55 | 61.31 | |
| Vanilla attentionModel=Vicuna2024.06 | 52.52 | 53.73 | 43.63 | 60.2 |