Question Answering on NaturalQuestions 20 documents
2,946Tokens ProcessedOriginal Prompt
Evaluation Results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Original Prompt2024.03 | 2,946 | 75.7 | 57.3 | 54.1 | 55.4 | 63.1 | — | — | — | — | |
| Original PromptEvaluation Model=GPT3.5-Turbo2023.10 | 2,946 | 75.7 | 57.3 | 54.1 | 55.4 | 63.1 | — | — | 4.1 | — | |
| GzipEvaluation Model=GPT3.5-Turbo, Compression Ratio Constraint=2x2023.10 | 1,567 | 64.6 | 63.8 | 60.5 | 58.3 | 57.3 | 64.4 | 1.9 | 2.1 | 1.9 | |
| OpenAIEvaluation Model=GPT3.5-Turbo, Compression Ratio Constraint=2x2023.10 | 1,550 | 73 | 65.6 | 66.5 | 65.4 | 65.5 | 69.9 | 1.9 | 4.9 | 0.8 | |
| SBERTEvaluation Model=GPT3.5-Turbo, Compression Ratio Constraint=2x2023.10 | 1,549 | 72.5 | 67.9 | 63.3 | 65 | 66.2 | 68.7 | 1.9 | 2.2 | 1.9 | |
| LongLLMLingua rkEvaluation Model=GPT3.5-Turbo, Compression Ratio Constraint=2x2023.10 | 1,548 | 73.9 | 67.7 | 68.7 | 66 | 65.6 | 74.3 | 1.9 | 2.3 | 1.8 | |
| BM25Evaluation Model=GPT3.5-Turbo, Compression Ratio Constraint=2x2023.10 | 1,545 | 53.7 | 49.3 | 47.9 | 49.9 | 46.9 | 50.3 | 1.9 | 2.1 | 1.9 | |
| Selective-ContextEvaluation Model=GPT3.5-Turbo, Compression Ratio Constraint=2x2023.10 | 1,478 | 45.4 | 39 | 33.8 | 33.5 | 41.5 | — | 2 | 7.4 | 0.6 | |
| LongLLMLinguaEvaluation Model=GPT3.5-Turbo, Compression Ratio Constraint=2x2023.10 | 1,429 | 77.2 | 72.9 | 70.8 | 70.5 | 70.6 | 76.2 | 2.1 | 2.9 | 1.4 | |
| LongLLMLinguaEvaluation Model=LongChat-13b, Compression Ratio Constraint=2x2023.10 | 1,429 | 68.7 | 59.4 | 57.3 | 55.9 | 58.4 | 66.1 | 2.1 | 2.9 | 1.4 | |
| LLMLinguaEvaluation Model=GPT3.5-Turbo, Compression Ratio Constraint=2x2023.10 | 1,410 | 39.7 | 39.5 | 40.4 | 37.1 | 42.3 | 41.5 | 2.1 | 2.8 | 1.5 | |
| GzipCompression Strategy=Question-Aware Compression, Constraint=4x2024.03 | 824 | 63.1 | 61 | 59.8 | 61.1 | 60.1 | 62.3 | 3.6 | — | — | |
| SBERTCompression Strategy=Question-Aware Compression, Constraint=4x2024.03 | 808 | 66.9 | 61.1 | 59 | 61.2 | 60.3 | 64.4 | 3.6 | — | — | |
| OpenAICompression Strategy=Question-Aware Compression, Constraint=4x2024.03 | 804 | 63.8 | 64.6 | 65.4 | 64.1 | 63.7 | 63.7 | 3.7 | — | — | |
| BM25Compression Strategy=Question-Aware Compression, Constraint=4x2024.03 | 798 | 40.6 | 38.6 | 38.2 | 37.4 | 36.6 | 36.3 | 3.7 | — | — | |
| BM25Evaluation Model=GPT3.5-Turbo, Compression Ratio Constraint=4x2023.10 | 798 | 40.6 | 38.6 | 38.2 | 37.4 | 36.6 | 36.3 | 3.7 | 1.5 | 2.7 | |
| Selective-ContextCompression Strategy=Question-Agnostic Compression, Constraint=4x2024.03 | 791 | 31.4 | 19.5 | 24.7 | 24.1 | 43.8 | — | 3.7 | — | — | |
| LLMLinguaCompression Strategy=Question-Agnostic Compression, Constraint=4x2024.03 | 775 | 25.5 | 27.5 | 23.5 | 26.5 | 30 | 27 | 3.8 | — | — | |
| LongLLMLinguaCompression Strategy=Question-Aware Compression, Constraint=4x2024.03 | 748 | 75 | 71.8 | 71.2 | 71.2 | 74.7 | 75.5 | 3.9 | — | — | |
| LLMLingua-2Compression Strategy=Question-Agnostic Compression, Constraint=4x2024.03 | 748 | 48.6 | 44.5 | 43.6 | 40.9 | 39.9 | 46.2 | 3.9 | — | — | |
| LongLLMLinguaEvaluation Model=GPT3.5-Turbo, Compression Ratio Constraint=4x2023.10 | 748 | 75 | 71.8 | 71.2 | 71.2 | 74.7 | 75.5 | 3.9 | 2.1 | 2 | |
| LongLLMLinguaEvaluation Model=LongChat-13b, Compression Ratio Constraint=4x2023.10 | 748 | 68.7 | 60.5 | 59.3 | 58.3 | 61.3 | 66.7 | 3.9 | 2.1 | 2 | |
| LLMLingua-2+Compression Strategy=Question-Aware Compression, Constraint=4x, Coarse-level compression=LongLLMLingua2024.03 | 739 | 74 | 70.4 | 67 | 66.9 | 65.3 | 71.9 | 3.9 | — | — | |
| Zero-shot2024.03 | 15 | — | — | 56.1 | — | — | — | 196 | — | — | |
| Zero-shotEvaluation Model=GPT3.5-Turbo2023.10 | 15 | 56.1 | — | — | — | — | — | 196 | 1.1 | 3.7 | |
| Zero-shotEvaluation Model=LongChat-13b2023.10 | 15 | 35 | — | — | — | — | — | 196 | 1.1 | 3.7 |