Context Compression Evaluation on BenchPress suite macro-averaged across all datasets
74.33Macro-averaged F1Qwen3-8B
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Qwen3-8BContext=Original2025.10 | 74.33 | — | |
| Qwen3-4BContext=Original2025.10 | 73.44 | — | |
| Gemma2-2BContext=Original2025.10 | 71.96 | — | |
| Mean-PoolingTeacher Model=Qwen3-8B, Compression Ratio=4x, Training Regime=Single2025.10 | 71.66 | — | |
| Mean-PoolingTeacher Model=Qwen3-8B, Compression Ratio=4x, Training Regime=Multi2025.10 | 70.55 | — | |
| Mean-PoolingTeacher Model=Qwen3-4B, Compression Ratio=4x, Training Regime=Single2025.10 | 70.39 | — | |
| Qwen3-1.7BContext=Original2025.10 | 69.93 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Qwen3-8B, Compression Ratio=4x, Training Regime=Multi2025.10 | 69.57 | — | |
| Mean-PoolingTeacher Model=Qwen3-4B, Compression Ratio=4x, Training Regime=Multi2025.10 | 69.36 | — | |
| Mean-PoolingTeacher Model=Gemma2-2B, Compression Ratio=4x, Training Regime=Single2025.10 | 69.33 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Qwen3-8B, Compression Ratio=4x, Training Regime=Single2025.10 | 69.2 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Qwen3-4B, Compression Ratio=4x, Training Regime=Multi2025.10 | 68.15 | — | |
| Mean-PoolingTeacher Model=Gemma2-2B, Compression Ratio=4x, Training Regime=Multi2025.10 | 68.09 | — | |
| Compression-Tokens (Causal)Teacher Model=Qwen3-8B, Compression Ratio=4x, Training Regime=Single2025.10 | 67.03 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Qwen3-4B, Compression Ratio=4x, Training Regime=Single2025.10 | 66.72 | — | |
| Mean-PoolingTeacher Model=Qwen3-1.7B, Compression Ratio=4x, Training Regime=Single2025.10 | 66.43 | — | |
| Compression-Tokens (Causal)Teacher Model=Qwen3-8B, Compression Ratio=4x, Training Regime=Multi2025.10 | 65.9 | — | |
| Llama3.2-1BContext=Original2025.10 | 65.82 | — | |
| Qwen3-0.6BContext=Original2025.10 | 65.36 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Gemma2-2B, Compression Ratio=4x, Training Regime=Multi2025.10 | 65.24 | — | |
| Compression-Tokens (Causal)Teacher Model=Qwen3-4B, Compression Ratio=4x, Training Regime=Single2025.10 | 64.88 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Gemma2-2B, Compression Ratio=4x, Training Regime=Single2025.10 | 64.76 | — | |
| Mean-PoolingTeacher Model=Qwen3-8B, Compression Ratio=16x, Training Regime=Multi2025.10 | 64.67 | — | |
| Mean-PoolingTeacher Model=Qwen3-1.7B, Compression Ratio=4x, Training Regime=Multi2025.10 | 64.17 | — | |
| Mean-PoolingTeacher Model=Qwen3-8B, Compression Ratio=16x, Training Regime=Single2025.10 | 63.85 | — | |
| Compression-Tokens (Causal)Teacher Model=Gemma2-2B, Compression Ratio=4x, Training Regime=Single2025.10 | 63.35 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Qwen3-8B, Compression Ratio=16x, Training Regime=Multi2025.10 | 63.01 | — | |
| PCC LargeTeacher Model=Llama3.1-8B, Compression Ratio=4x, Training Regime=Single2025.10 | 62.98 | — | |
| Mean-PoolingTeacher Model=Llama3.2-1B, Compression Ratio=4x, Training Regime=Single2025.10 | 62.81 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Qwen3-1.7B, Compression Ratio=4x, Training Regime=Multi2025.10 | 62.6 | — | |
| Compression-Tokens (Causal)Teacher Model=Qwen3-4B, Compression Ratio=4x, Training Regime=Multi2025.10 | 62.53 | — | |
| Compression-Tokens (Causal)Teacher Model=Gemma2-2B, Compression Ratio=4x, Training Regime=Multi2025.10 | 62.18 | — | |
| PCC LiteTeacher Model=GPT2-Large & Llama3.1-8B, Compression Ratio=4x, Training Regime=Single2025.10 | 62.08 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Qwen3-1.7B, Compression Ratio=4x, Training Regime=Single2025.10 | 62.04 | — | |
| Mean-PoolingTeacher Model=Qwen3-4B, Compression Ratio=16x, Training Regime=Single2025.10 | 61.79 | — | |
| Mean-PoolingTeacher Model=Qwen3-4B, Compression Ratio=16x, Training Regime=Multi2025.10 | 61.72 | — | |
| Mean-PoolingTeacher Model=Gemma2-2B, Compression Ratio=16x, Training Regime=Single2025.10 | 61.39 | — | |
| Mean-PoolingTeacher Model=Qwen3-0.6B, Compression Ratio=4x, Training Regime=Single2025.10 | 61.17 | — | |
| Mean-PoolingTeacher Model=Gemma2-2B, Compression Ratio=16x, Training Regime=Multi2025.10 | 61.04 | — | |
| Mean-PoolingTeacher Model=Llama3.2-1B, Compression Ratio=4x, Training Regime=Multi2025.10 | 60.56 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Qwen3-4B, Compression Ratio=16x, Training Regime=Multi2025.10 | 60.48 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Qwen3-8B, Compression Ratio=16x, Training Regime=Single2025.10 | 60.27 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Gemma2-2B, Compression Ratio=16x, Training Regime=Multi2025.10 | 58.43 | — | |
| Compression-Tokens (Causal)Teacher Model=Qwen3-8B, Compression Ratio=16x, Training Regime=Multi2025.10 | 58.41 | — | |
| Mean-PoolingTeacher Model=Qwen3-0.6B, Compression Ratio=4x, Training Regime=Multi2025.10 | 58.36 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Llama3.2-1B, Compression Ratio=4x, Training Regime=Single2025.10 | 57.91 | — | |
| Compression-Tokens (Causal)Teacher Model=Qwen3-1.7B, Compression Ratio=4x, Training Regime=Multi2025.10 | 57.73 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Qwen3-4B, Compression Ratio=16x, Training Regime=Single2025.10 | 57.68 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Llama3.2-1B, Compression Ratio=4x, Training Regime=Multi2025.10 | 57.52 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Qwen3-0.6B, Compression Ratio=4x, Training Regime=Multi2025.10 | 57.03 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Gemma2-2B, Compression Ratio=16x, Training Regime=Single2025.10 | 56.39 | — | |
| Compression-Tokens (Causal)Teacher Model=Llama3.2-1B, Compression Ratio=4x, Training Regime=Single2025.10 | 56.31 | — | |
| Compression-Tokens (Causal)Teacher Model=Qwen3-8B, Compression Ratio=16x, Training Regime=Single2025.10 | 56.21 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Qwen3-0.6B, Compression Ratio=4x, Training Regime=Single2025.10 | 55.59 | — | |
| Mean-PoolingTeacher Model=Qwen3-1.7B, Compression Ratio=16x, Training Regime=Single2025.10 | 55.43 | — | |
| Compression-Tokens (Causal)Teacher Model=Qwen3-4B, Compression Ratio=16x, Training Regime=Single2025.10 | 55.22 | — | |
| Compression-Tokens (Causal)Teacher Model=Gemma2-2B, Compression Ratio=16x, Training Regime=Single2025.10 | 55.07 | — | |
| Compression-Tokens (Causal)Teacher Model=Gemma2-2B, Compression Ratio=16x, Training Regime=Multi2025.10 | 54.7 | — | |
| Mean-PoolingTeacher Model=Qwen3-1.7B, Compression Ratio=16x, Training Regime=Multi2025.10 | 54.47 | — | |
| Compression-Tokens (Causal)Teacher Model=Qwen3-0.6B, Compression Ratio=4x, Training Regime=Single2025.10 | 54.4 | — | |
| Compression-Tokens (Causal)Teacher Model=Qwen3-4B, Compression Ratio=16x, Training Regime=Multi2025.10 | 54.28 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Qwen3-1.7B, Compression Ratio=16x, Training Regime=Multi2025.10 | 54.11 | — | |
| Compression-Tokens (Causal)Teacher Model=Llama3.2-1B, Compression Ratio=4x, Training Regime=Multi2025.10 | 53.74 | — | |
| Compression-Tokens (Causal)Teacher Model=Qwen3-0.6B, Compression Ratio=4x, Training Regime=Multi2025.10 | 51.85 | — | |
| Mean-PoolingTeacher Model=Llama3.2-1B, Compression Ratio=16x, Training Regime=Multi2025.10 | 51.56 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Qwen3-1.7B, Compression Ratio=16x, Training Regime=Single2025.10 | 51.53 | — | |
| PCC LiteTeacher Model=GPT2-Large & Llama3.1-8B, Compression Ratio=16x, Training Regime=Single2025.10 | 51.3 | — | |
| Compression-Tokens (Causal)Teacher Model=Qwen3-1.7B, Compression Ratio=4x, Training Regime=Single2025.10 | 50.9 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Llama3.2-1B, Compression Ratio=16x, Training Regime=Multi2025.10 | 50.06 | — | |
| Compression-Tokens (Causal)Teacher Model=Qwen3-1.7B, Compression Ratio=16x, Training Regime=Single2025.10 | 49.83 | — | |
| PCC LargeTeacher Model=Llama3.1-8B, Compression Ratio=16x, Training Regime=Single2025.10 | 49.37 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Llama3.2-1B, Compression Ratio=16x, Training Regime=Single2025.10 | 49.2 | — | |
| Compression-Tokens (Causal)Teacher Model=Qwen3-1.7B, Compression Ratio=16x, Training Regime=Multi2025.10 | 48.68 | — | |
| Mean-PoolingTeacher Model=Qwen3-8B, Compression Ratio=128x, Training Regime=Single2025.10 | 47.9 | — | |
| Mean-PoolingTeacher Model=Qwen3-0.6B, Compression Ratio=16x, Training Regime=Multi2025.10 | 47.64 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Qwen3-0.6B, Compression Ratio=16x, Training Regime=Multi2025.10 | 47.62 | — | |
| Mean-PoolingTeacher Model=Qwen3-0.6B, Compression Ratio=16x, Training Regime=Single2025.10 | 47.59 | — | |
| Compression-Tokens (Causal)Teacher Model=Llama3.2-1B, Compression Ratio=16x, Training Regime=Single2025.10 | 47.51 | — | |
| Compression-Tokens (Causal)Teacher Model=Qwen3-8B, Compression Ratio=128x, Training Regime=Single2025.10 | 47.47 | — | |
| Mean-PoolingTeacher Model=Llama3.2-1B, Compression Ratio=16x, Training Regime=Single2025.10 | 47.28 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Qwen3-8B, Compression Ratio=128x, Training Regime=Multi2025.10 | 46.97 | — | |
| Compression-Tokens (Causal)Teacher Model=Llama3.2-1B, Compression Ratio=16x, Training Regime=Multi2025.10 | 46.96 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Qwen3-8B, Compression Ratio=128x, Training Regime=Single2025.10 | 46.93 | — | |
| Mean-PoolingTeacher Model=Qwen3-8B, Compression Ratio=128x, Training Regime=Multi2025.10 | 45.92 | — | |
| Mean-PoolingTeacher Model=Gemma2-2B, Compression Ratio=128x, Training Regime=Single2025.10 | 44.98 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Qwen3-0.6B, Compression Ratio=16x, Training Regime=Single2025.10 | 44.82 | — | |
| Compression-Tokens (Causal)Teacher Model=Qwen3-8B, Compression Ratio=128x, Training Regime=Multi2025.10 | 44.76 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Gemma2-2B, Compression Ratio=128x, Training Regime=Single2025.10 | 44.73 | — | |
| Compression-Tokens (Causal)Teacher Model=Gemma2-2B, Compression Ratio=128x, Training Regime=Single2025.10 | 44.46 | — | |
| Mean-PoolingTeacher Model=Gemma2-2B, Compression Ratio=128x, Training Regime=Multi2025.10 | 43.71 | — | |
| Mean-PoolingTeacher Model=Qwen3-4B, Compression Ratio=128x, Training Regime=Single2025.10 | 43.62 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Gemma2-2B, Compression Ratio=128x, Training Regime=Multi2025.10 | 43.17 | — | |
| Compression-Tokens (Causal)Teacher Model=Qwen3-4B, Compression Ratio=128x, Training Regime=Single2025.10 | 43.08 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Qwen3-4B, Compression Ratio=128x, Training Regime=Multi2025.10 | 42.66 | — | |
| Compression-Tokens (Causal)Teacher Model=Qwen3-0.6B, Compression Ratio=16x, Training Regime=Multi2025.10 | 42.59 | — | |
| LLMLingua2Teacher Model=Qwen3-8B, Compression Ratio=4x, Training Regime=Single2025.10 | 42.52 | — | |
| Compression-Tokens (Causal)Teacher Model=Gemma2-2B, Compression Ratio=128x, Training Regime=Multi2025.10 | 42.49 | — | |
| ICAETeacher Model=Mistral-7B, Compression Ratio=4x, Training Regime=Single2025.10 | 42.4 | — | |
| Compression-Tokens (Bidirectional)Teacher Model=Qwen3-4B, Compression Ratio=128x, Training Regime=Single2025.10 | 41.61 | — | |
| Compression-Tokens (Causal)Teacher Model=Qwen3-0.6B, Compression Ratio=16x, Training Regime=Single2025.10 | 41.57 | — |