Language Model Evaluation on BenchPress short-context (test)
68.84AccuracyQwen3-8B
Evaluation results
| Method | Links | |
|---|---|---|
| Qwen3-8BCompression Ratio=Original2025.10 | 68.84 | |
| Qwen3-4BCompression Ratio=Original2025.10 | 67.69 | |
| Gemma2-2BCompression Ratio=Original2025.10 | 66.14 | |
| Mean-PoolingBackbone=Qwen3-8B, Compression Ratio=4x, Aggregation Mode=Single2025.10 | 65.91 | |
| Mean-PoolingBackbone=Qwen3-8B, Compression Ratio=4x, Aggregation Mode=Multi2025.10 | 65.06 | |
| Qwen3-1.7BCompression Ratio=Original2025.10 | 64.21 | |
| Mean-PoolingBackbone=Qwen3-4B, Compression Ratio=4x, Aggregation Mode=Single2025.10 | 64.05 | |
| Compression-Tokens (Bidirectional)Backbone=Qwen3-8B, Compression Ratio=4x, Aggregation Mode=Multi2025.10 | 63.58 | |
| Mean-PoolingBackbone=Qwen3-4B, Compression Ratio=4x, Aggregation Mode=Multi2025.10 | 62.94 | |
| Mean-PoolingBackbone=Gemma2-2B, Compression Ratio=4x, Aggregation Mode=Single2025.10 | 62.51 | |
| Compression-Tokens (Bidirectional)Backbone=Qwen3-8B, Compression Ratio=4x, Aggregation Mode=Single2025.10 | 62.5 | |
| Mean-PoolingBackbone=Gemma2-2B, Compression Ratio=4x, Aggregation Mode=Multi2025.10 | 61.28 | |
| Compression-Tokens (Bidirectional)Backbone=Qwen3-4B, Compression Ratio=4x, Aggregation Mode=Multi2025.10 | 60.99 | |
| Llama3.2-1BCompression Ratio=Original2025.10 | 60.3 | |
| Mean-PoolingBackbone=Qwen3-1.7B, Compression Ratio=4x, Aggregation Mode=Single2025.10 | 60.03 | |
| Compression-Tokens (Causal)Backbone=Qwen3-8B, Compression Ratio=4x, Aggregation Mode=Single2025.10 | 59.79 | |
| Qwen3-0.6BCompression Ratio=Original2025.10 | 59.67 | |
| Compression-Tokens (Bidirectional)Backbone=Qwen3-4B, Compression Ratio=4x, Aggregation Mode=Single2025.10 | 59.35 | |
| Compression-Tokens (Causal)Backbone=Qwen3-8B, Compression Ratio=4x, Aggregation Mode=Multi2025.10 | 58.58 | |
| Compression-Tokens (Bidirectional)Backbone=Gemma2-2B, Compression Ratio=4x, Aggregation Mode=Multi2025.10 | 57.54 | |
| Mean-PoolingBackbone=Qwen3-1.7B, Compression Ratio=4x, Aggregation Mode=Multi2025.10 | 57.28 | |
| Compression-Tokens (Causal)Backbone=Qwen3-4B, Compression Ratio=4x, Aggregation Mode=Single2025.10 | 57.12 | |
| Compression-Tokens (Bidirectional)Backbone=Gemma2-2B, Compression Ratio=4x, Aggregation Mode=Single2025.10 | 56.94 | |
| Mean-PoolingBackbone=Qwen3-8B, Compression Ratio=16x, Aggregation Mode=Multi2025.10 | 56.77 | |
| Mean-PoolingBackbone=Llama3.2-1B, Compression Ratio=4x, Aggregation Mode=Single2025.10 | 56.62 | |
| Mean-PoolingBackbone=Qwen3-8B, Compression Ratio=16x, Aggregation Mode=Single2025.10 | 55.68 | |
| Compression-Tokens (Causal)Backbone=Gemma2-2B, Compression Ratio=4x, Aggregation Mode=Single2025.10 | 55.5 | |
| Compression-Tokens (Bidirectional)Backbone=Qwen3-8B, Compression Ratio=16x, Aggregation Mode=Multi2025.10 | 55.26 | |
| PCC LargeBackbone=Llama3.1-8B, Compression Ratio=4x, Aggregation Mode=Single2025.10 | 55.17 | |
| Compression-Tokens (Bidirectional)Backbone=Qwen3-1.7B, Compression Ratio=4x, Aggregation Mode=Multi2025.10 | 54.7 | |
| Mean-PoolingBackbone=Qwen3-0.6B, Compression Ratio=4x, Aggregation Mode=Single2025.10 | 54.36 | |
| Compression-Tokens (Bidirectional)Backbone=Qwen3-1.7B, Compression Ratio=4x, Aggregation Mode=Single2025.10 | 54.31 | |
| Compression-Tokens (Causal)Backbone=Qwen3-4B, Compression Ratio=4x, Aggregation Mode=Multi2025.10 | 54.17 | |
| PCC LiteBackbone=GPT2-Large & Llama3.1-8B, Compression Ratio=4x, Aggregation Mode=Single2025.10 | 54.05 | |
| PISCOBackbone=Llama3.1-8B, Compression Ratio=16x, Aggregation Mode=Single2025.10 | 53.62 | |
| Mean-PoolingBackbone=Llama3.2-1B, Compression Ratio=4x, Aggregation Mode=Multi2025.10 | 53.5 | |
| Compression-Tokens (Causal)Backbone=Gemma2-2B, Compression Ratio=4x, Aggregation Mode=Multi2025.10 | 52.94 | |
| Mean-PoolingBackbone=Qwen3-4B, Compression Ratio=16x, Aggregation Mode=Multi2025.10 | 52.82 | |
| Mean-PoolingBackbone=Qwen3-4B, Compression Ratio=16x, Aggregation Mode=Single2025.10 | 52.77 | |
| Mean-PoolingBackbone=Gemma2-2B, Compression Ratio=16x, Aggregation Mode=Single2025.10 | 52.55 | |
| Compression-Tokens (Bidirectional)Backbone=Qwen3-4B, Compression Ratio=16x, Aggregation Mode=Multi2025.10 | 52.23 | |
| Compression-Tokens (Bidirectional)Backbone=Qwen3-8B, Compression Ratio=16x, Aggregation Mode=Single2025.10 | 52.22 | |
| Mean-PoolingBackbone=Gemma2-2B, Compression Ratio=16x, Aggregation Mode=Multi2025.10 | 51.9 | |
| Mean-PoolingBackbone=Qwen3-0.6B, Compression Ratio=4x, Aggregation Mode=Multi2025.10 | 51.16 | |
| Compression-Tokens (Bidirectional)Backbone=Llama3.2-1B, Compression Ratio=4x, Aggregation Mode=Single2025.10 | 50.89 | |
| Compression-Tokens (Bidirectional)Backbone=Llama3.2-1B, Compression Ratio=4x, Aggregation Mode=Multi2025.10 | 50.1 | |
| Compression-Tokens (Causal)Backbone=Qwen3-8B, Compression Ratio=16x, Aggregation Mode=Multi2025.10 | 49.99 | |
| Compression-Tokens (Bidirectional)Backbone=Gemma2-2B, Compression Ratio=16x, Aggregation Mode=Multi2025.10 | 49.52 | |
| Compression-Tokens (Causal)Backbone=Qwen3-1.7B, Compression Ratio=4x, Aggregation Mode=Multi2025.10 | 49.31 | |
| ICAEBackbone=Mistral-7B, Compression Ratio=4x, Aggregation Mode=Single2025.10 | 49.18 | |
| Compression-Tokens (Bidirectional)Backbone=Qwen3-4B, Compression Ratio=16x, Aggregation Mode=Single2025.10 | 48.92 | |
| Compression-Tokens (Causal)Backbone=Llama3.2-1B, Compression Ratio=4x, Aggregation Mode=Single2025.10 | 48.85 | |
| Compression-Tokens (Bidirectional)Backbone=Qwen3-0.6B, Compression Ratio=4x, Aggregation Mode=Multi2025.10 | 48.84 | |
| Compression-Tokens (Bidirectional)Backbone=Qwen3-0.6B, Compression Ratio=4x, Aggregation Mode=Single2025.10 | 48.04 | |
| Compression-Tokens (Causal)Backbone=Qwen3-8B, Compression Ratio=16x, Aggregation Mode=Single2025.10 | 47.07 | |
| Compression-Tokens (Bidirectional)Backbone=Gemma2-2B, Compression Ratio=16x, Aggregation Mode=Single2025.10 | 46.9 | |
| Mean-PoolingBackbone=Qwen3-1.7B, Compression Ratio=16x, Aggregation Mode=Single2025.10 | 46.63 | |
| Compression-Tokens (Causal)Backbone=Qwen3-0.6B, Compression Ratio=4x, Aggregation Mode=Single2025.10 | 46.36 | |
| Compression-Tokens (Causal)Backbone=Gemma2-2B, Compression Ratio=16x, Aggregation Mode=Single2025.10 | 46.13 | |
| Compression-Tokens (Causal)Backbone=Qwen3-4B, Compression Ratio=16x, Aggregation Mode=Single2025.10 | 46.06 | |
| Compression-Tokens (Causal)Backbone=Gemma2-2B, Compression Ratio=16x, Aggregation Mode=Multi2025.10 | 45.59 | |
| Compression-Tokens (Causal)Backbone=Qwen3-4B, Compression Ratio=16x, Aggregation Mode=Multi2025.10 | 45.47 | |
| Compression-Tokens (Causal)Backbone=Llama3.2-1B, Compression Ratio=4x, Aggregation Mode=Multi2025.10 | 45.41 | |
| Compression-Tokens (Bidirectional)Backbone=Qwen3-1.7B, Compression Ratio=16x, Aggregation Mode=Multi2025.10 | 45.13 | |
| Mean-PoolingBackbone=Qwen3-1.7B, Compression Ratio=16x, Aggregation Mode=Multi2025.10 | 45.07 | |
| PCC LiteBackbone=GPT2-Large & Llama3.1-8B, Compression Ratio=16x, Aggregation Mode=Single2025.10 | 43.67 | |
| Compression-Tokens (Causal)Backbone=Qwen3-0.6B, Compression Ratio=4x, Aggregation Mode=Multi2025.10 | 43.62 | |
| Compression-Tokens (Causal)Backbone=Qwen3-1.7B, Compression Ratio=4x, Aggregation Mode=Single2025.10 | 43.01 | |
| Mean-PoolingBackbone=Llama3.2-1B, Compression Ratio=16x, Aggregation Mode=Multi2025.10 | 42.76 | |
| Compression-Tokens (Bidirectional)Backbone=Qwen3-1.7B, Compression Ratio=16x, Aggregation Mode=Single2025.10 | 42.66 | |
| Compression-Tokens (Bidirectional)Backbone=Llama3.2-1B, Compression Ratio=16x, Aggregation Mode=Multi2025.10 | 41.91 | |
| PCC LargeBackbone=Llama3.1-8B, Compression Ratio=16x, Aggregation Mode=Single2025.10 | 41.79 | |
| Compression-Tokens (Causal)Backbone=Qwen3-1.7B, Compression Ratio=16x, Aggregation Mode=Single2025.10 | 41.31 | |
| Compression-Tokens (Causal)Backbone=Qwen3-1.7B, Compression Ratio=16x, Aggregation Mode=Multi2025.10 | 40.45 | |
| Compression-Tokens (Bidirectional)Backbone=Llama3.2-1B, Compression Ratio=16x, Aggregation Mode=Single2025.10 | 40.44 | |
| Mean-PoolingBackbone=Qwen3-8B, Compression Ratio=128x, Aggregation Mode=Single2025.10 | 39.95 | |
| Compression-Tokens (Bidirectional)Backbone=Qwen3-0.6B, Compression Ratio=16x, Aggregation Mode=Multi2025.10 | 39.5 | |
| Compression-Tokens (Bidirectional)Backbone=Qwen3-8B, Compression Ratio=128x, Aggregation Mode=Multi2025.10 | 39.36 | |
| Compression-Tokens (Causal)Backbone=Llama3.2-1B, Compression Ratio=16x, Aggregation Mode=Single2025.10 | 39.16 | |
| Compression-Tokens (Causal)Backbone=Qwen3-8B, Compression Ratio=128x, Aggregation Mode=Single2025.10 | 39.09 | |
| Mean-PoolingBackbone=Llama3.2-1B, Compression Ratio=16x, Aggregation Mode=Single2025.10 | 38.84 | |
| Compression-Tokens (Bidirectional)Backbone=Qwen3-8B, Compression Ratio=128x, Aggregation Mode=Single2025.10 | 38.72 | |
| Mean-PoolingBackbone=Qwen3-0.6B, Compression Ratio=16x, Aggregation Mode=Multi2025.10 | 38.68 | |
| Compression-Tokens (Causal)Backbone=Llama3.2-1B, Compression Ratio=16x, Aggregation Mode=Multi2025.10 | 38.51 | |
| Mean-PoolingBackbone=Qwen3-0.6B, Compression Ratio=16x, Aggregation Mode=Single2025.10 | 38.3 | |
| Mean-PoolingBackbone=Qwen3-8B, Compression Ratio=128x, Aggregation Mode=Multi2025.10 | 37.8 | |
| Compression-Tokens (Causal)Backbone=Qwen3-8B, Compression Ratio=128x, Aggregation Mode=Multi2025.10 | 37.05 | |
| Mean-PoolingBackbone=Gemma2-2B, Compression Ratio=128x, Aggregation Mode=Single2025.10 | 36.61 | |
| Compression-Tokens (Causal)Backbone=Gemma2-2B, Compression Ratio=128x, Aggregation Mode=Single2025.10 | 36.14 | |
| Compression-Tokens (Bidirectional)Backbone=Gemma2-2B, Compression Ratio=128x, Aggregation Mode=Single2025.10 | 36.14 | |
| Compression-Tokens (Bidirectional)Backbone=Qwen3-0.6B, Compression Ratio=16x, Aggregation Mode=Single2025.10 | 36.11 | |
| Mean-PoolingBackbone=Qwen3-4B, Compression Ratio=128x, Aggregation Mode=Single2025.10 | 35.49 | |
| Mean-PoolingBackbone=Gemma2-2B, Compression Ratio=128x, Aggregation Mode=Multi2025.10 | 34.93 | |
| Compression-Tokens (Causal)Backbone=Qwen3-4B, Compression Ratio=128x, Aggregation Mode=Single2025.10 | 34.83 | |
| Compression-Tokens (Bidirectional)Backbone=Gemma2-2B, Compression Ratio=128x, Aggregation Mode=Multi2025.10 | 34.75 | |
| Compression-Tokens (Bidirectional)Backbone=Qwen3-4B, Compression Ratio=128x, Aggregation Mode=Multi2025.10 | 34.33 | |
| Compression-Tokens (Causal)Backbone=Qwen3-0.6B, Compression Ratio=16x, Aggregation Mode=Multi2025.10 | 34.3 | |
| Compression-Tokens (Causal)Backbone=Gemma2-2B, Compression Ratio=128x, Aggregation Mode=Multi2025.10 | 33.94 | |
| LLMLingua2Backbone=Qwen3-8B, Compression Ratio=4x, Aggregation Mode=Multi2025.10 | 33.63 | |
| Compression-Tokens (Causal)Backbone=Qwen3-4B, Compression Ratio=128x, Aggregation Mode=Multi2025.10 | 33.48 |