Exact Match on Big-bench Hard (BBH)
53.53Exact Match (EM)AdmTree
Evaluation results
| Method | Links | |
|---|---|---|
| AdmTreeCompression Strategy=quarter-st. const., Compression Ratio=~7 x2025.12 | 53.53 | |
| AdmTreeCompression Strategy=half-st. const., Compression Ratio=~5 x2025.12 | 49.7 | |
| AdmTreeCompression Strategy=1-st. const., Compression Ratio=~3 x2025.12 | 45.06 | |
| BeaconCompression Strategy=half-st. const., Compression Ratio=~5 x2025.12 | 44.25 | |
| BeaconCompression Strategy=1-st. const., Compression Ratio=~3 x2025.12 | 43.95 | |
| BaselineEvaluation protocol=Full-shot2025.12 | 43.34 | |
| BeaconCompression Strategy=quarter-st. const., Compression Ratio=~7 x2025.12 | 42.44 | |
| LongLLMLinguaCompression Strategy=half-st. const., Compression Ratio=~5 x2025.12 | 39.42 | |
| LongLLMLinguaCompression Strategy=quarter-st. const., Compression Ratio=~7 x2025.12 | 38.81 | |
| SnapkvCompression Strategy=1-st. const., Compression Ratio=~3 x2025.12 | 38.61 | |
| SnapkvCompression Strategy=half-st. const., Compression Ratio=~5 x2025.12 | 36.83 | |
| BaselineEvaluation protocol=Zero-shot2025.12 | 36.49 | |
| SnapkvCompression Strategy=quarter-st. const., Compression Ratio=~7 x2025.12 | 36.01 | |
| LongLLMLinguaCompression Strategy=1-st. const., Compression Ratio=~3 x2025.12 | 33.37 | |
| ICAECompression Strategy=1-st. const., Compression Ratio=~3 x2025.12 | 21.67 | |
| ICAECompression Strategy=half-st. const., Compression Ratio=~5 x2025.12 | 20.12 | |
| ICAECompression Strategy=quarter-st. const., Compression Ratio=~7 x2025.12 | 19.83 | |
| AutoCompressorCompression Strategy=1-st. const., Compression Ratio=~3 x2025.12 | 13.71 | |
| AutoCompressorCompression Strategy=half-st. const., Compression Ratio=~5 x2025.12 | 11.56 | |
| AutoCompressorCompression Strategy=quarter-st. const., Compression Ratio=~7 x2025.12 | 11.23 |