Zero-shot Question Answering on Evaluation Suite (WinoGrande, PiQA, HellaSwag, ARC-E, ARC-C)
69.39WinoGrande AccuracyPocketLLM
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| PocketLLMCompress Ratio=2x~8x, Avg bits=3.98, Fine-tuned=true2025.11 | 69.39 | 78.54 | 57.45 | 76.18 | 43.17 | 64.95 | |
| GPTQCompress Ratio=2x~8x, Avg bits=4.00, Fine-tuned=true2025.11 | 68.19 | 76.61 | 55.44 | 66.2 | 36.77 | 60.64 | |
| PocketLLMCompress Ratio=2x~8x, Avg bits=3.98, Fine-tuned=false2025.11 | 68.19 | 76.93 | 55.01 | 73.99 | 41.38 | 63.1 | |
| VPTQCompress Ratio=~ 10x, Avg bits=3.01, Fine-tuned=true2025.11 | 68 | 77.3 | 56 | 69.1 | 39.3 | 61.72 | |
| GPTVQCompress Ratio=~ 10x, Avg bits=3.13, Fine-tuned=true2025.11 | 67.64 | 77.64 | — | 72.73 | 43.69 | — | |
| PocketLLMCompress Ratio=~ 10x, Avg bits=2.98, Fine-tuned=true2025.11 | 67.4 | 78.13 | 57.17 | 74.12 | 43.52 | 64.07 | |
| AQLMCompress Ratio=2x~8x, Avg bits=4.04, Fine-tuned=true2025.11 | 67.32 | 78.24 | 55.99 | 70.16 | 41.04 | 62.55 | |
| Llama 2-7BCompress Ratio=-, Avg bits=322025.11 | 67.25 | 78.45 | 56.69 | 76.01 | 43.03 | 64.29 | |
| PocketLLMCompress Ratio=~ 16x, Avg bits=2.02, Fine-tuned=true2025.11 | 67.25 | 76.71 | 53.24 | 69.07 | 36.77 | 60.61 | |
| VPTQCompress Ratio=2x~8x, Avg bits=4.01, Fine-tuned=true2025.11 | 67.1 | 78.1 | 55 | 69 | 39.7 | 61.98 | |
| QTIPCompress Ratio=2x~8x, Avg bits=4.01, Fine-tuned=true2025.11 | 67.1 | 78.4 | — | 68.9 | 40 | — | |
| SpQRCompress Ratio=2x~8x, Avg bits=3.98, Fine-tuned=true2025.11 | 66.93 | 78.35 | 56.1 | 69.11 | 39.68 | 62.17 | |
| AQLMCompress Ratio=~ 10x, Avg bits=3.04, Fine-tuned=true2025.11 | 66.93 | 76.88 | 54.12 | 68.06 | 38.4 | 60.88 | |
| QTIPCompress Ratio=~ 10x, Avg bits=3.01, Fine-tuned=true2025.11 | 66.9 | 78.1 | — | 68.1 | 38.9 | — | |
| QuIP#Compress Ratio=2x~8x, Avg bits=4.02, Fine-tuned=true2025.11 | 66.85 | 77.91 | 55.78 | 68.06 | 39.68 | 61.66 | |
| WandaCompress Ratio=2x~8x, Avg bits=8.00, Fine-tuned=false2025.11 | 66.61 | — | 46.45 | 69.15 | 35.84 | — | |
| SoBPCompress Ratio=2x~8x, Avg bits=11.20, Fine-tuned=false2025.11 | 66.22 | 73.5 | — | 59.81 | 37.63 | — | |
| BolacoCompress Ratio=2x~8x, Avg bits=11.20, Fine-tuned=true2025.11 | 65.67 | 75.19 | — | 69.15 | 38.74 | — | |
| AQLMCompress Ratio=~ 16x, Avg bits=2.02, Fine-tuned=true2025.11 | 65.67 | 74.76 | 49.55 | 63.68 | 32.76 | 57.28 | |
| BolacoCompress Ratio=2x~8x, Avg bits=11.20, Fine-tuned=false2025.11 | 64.8 | 71.44 | — | 66.46 | 34.39 | — | |
| QTIPCompress Ratio=~ 16x, Avg bits=2.00, Fine-tuned=true2025.11 | 64.7 | 75.9 | — | 67.25 | 35.7 | — | |
| VPTQCompress Ratio=~ 16x, Avg bits=2.02, Fine-tuned=true2025.11 | 64.33 | 75.19 | 52.08 | 63.8 | 35.24 | 58.13 | |
| LLM-PrunerCompress Ratio=2x~8x, Avg bits=11.20, Fine-tuned=true2025.11 | 64.25 | 76.01 | — | 66.62 | 37.2 | — | |
| DuQuantCompress Ratio=2x~8x, Avg bits=4.00, Fine-tuned=true2025.11 | 63.93 | 75.68 | — | 50 | 37.46 | — | |
| SpQRCompress Ratio=~ 10x, Avg bits=2.98, Fine-tuned=true2025.11 | 63.54 | 74.81 | 51.85 | 67.42 | 37.71 | 59.07 | |
| PocketLLMCompress Ratio=~ 10x, Avg bits=2.98, Fine-tuned=false2025.11 | 63.54 | 75.41 | 51.67 | 69.99 | 37.03 | 59.53 | |
| SoLACompress Ratio=2x~8x, Avg bits=11.20, Fine-tuned=false2025.11 | 62.83 | 69.97 | — | 59.13 | 32.68 | — | |
| QuIP#Compress Ratio=~ 16x, Avg bits=2.02, Fine-tuned=true2025.11 | 62.43 | 71.38 | 42.94 | 55.56 | 28.84 | 52.23 | |
| DuQuantCompress Ratio=2x~8x, Avg bits=4.00, Fine-tuned=false2025.11 | 62.12 | 75.24 | — | 51.89 | 36.77 | — | |
| GPTVQCompress Ratio=~ 16x, Avg bits=2.13, Fine-tuned=true2025.11 | 60.93 | 70.73 | — | 58.08 | 31.48 | — | |
| PocketLLMCompress Ratio=~ 20x, Avg bits=1.60, Fine-tuned=true2025.11 | 60.3 | 73.5 | 47.63 | 63.58 | 32.25 | 55.45 | |
| GPTQCompress Ratio=~ 10x, Avg bits=3.00, Fine-tuned=true2025.11 | 59.19 | 71.49 | 45.21 | 58.46 | 31.06 | 53.08 | |
| LLM-PrunerCompress Ratio=2x~8x, Avg bits=11.20, Fine-tuned=false2025.11 | 58.72 | 71.93 | — | 61.41 | 33.96 | — | |
| PocketLLMCompress Ratio=~ 16x, Avg bits=2.02, Fine-tuned=false2025.11 | 58.72 | 70.95 | 44.02 | 58.16 | 30.38 | 52.45 | |
| PocketLLMCompress Ratio=~ 20x, Avg bits=1.60, Fine-tuned=false2025.11 | 52.01 | 54.08 | 27.2 | 33.04 | 23.21 | 37.91 |