Commonsense Reasoning on WinoGrande (val)
73.88AccuracyFP16
Evaluation Results
| Method | Links | |
|---|---|---|
| FP16Backbone=Mistral-7b, Bits Per FPN=162024.05 | 73.88 | |
| KVQuant-4bBackbone=Mistral-7b, Bits Per FPN=4.00-4.022024.05 | 73.88 | |
| KVQuant-4b-1%Backbone=Mistral-7b, Bits Per FPN=4.32-4.352024.05 | 73.72 | |
| CQ-2c8bBackbone=Mistral-7b, Bits Per FPN=4.002024.05 | 73.48 | |
| KVQuant-4b-1%Backbone=LLaMA-13b, Bits Per FPN=4.32-4.352024.05 | 73.4 | |
| FP16Backbone=LLaMA-13b, Bits Per FPN=162024.05 | 72.69 | |
| KVQuant-4bBackbone=LLaMA-13b, Bits Per FPN=4.00-4.022024.05 | 72.61 | |
| CQ-2c8bBackbone=LLaMA-2-13b, Bits Per FPN=4.002024.05 | 72.53 | |
| CQ-2c8bBackbone=LLaMA-13b, Bits Per FPN=4.002024.05 | 72.45 | |
| KVQuant-4b-1%Backbone=LLaMA-2-13b, Bits Per FPN=4.32-4.352024.05 | 72.3 | |
| FP16Backbone=LLaMA-2-13b, Bits Per FPN=162024.05 | 71.98 | |
| KVQuant-4bBackbone=LLaMA-2-13b, Bits Per FPN=4.00-4.022024.05 | 71.59 | |
| KVQuant-2b-1%Backbone=LLaMA-13b, Bits Per FPN=2.32-2.352024.05 | 71.43 | |
| KVQuant-2b-1%Backbone=Mistral-7b, Bits Per FPN=2.32-2.352024.05 | 70.8 | |
| KVQuant-4b-1%Backbone=LLaMA-7b, Bits Per FPN=4.32-4.352024.05 | 70.72 | |
| CQ-4c8bBackbone=LLaMA-13b, Bits Per FPN=2.002024.05 | 70.72 | |
| CQ-2c8bBackbone=LLaMA-7b, Bits Per FPN=4.002024.05 | 70.4 | |
| KVQuant-2b-1%Backbone=LLaMA-2-13b, Bits Per FPN=2.32-2.352024.05 | 70.17 | |
| FP16Backbone=LLaMA-7b, Bits Per FPN=162024.05 | 69.93 | |
| KVQuant-4bBackbone=LLaMA-7b, Bits Per FPN=4.00-4.022024.05 | 69.53 | |
| CQ-4c8bBackbone=Mistral-7b, Bits Per FPN=2.002024.05 | 69.38 | |
| CQ-4c8bBackbone=LLaMA-2-13b, Bits Per FPN=2.002024.05 | 69.06 | |
| FP16Backbone=LLaMA-2-7b, Bits Per FPN=162024.05 | 68.9 | |
| Qwen-1.5 14B (Teacher)Model Scale=14B2024.07 | 68.74 | |
| KVQuant-4b-1%Backbone=LLaMA-2-7b, Bits Per FPN=4.32-4.352024.05 | 68.67 | |
| CQ-2c8bBackbone=LLaMA-2-7b, Bits Per FPN=4.002024.05 | 68.27 | |
| KVQuant-2b-1%Backbone=LLaMA-7b, Bits Per FPN=2.32-2.352024.05 | 68.03 | |
| KVQuant-4bBackbone=LLaMA-2-7b, Bits Per FPN=4.00-4.022024.05 | 67.96 | |
| KVQuant-2b-1%Backbone=LLaMA-2-7b, Bits Per FPN=2.32-2.352024.05 | 67.64 | |
| CQ-4c8bBackbone=LLaMA-7b, Bits Per FPN=2.002024.05 | 67.48 | |
| CQ-4c8bBackbone=LLaMA-2-7b, Bits Per FPN=2.002024.05 | 66.45 | |
| CQ-8c10bBackbone=LLaMA-13b, Bits Per FPN=1.252024.05 | 65.27 | |
| D2-LoRABackbone=Llama-3.2-3B-Instruct, Training Epochs=2, Scoring Method=CLL2026.02 | 64 | |
| CQ-8c10bBackbone=Mistral-7b, Bits Per FPN=1.252024.05 | 63.93 | |
| LLaMA2-13BModel Size=13B, Shots=0-shot2024.07 | 63.77 | |
| LoRABackbone=Qwen2.5-7B-Instruct, Training Epochs=1, Scoring Method=CLL2026.02 | 63.6 | |
| KVQuant-2bBackbone=Mistral-7b, Bits Per FPN=2.00-2.022024.05 | 63.46 | |
| D2-LoRABackbone=Qwen2.5-7B-Instruct, Training Epochs=1, Scoring Method=CLL2026.02 | 63 | |
| CQ-8c10bBackbone=LLaMA-2-13b, Bits Per FPN=1.252024.05 | 62.98 | |
| DoRABackbone=Llama-3.2-3B-Instruct, Training Epochs=2, Scoring Method=CLL2026.02 | 62.6 | |
| DoRABackbone=Qwen2.5-7B-Instruct, Training Epochs=1, Scoring Method=CLL2026.02 | 61.6 | |
| CQ-8c8bBackbone=LLaMA-13b, Bits Per FPN=1.002024.05 | 61.56 | |
| KVQuant-1b-1%Backbone=LLaMA-13b, Bits Per FPN=1.32-1.352024.05 | 61.01 | |
| CQ-8c10bBackbone=LLaMA-7b, Bits Per FPN=1.252024.05 | 60.46 | |
| KVQuant-2bBackbone=LLaMA-13b, Bits Per FPN=2.00-2.022024.05 | 59.35 | |
| LoRABackbone=Llama-3.2-3B-Instruct, Training Epochs=2, Scoring Method=CLL2026.02 | 59.2 | |
| CQ-8c10bBackbone=LLaMA-2-7b, Bits Per FPN=1.252024.05 | 59.19 | |
| Qwen-1.5 1.8B + DDKDistillation=DDK2024.07 | 59.1 | |
| CQ-8c8bBackbone=Mistral-7b, Bits Per FPN=1.002024.05 | 58.25 | |
| KVQuant-1b-1%Backbone=Mistral-7b, Bits Per FPN=1.32-1.352024.05 | 58.17 | |
| Qwen-1.5 1.8B + KDDistillation=KD2024.07 | 58.01 | |
| Qwen-1.5 1.8B (Student)Model Scale=1.8B2024.07 | 57.85 | |
| KVQuant-1b-1%Backbone=LLaMA-2-7b, Bits Per FPN=1.32-1.352024.05 | 57.77 | |
| TinyLLaMA-1.1B + DDKModel Size=1.1B, Shots=0-shot2024.07 | 57.62 | |
| Qwen-1.5 1.8B + TEDDistillation=TED2024.07 | 57.38 | |
| KVQuant-1b-1%Backbone=LLaMA-2-13b, Bits Per FPN=1.32-1.352024.05 | 57.3 | |
| CQ-8c8bBackbone=LLaMA-2-13b, Bits Per FPN=1.002024.05 | 57.14 | |
| Qwen-1.5 1.8B + MiniLLMDistillation=MiniLLM2024.07 | 57.14 | |
| Qwen-1.5 1.8B + CPTDistillation=CPT2024.07 | 56.98 | |
| Qwen-1.5 1.8B + CPT & DoReMiDistillation=CPT & DoReMi2024.07 | 56.75 | |
| KVQuant-1b-1%Backbone=LLaMA-7b, Bits Per FPN=1.32-1.352024.05 | 56.67 | |
| CQ-8c8bBackbone=LLaMA-7b, Bits Per FPN=1.002024.05 | 56.51 | |
| TinyLLaMA-1.1B + CPTModel Size=1.1B, Shots=0-shot2024.07 | 56.2 | |
| TinyLLaMA-1.1B + TEDModel Size=1.1B, Shots=0-shot2024.07 | 55.72 | |
| TinyLLaMA-1.1BModel Size=1.1B, Shots=0-shot2024.07 | 55.49 | |
| TinyLLaMA-1.1B + CPT & DoReMiModel Size=1.1B, Shots=0-shot2024.07 | 55.25 | |
| CQ-8c8bBackbone=LLaMA-2-7b, Bits Per FPN=1.002024.05 | 55.01 | |
| TinyLLaMA-1.1B + MiniLLMModel Size=1.1B, Shots=0-shot2024.07 | 54.46 | |
| Mistral (Full-Attention)Model Scale=1.4B2024.07 | 54.4 | |
| TinyLLaMA-1.1B + KDModel Size=1.1B, Shots=0-shot2024.07 | 53.91 | |
| KVQuant-2bBackbone=LLaMA-7b, Bits Per FPN=2.00-2.022024.05 | 53.59 | |
| BMoJo (Fading)Model Scale=1.4B2024.07 | 53.3 | |
| Hybrid (Sliding Attention + SSM)Model Scale=1.4B2024.07 | 52.6 | |
| BMoJo (Fading + Eidetic)Model Scale=1.4B2024.07 | 52.1 | |
| Hybrid (Sliding Attention + SSM)Model Scale=370M2024.07 | 51.9 | |
| Mamba (SSM)Model Scale=1.4B2024.07 | 51.9 | |
| BMoJo (Fading)Model Scale=370M2024.07 | 51.8 | |
| KVQuant-2bBackbone=LLaMA-2-7b, Bits Per FPN=2.00-2.022024.05 | 51.7 | |
| Mamba (SSM)Model Scale=370M2024.07 | 51.7 | |
| KVQuant-2bBackbone=LLaMA-2-13b, Bits Per FPN=2.00-2.022024.05 | 51.3 | |
| KVQuant-1bBackbone=LLaMA-2-7b, Bits Per FPN=1.00-1.022024.05 | 50.91 | |
| BMoJo (Fading + Eidetic)Model Scale=370M2024.07 | 50.7 | |
| KVQuant-1bBackbone=LLaMA-7b, Bits Per FPN=1.00-1.022024.05 | 50.51 | |
| Mistral (Full-Attention)Model Scale=370M2024.07 | 50.4 | |
| KVQuant-1bBackbone=Mistral-7b, Bits Per FPN=1.00-1.022024.05 | 49.8 | |
| KVQuant-1bBackbone=LLaMA-2-13b, Bits Per FPN=1.00-1.022024.05 | 49.41 | |
| KVQuant-1bBackbone=LLaMA-13b, Bits Per FPN=1.00-1.022024.05 | 48.46 |