Language Understanding on MMLU (5-shot, test)
74.2AccuracyFP16
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| FP16Fine-tuned=false, PTQ=false, W-bit=16, Pruning Ratio=0%, Backbone=Qwen-2.5-7B2025.12 | 74.2 | — | — | — | — | — | — | |
| TRT-AWQFine-tuned=false, PTQ=true, W-bit=4*, Pruning Ratio=0%, Backbone=Qwen-2.5-7B2025.12 | 72.5 | — | — | — | — | — | — | |
| TAO-HQQFine-tuned=false, PTQ=true, W-bit=4*, Pruning Ratio=0%, Backbone=Qwen-2.5-7B2025.12 | 72.5 | — | — | — | — | — | — | |
| Uniform Regularized LoopingModel=Gemma, Size=9B, Shots=5-shot2026.02 | 72.42 | — | — | — | — | — | — | |
| Moving Average Regularized LoopingModel=Gemma, Size=9B, Shots=5-shot2026.02 | 72.23 | — | — | — | — | — | — | |
| Noise AblationModel=Gemma, Size=9B, Shots=5-shot2026.02 | 72.2 | — | — | — | — | — | — | |
| Auto-Align Regularized LoopingModel=Gemma, Size=9B, Shots=5-shot2026.02 | 72.18 | — | — | — | — | — | — | |
| BaselineModel=Gemma, Size=9B, Shots=5-shot2026.02 | 72.17 | — | — | — | — | — | — | |
| GradientBackbone=Qwen2-7b, Fine-tuning Dataset=Alpaca, Trainable Parameters Percentage=0.53%, Evaluation Protocol=5-shot2024.12 | 70.96 | — | — | — | — | — | — | |
| MagnitudeBackbone=Qwen2-7b, Fine-tuning Dataset=OASST2, Trainable Parameters Percentage=0.53%, Evaluation Protocol=5-shot2024.12 | 70.76 | — | — | — | — | — | — | |
| SynFlowBackbone=Qwen2-7b, Fine-tuning Dataset=OASST2, Trainable Parameters Percentage=0.53%, Evaluation Protocol=5-shot2024.12 | 70.66 | — | — | — | — | — | — | |
| GradientBackbone=Qwen2-7b, Fine-tuning Dataset=OASST2, Trainable Parameters Percentage=0.53%, Evaluation Protocol=5-shot2024.12 | 70.55 | — | — | — | — | — | — | |
| FORCEBackbone=Qwen2-7b, Fine-tuning Dataset=OASST2, Trainable Parameters Percentage=0.53%, Evaluation Protocol=5-shot2024.12 | 70.53 | — | — | — | — | — | — | |
| SNIPBackbone=Qwen2-7b, Fine-tuning Dataset=OASST2, Trainable Parameters Percentage=0.53%, Evaluation Protocol=5-shot2024.12 | 70.52 | — | — | — | — | — | — | |
| GPTQFine-tuned=false, PTQ=true, W-bit=4, Pruning Ratio=0%, Backbone=Qwen-2.5-7B2025.12 | 70.5 | — | — | — | — | — | — | |
| LoRABackbone=Qwen2-7b, Fine-tuning Dataset=OASST2, Trainable Parameters Percentage=0.53%, Evaluation Protocol=5-shot2024.12 | 70.42 | — | — | — | — | — | — | |
| UniQLFine-tuned=false, PTQ=true, W-bit=4, Pruning Ratio=0%, Backbone=Qwen-2.5-7B2025.12 | 70.3 | — | — | — | — | — | — | |
| MagnitudeBackbone=Qwen2-7b, Fine-tuning Dataset=Alpaca, Trainable Parameters Percentage=0.53%, Evaluation Protocol=5-shot2024.12 | 70.12 | — | — | — | — | — | — | |
| GRASPBackbone=Qwen2-7b, Fine-tuning Dataset=OASST2, Trainable Parameters Percentage=0.53%, Evaluation Protocol=5-shot2024.12 | 69.91 | — | — | — | — | — | — | |
| SynFlowBackbone=Qwen2-7b, Fine-tuning Dataset=Alpaca, Trainable Parameters Percentage=0.53%, Evaluation Protocol=5-shot2024.12 | 69.8 | — | — | — | — | — | — | |
| LoRABackbone=Qwen2-7b, Fine-tuning Dataset=Alpaca, Trainable Parameters Percentage=0.53%, Evaluation Protocol=5-shot2024.12 | 69.77 | — | — | — | — | — | — | |
| Taylor-FOBackbone=Qwen2-7b, Fine-tuning Dataset=OASST2, Trainable Parameters Percentage=0.53%, Evaluation Protocol=5-shot2024.12 | 69.1 | — | — | — | — | — | — | |
| Fisher-InfoBackbone=Qwen2-7b, Fine-tuning Dataset=OASST2, Trainable Parameters Percentage=0.53%, Evaluation Protocol=5-shot2024.12 | 69.1 | — | — | — | — | — | — | |
| FORCEBackbone=Qwen2-7b, Fine-tuning Dataset=Alpaca, Trainable Parameters Percentage=0.53%, Evaluation Protocol=5-shot2024.12 | 69.01 | — | — | — | — | — | — | |
| SNIPBackbone=Qwen2-7b, Fine-tuning Dataset=Alpaca, Trainable Parameters Percentage=0.53%, Evaluation Protocol=5-shot2024.12 | 68.75 | — | — | — | — | — | — | |
| Taylor-FOBackbone=Qwen2-7b, Fine-tuning Dataset=Alpaca, Trainable Parameters Percentage=0.53%, Evaluation Protocol=5-shot2024.12 | 68.75 | — | — | — | — | — | — | |
| FP16q (mean bit precision)=16.000, tok/s (throughput)=103.8, Evaluation Protocol=5-shot2025.12 | 68.39 | — | — | — | — | — | — | |
| FP16Fine-tuned=false, PTQ=false, W-bit=16, Pruning Ratio=0%, Backbone=Nemotron-H-8B2025.12 | 67.6 | — | — | — | — | — | — | |
| UniQLFine-tuned=false, PTQ=true, W-bit=4, Pruning Ratio=0%, Backbone=Nemotron-H-8B2025.12 | 67.5 | — | — | — | — | — | — | |
| GRASPBackbone=Qwen2-7b, Fine-tuning Dataset=Alpaca, Trainable Parameters Percentage=0.53%, Evaluation Protocol=5-shot2024.12 | 66.69 | — | — | — | — | — | — | |
| Fisher-InfoBackbone=Qwen2-7b, Fine-tuning Dataset=Alpaca, Trainable Parameters Percentage=0.53%, Evaluation Protocol=5-shot2024.12 | 66.45 | — | — | — | — | — | — | |
| FP16Fine-tuned=false, PTQ=false, W-bit=16, Pruning Ratio=0%, Backbone=Llama-3.1-8B2025.12 | 65.6 | — | — | — | — | — | — | |
| GPTQFine-tuned=false, PTQ=true, W-bit=4, Pruning Ratio=0%, Backbone=Nemotron-H-8B2025.12 | 64 | — | — | — | — | — | — | |
| UniQLFine-tuned=false, PTQ=true, W-bit=4, Pruning Ratio=0%, Backbone=Llama-3.1-8B2025.12 | 63.2 | — | — | — | — | — | — | |
| TRT-AWQFine-tuned=false, PTQ=true, W-bit=4*, Pruning Ratio=0%, Backbone=Llama-3.1-8B2025.12 | 63 | — | — | — | — | — | — | |
| TAO-HQQFine-tuned=false, PTQ=true, W-bit=4*, Pruning Ratio=0%, Backbone=Llama-3.1-8B2025.12 | 62.9 | — | — | — | — | — | — | |
| GPTQFine-tuned=false, PTQ=true, W-bit=4, Pruning Ratio=0%, Backbone=Llama-3.1-8B2025.12 | 61.5 | — | — | — | — | — | — | |
| SVD-LLMFine-tuned=true, PTQ=false, W-bit=16, Pruning Ratio=15%, Backbone=Qwen-2.5-7B2025.12 | 61.1 | — | — | — | — | — | — | |
| AQLM-1x16 + PV-Tuningq (mean bit precision)=2.213, tok/s (throughput)=49.0, PV-Tuning=true, Evaluation Protocol=5-shot2025.12 | 60.72 | — | — | — | — | — | — | |
| UniQLFine-tuned=false, PTQ=false, W-bit=16, Pruning Ratio=15%, Backbone=Llama-3.1-8B2025.12 | 60.2 | — | — | — | — | — | — | |
| UniQLFine-tuned=true, PTQ=false, W-bit=16, Pruning Ratio=15%, Backbone=Qwen-2.5-7B2025.12 | 59.9 | — | — | — | — | — | — | |
| MoDeGPTFine-tuned=false, PTQ=false, W-bit=16, Pruning Ratio=15%, Backbone=Llama-3.1-8B2025.12 | 59.5 | — | — | — | — | — | — | |
| UniQLFine-tuned=true, PTQ=false, W-bit=16, Pruning Ratio=15%, Backbone=Llama-3.1-8B2025.12 | 59.2 | — | — | — | — | — | — | |
| AQLM-1x16q (mean bit precision)=2.213, tok/s (throughput)=49.0, Evaluation Protocol=5-shot2025.12 | 58.74 | — | — | — | — | — | — | |
| CodeGEMM-m1v4g128 + PV-Tuningq (mean bit precision)=2.126, tok/s (throughput)=228.3, PV-Tuning=true, Evaluation Protocol=5-shot2025.12 | 57.42 | — | — | — | — | — | — | |
| UniQLFine-tuned=true, PTQ=true, W-bit=4, Pruning Ratio=15%, Backbone=Llama-3.1-8B2025.12 | 56.9 | — | — | — | — | — | — | |
| SVD-LLMFine-tuned=true, PTQ=true, W-bit=4*, Pruning Ratio=15%, Backbone=Qwen-2.5-7B2025.12 | 56.3 | — | — | — | — | — | — | |
| UniQLFine-tuned=true, PTQ=false, W-bit=16, Pruning Ratio=15%, Backbone=Nemotron-H-8B2025.12 | 56.1 | — | — | — | — | — | — | |
| UniQLFine-tuned=false, PTQ=false, W-bit=16, Pruning Ratio=15%, Backbone=Qwen-2.5-7B2025.12 | 55.9 | — | — | — | — | — | — | |
| CodeGEMM-m2v8g128 + PV-Tuningq (mean bit precision)=2.127, tok/s (throughput)=214.4, PV-Tuning=true, Evaluation Protocol=5-shot2025.12 | 55.42 | — | — | — | — | — | — | |
| AQLM-2x8 + PV-Tuningq (mean bit precision)=2.005, tok/s (throughput)=124.5, PV-Tuning=true, Evaluation Protocol=5-shot2025.12 | 55.13 | — | — | — | — | — | — | |
| Uniform Regularized LoopingModel=Gemma, Size=2B, Shots=5-shot2026.02 | 54.49 | — | — | — | — | — | — | |
| Moving Average Regularized LoopingModel=Gemma, Size=2B, Shots=5-shot2026.02 | 54.35 | — | — | — | — | — | — | |
| Auto-Align Regularized LoopingModel=Gemma, Size=2B, Shots=5-shot2026.02 | 54.13 | — | — | — | — | — | — | |
| Noise AblationModel=Gemma, Size=2B, Shots=5-shot2026.02 | 53.97 | — | — | — | — | — | — | |
| BaselineModel=Gemma, Size=2B, Shots=5-shot2026.02 | 53.93 | — | — | — | — | — | — | |
| GradientBackbone=Gemma2-2b, Fine-tuning Dataset=Alpaca, Trainable Parameters Percentage=0.97%, Evaluation Protocol=5-shot2024.12 | 53.11 | — | — | — | — | — | — | |
| GradientBackbone=Gemma2-2b, Fine-tuning Dataset=OASST2, Trainable Parameters Percentage=0.97%, Evaluation Protocol=5-shot2024.12 | 53.11 | — | — | — | — | — | — | |
| LoRABackbone=Gemma2-2b, Fine-tuning Dataset=Alpaca, Trainable Parameters Percentage=0.97%, Evaluation Protocol=5-shot2024.12 | 53.07 | — | — | — | — | — | — | |
| SynFlowBackbone=Gemma2-2b, Fine-tuning Dataset=OASST2, Trainable Parameters Percentage=0.97%, Evaluation Protocol=5-shot2024.12 | 53.07 | — | — | — | — | — | — | |
| MagnitudeBackbone=Gemma2-2b, Fine-tuning Dataset=OASST2, Trainable Parameters Percentage=0.97%, Evaluation Protocol=5-shot2024.12 | 53.03 | — | — | — | — | — | — | |
| MagnitudeBackbone=Gemma2-2b, Fine-tuning Dataset=Alpaca, Trainable Parameters Percentage=0.97%, Evaluation Protocol=5-shot2024.12 | 52.97 | — | — | — | — | — | — | |
| Taylor-FOBackbone=Gemma2-2b, Fine-tuning Dataset=OASST2, Trainable Parameters Percentage=0.97%, Evaluation Protocol=5-shot2024.12 | 52.96 | — | — | — | — | — | — | |
| SNIPBackbone=Gemma2-2b, Fine-tuning Dataset=OASST2, Trainable Parameters Percentage=0.97%, Evaluation Protocol=5-shot2024.12 | 52.89 | — | — | — | — | — | — | |
| FORCEBackbone=Gemma2-2b, Fine-tuning Dataset=OASST2, Trainable Parameters Percentage=0.97%, Evaluation Protocol=5-shot2024.12 | 52.88 | — | — | — | — | — | — | |
| SynFlowBackbone=Gemma2-2b, Fine-tuning Dataset=Alpaca, Trainable Parameters Percentage=0.97%, Evaluation Protocol=5-shot2024.12 | 52.84 | — | — | — | — | — | — | |
| SNIPBackbone=Gemma2-2b, Fine-tuning Dataset=Alpaca, Trainable Parameters Percentage=0.97%, Evaluation Protocol=5-shot2024.12 | 52.81 | — | — | — | — | — | — | |
| Taylor-FOBackbone=Gemma2-2b, Fine-tuning Dataset=Alpaca, Trainable Parameters Percentage=0.97%, Evaluation Protocol=5-shot2024.12 | 52.81 | — | — | — | — | — | — | |
| FORCEBackbone=Gemma2-2b, Fine-tuning Dataset=Alpaca, Trainable Parameters Percentage=0.97%, Evaluation Protocol=5-shot2024.12 | 52.79 | — | — | — | — | — | — | |
| Fisher-InfoBackbone=Gemma2-2b, Fine-tuning Dataset=Alpaca, Trainable Parameters Percentage=0.97%, Evaluation Protocol=5-shot2024.12 | 52.7 | — | — | — | — | — | — | |
| UniQLFine-tuned=true, PTQ=true, W-bit=4, Pruning Ratio=15%, Backbone=Qwen-2.5-7B2025.12 | 52.7 | — | — | — | — | — | — | |
| Fisher-InfoBackbone=Gemma2-2b, Fine-tuning Dataset=OASST2, Trainable Parameters Percentage=0.97%, Evaluation Protocol=5-shot2024.12 | 52.65 | — | — | — | — | — | — | |
| GRASPBackbone=Gemma2-2b, Fine-tuning Dataset=OASST2, Trainable Parameters Percentage=0.97%, Evaluation Protocol=5-shot2024.12 | 52.6 | — | — | — | — | — | — | |
| UniQLFine-tuned=true, PTQ=true, W-bit=4, Pruning Ratio=15%, Backbone=Nemotron-H-8B2025.12 | 52.6 | — | — | — | — | — | — | |
| LoRABackbone=Gemma2-2b, Fine-tuning Dataset=OASST2, Trainable Parameters Percentage=0.97%, Evaluation Protocol=5-shot2024.12 | 52.59 | — | — | — | — | — | — | |
| GRASPBackbone=Gemma2-2b, Fine-tuning Dataset=Alpaca, Trainable Parameters Percentage=0.97%, Evaluation Protocol=5-shot2024.12 | 52.38 | — | — | — | — | — | — | |
| SVD-LLMFine-tuned=false, PTQ=false, W-bit=16, Pruning Ratio=15%, Backbone=Qwen-2.5-7B2025.12 | 49.9 | — | — | — | — | — | — | |
| EfficientQATBits=4, Group=64, Model Size=13B2024.07 | 49.5 | — | — | — | — | — | — | |
| IR-QLoRABits=4, Group=64, Model Size=13B2024.07 | 49.3 | — | — | — | — | — | — | |
| QA-LoRABits=4, Group=32, Model Size=13B2024.07 | 49.2 | — | — | — | — | — | — | |
| QLoRABits=4+16, Model Size=13B2024.07 | 48.4 | — | — | — | — | — | — | |
| EfficientQATBits=4, Group=-1, Model Size=13B2024.07 | 48.2 | — | — | — | — | — | — | |
| EfficientQATBits=3, Group=64, Model Size=13B2024.07 | 48.2 | — | — | — | — | — | — | |
| VeLoRALLaMA Size=13B, Fine-tuning Dataset=Alpaca, Number of shots=5-shot, Base Method=QLoRA2024.05 | 48 | 8.48 | — | — | — | — | — | |
| QLoRA w/ GPTQBits=4, Group=32, Model Size=13B2024.07 | 48 | — | — | — | — | — | — | |
| QLoRALLaMA Size=13B, Fine-tuning Dataset=Alpaca, Number of shots=5-shot2024.05 | 47.5 | 9.91 | — | — | — | — | — | |
| PEQABits=4, Group=64, Model Size=13B2024.07 | 47.4 | — | — | — | — | — | — | |
| LoRA w/ Float4LLaMA Size=13B, Fine-tuning Dataset=Alpaca, Precision=Float4, Number of shots=5-shot2024.05 | 47.3 | 9.91 | — | — | — | — | — | |
| QA-LoRABits=3, Group=32, Model Size=13B2024.07 | 47.3 | — | — | — | — | — | — | |
| LoRA w/ BFloat16LLaMA Size=13B, Fine-tuning Dataset=Alpaca, Precision=BFloat16, Number of shots=5-shot2024.05 | 47.2 | 15.82 | — | — | — | — | — | |
| LLaMASize=13B, 5-shot=true2023.05 | 46.9 | — | — | — | — | — | — | |
| FP16Bits=16, Model Size=13B, Evaluation Protocol=5-shot, Instruction-tuning=Alpaca2024.07 | 46.3 | — | — | — | — | — | — | |
| PEQABits=3, Group=64, Model Size=13B2024.07 | 46.3 | — | — | — | — | — | — | |
| QLoRA w/ GPTQBits=3, Group=32, Model Size=13B2024.07 | 46.1 | — | — | — | — | — | — | |
| LLaMA2-7bPrune%=0%, #Params=6.7B, shots=5-shot2024.12 | 45.6 | — | 43.3 | 51.6 | 36.3 | 52.1 | — | |
| CodeGEMM-m1v4g128q (mean bit precision)=2.126, tok/s (throughput)=228.3, Evaluation Protocol=5-shot2025.12 | 45.16 | — | — | — | — | — | — | |
| PEQABits=4, Group=-1, Model Size=13B2024.07 | 45 | — | — | — | — | — | — | |
| VeLoRALLaMA Size=Mean, Fine-tuning Dataset=Alpaca, Number of shots=5-shot, Base Method=QLoRA2024.05 | 43.8 | — | — | — | — | — | — | |
| QLoRALLaMA Size=Mean, Fine-tuning Dataset=Alpaca, Number of shots=5-shot2024.05 | 43.3 | — | — | — | — | — | — | |
| CodeGEMM-m2v8g128q (mean bit precision)=2.127, tok/s (throughput)=214.4, Evaluation Protocol=5-shot2025.12 | 42.83 | — | — | — | — | — | — |