List your company
ResearchFormerly WizWand

Massive Multitask Language Understanding on MMLU (5-shot)

59.69MMLU (5-shot)FP16
23.591632.963342.33551.7067Jun 5, 2025
Updated 3mo ago

Evaluation results

MethodLinks
FP16Model=Llama-3.2 3B-it, # Bits (W-A-KV)=16-16-162025.0659.69
FlatQuantModel=Llama-3.2 3B-it, # Bits (W-A-KV)=4-8-42025.0655.46
FPTQuantModel=Llama-3.2 3B-it, # Bits (W-A-KV)=4-8-42025.0652.62
SpinQuantModel=Llama-3.2 3B-it, # Bits (W-A-KV)=4-8-42025.0652.14
QuaRotModel=Llama-3.2 3B-it, # Bits (W-A-KV)=4-8-42025.0651.68
FlatQuantModel=Llama-3.2 3B-it, # Bits (W-A-KV)=4-4-42025.0649.01
RTN-optModel=Llama-3.2 3B-it, # Bits (W-A-KV)=4-8-42025.0646.45
FPTQuantModel=Llama-3.2 3B-it, # Bits (W-A-KV)=4-4-42025.0643.71
SpinQuantModel=Llama-3.2 3B-it, # Bits (W-A-KV)=4-4-42025.0638.17
QuaRotModel=Llama-3.2 3B-it, # Bits (W-A-KV)=4-4-42025.0635.72
RTN-optModel=Llama-3.2 3B-it, # Bits (W-A-KV)=4-4-42025.0624.98