List your company
ResearchFormerly WizWand

Commonsense Reasoning on PhysicalQA

81.8AccuracyBaseline
51.608859.446967.28575.1231May 24, 2023Sep 22, 2023Jan 21, 2024May 21, 2024Sep 19, 2024Jan 18, 2025May 19, 2025
Updated 8mo ago

Evaluation results

MethodLinks
BaselineFormat=Baseline, Bit width (b)=16.00, Quantization Aware Training (QAT)=true, Backbone=Llama 3.1 8B2025.0581.8
Tensor RMS + CFormat=Tensor RMS + C, Bit width (b)=3.00, Quantization Aware Training (QAT)=true, Backbone=Llama 3.1 8B2025.0580.6
Batch PartitioningLM=RoBERTa, Avg.=85.14, Speed-up=1.4x2023.0580.2
OK-TransformerLM=RoBERTa, Avg.=84.75, Speed-up=1.0x2023.0580.09
Block AbsmaxFormat=Block Absmax, Bit width (b)=3.25, Quantization Aware Training (QAT)=true, Backbone=Llama 3.1 8B2025.0580
Tensor RMS + SpFormat=Tensor RMS + Sp, Bit width (b)=3.05, Quantization Aware Training (QAT)=true, Backbone=Llama 3.1 8B2025.0579.8
RoBERTaLM=RoBERTa, Avg.=83.952023.0579.76
Channel AbsmaxFormat=Channel Absmax, Bit width (b)=3.00, Quantization Aware Training (QAT)=true, Backbone=Llama 3.1 8B2025.0578.2
Tensor AbsmaxFormat=Tensor Absmax, Bit width (b)=3.00, Quantization Aware Training (QAT)=true, Backbone=Llama 3.1 8B2025.0575.7
Batch PartitioningLM=BERT, Avg.=67.40, Speed-up=1.4x2023.0569.53
OK-TransformerLM=BERT, Avg.=66.56, Speed-up=1.0x2023.0569.09
BERTLM=BERT, Avg.=65.442023.0568.71
Frozen knowledgeLM=BERT, Avg.=65.59, Speed-up=1.4x2023.0568.06
Tensor RMSFormat=Tensor RMS, Bit width (b)=3.00, Quantization Aware Training (QAT)=true, Backbone=Llama 3.1 8B2025.0556.2
Frozen knowledgeLM=RoBERTa, Avg.=80.01, Speed-up=1.5x2023.0552.77