Commonsense Reasoning on PhysicalQA
81.8AccuracyBaseline
Evaluation Results
| Method | Links | |
|---|---|---|
| BaselineFormat=Baseline, Bit width (b)=16.00, Quantization Aware Training (QAT)=true, Backbone=Llama 3.1 8B2025.05 | 81.8 | |
| Tensor RMS + CFormat=Tensor RMS + C, Bit width (b)=3.00, Quantization Aware Training (QAT)=true, Backbone=Llama 3.1 8B2025.05 | 80.6 | |
| Batch PartitioningLM=RoBERTa, Avg.=85.14, Speed-up=1.4x2023.05 | 80.2 | |
| OK-TransformerLM=RoBERTa, Avg.=84.75, Speed-up=1.0x2023.05 | 80.09 | |
| Block AbsmaxFormat=Block Absmax, Bit width (b)=3.25, Quantization Aware Training (QAT)=true, Backbone=Llama 3.1 8B2025.05 | 80 | |
| Tensor RMS + SpFormat=Tensor RMS + Sp, Bit width (b)=3.05, Quantization Aware Training (QAT)=true, Backbone=Llama 3.1 8B2025.05 | 79.8 | |
| RoBERTaLM=RoBERTa, Avg.=83.952023.05 | 79.76 | |
| Channel AbsmaxFormat=Channel Absmax, Bit width (b)=3.00, Quantization Aware Training (QAT)=true, Backbone=Llama 3.1 8B2025.05 | 78.2 | |
| Tensor AbsmaxFormat=Tensor Absmax, Bit width (b)=3.00, Quantization Aware Training (QAT)=true, Backbone=Llama 3.1 8B2025.05 | 75.7 | |
| Batch PartitioningLM=BERT, Avg.=67.40, Speed-up=1.4x2023.05 | 69.53 | |
| OK-TransformerLM=BERT, Avg.=66.56, Speed-up=1.0x2023.05 | 69.09 | |
| BERTLM=BERT, Avg.=65.442023.05 | 68.71 | |
| Frozen knowledgeLM=BERT, Avg.=65.59, Speed-up=1.4x2023.05 | 68.06 | |
| Tensor RMSFormat=Tensor RMS, Bit width (b)=3.00, Quantization Aware Training (QAT)=true, Backbone=Llama 3.1 8B2025.05 | 56.2 | |
| Frozen knowledgeLM=RoBERTa, Avg.=80.01, Speed-up=1.5x2023.05 | 52.77 |