List your company
ResearchFormerly WizWand

Question Answering on ARC-E (0-shot)

87.88AccuracyMoving Average Regularized Looping
22.224839.269956.31573.3601Sep 30, 2025Nov 2, 2025Dec 6, 2025Jan 9, 2026Feb 12, 2026Mar 18, 2026Apr 21, 2026
Updated 6mo ago

Evaluation results

MethodLinks
Moving Average Regularized LoopingModel=Gemma, Size=9B, Shots=0-shot2026.0287.88
BaselineModel=Gemma, Size=9B, Shots=0-shot2026.0287.84
Uniform Regularized LoopingModel=Gemma, Size=9B, Shots=0-shot2026.0287.84
Auto-Align Regularized LoopingModel=Gemma, Size=9B, Shots=0-shot2026.0287.84
Noise AblationModel=Gemma, Size=9B, Shots=0-shot2026.0287.84
FP16Backbone=LLaMA-3.1-8B, Quantization=None, Protocol=0-shot2026.0282.4
TRLModel Backbone=Qwen2.5-7B, Evaluation Protocol=0-shot2025.0981.57
OPPOModel Backbone=Qwen2.5-7B, Evaluation Protocol=0-shot2025.0981.36
BaselineModel=Gemma, Size=2B, Shots=0-shot2026.0280.3
Auto-Align Regularized LoopingModel=Gemma, Size=2B, Shots=0-shot2026.0280.22
Moving Average Regularized LoopingModel=Gemma, Size=2B, Shots=0-shot2026.0280.18
Uniform Regularized LoopingModel=Gemma, Size=2B, Shots=0-shot2026.0280.01
Noise AblationModel=Gemma, Size=2B, Shots=0-shot2026.0279.84
FP16q (mean bit precision)=16.000, tok/s (throughput)=103.8, Evaluation Protocol=0-shot2025.1279.63
BaselineModel=Llama, Size=8B, Shots=0-shot2026.0277.69
Auto-Align Regularized LoopingModel=Llama, Size=8B, Shots=0-shot2026.0277.61
Noise AblationModel=Llama, Size=8B, Shots=0-shot2026.0277.57
Moving Average Regularized LoopingModel=Llama, Size=8B, Shots=0-shot2026.0277.4
Uniform Regularized LoopingModel=Llama, Size=8B, Shots=0-shot2026.0277.31
AstroBackbone=LLaMA-3.1-8B, Quantization=W3A16g128, Protocol=0-shot2026.0277.1
OPPOModel Backbone=Qwen2.5-3B, Evaluation Protocol=0-shot2025.0975.08
AQLM-1x16 + PV-Tuningq (mean bit precision)=2.213, tok/s (throughput)=49.0, PV-Tuning=true, Evaluation Protocol=0-shot2025.1274.92
TRLModel Backbone=Qwen2.5-3B, Evaluation Protocol=0-shot2025.0974.54
AQLM-1x16q (mean bit precision)=2.213, tok/s (throughput)=49.0, Evaluation Protocol=0-shot2025.1273.99
CodeGEMM-m2v8g128 + PV-Tuningq (mean bit precision)=2.127, tok/s (throughput)=214.4, PV-Tuning=true, Evaluation Protocol=0-shot2025.1273.91
MagRBackbone=LLaMA-3.1-8B, Quantization=W3A16g128, Protocol=0-shot2026.0273.6
CodeGEMM-m1v4g128 + PV-Tuningq (mean bit precision)=2.126, tok/s (throughput)=228.3, PV-Tuning=true, Evaluation Protocol=0-shot2025.1273.15
OmniQuantBackbone=LLaMA-3.1-8B, Quantization=W3A16g128, Protocol=0-shot2026.0271.6
AQLM-2x8 + PV-Tuningq (mean bit precision)=2.005, tok/s (throughput)=124.5, PV-Tuning=true, Evaluation Protocol=0-shot2025.1271.25
CodeGEMM-m1v4g128q (mean bit precision)=2.126, tok/s (throughput)=228.3, Evaluation Protocol=0-shot2025.1263.97
CodeGEMM-m2v8g128q (mean bit precision)=2.127, tok/s (throughput)=214.4, Evaluation Protocol=0-shot2025.1260.14
FP16Bits=–, Backbone=LLAMA-2-7B2026.0453.58
FBI-LLMBits=W1A16, Backbone=LLAMA-2-7B2026.0453
AQLM-2x8q (mean bit precision)=2.005, tok/s (throughput)=124.5, Evaluation Protocol=0-shot2025.1246.25
LBLLMBits=W(1+1)A4, Backbone=LLAMA-2-7B2026.0444.65
OnebitBits=W1A16, Backbone=LLAMA-2-7B2026.0442.47
FlexRound-q2g128q (mean bit precision)=2.125, tok/s (throughput)=205.3, Evaluation Protocol=0-shot2025.1224.75