List your company
ResearchFormerly WizWand

Science Question Answering on SciQ (Accuracy)

94.3Accuracy (SciQ)Original
66.53273.74180.9588.159May 29, 2025Aug 1, 2025Oct 4, 2025Dec 7, 2025Feb 9, 2026Apr 14, 2026Jun 17, 2026
Updated 4mo ago

Evaluation results

MethodLinks
OriginalModel=LLaMA-3.1-8B, Ratio=1.0, Evaluation Protocol=0-shot2026.0494.3
mHCSize=3B, Evaluation Protocol=Five-shot2026.0693.5
Depth-AttentionSize=3B, Evaluation Protocol=Five-shot2026.0692.8
Attention ResidualsSize=3B, Evaluation Protocol=Five-shot2026.0692.7
Attention ResidualsSize=1.5B, Evaluation Protocol=Five-shot2026.0691.9
DenseFormerSize=3B, Evaluation Protocol=Five-shot2026.0691.9
mHCSize=1.5B, Evaluation Protocol=Five-shot2026.0691.8
Depth-AttentionSize=1.5B, Evaluation Protocol=Five-shot2026.0691.5
VanillaSize=3B, Evaluation Protocol=Five-shot2026.0691.2
OriginalModel=LLaMA-2-7B, Ratio=1.0, Evaluation Protocol=0-shot2026.0491.1
DenseFormerSize=1.5B, Evaluation Protocol=Five-shot2026.0691
VanillaSize=1.5B, Evaluation Protocol=Five-shot2026.0690.1
Depth-AttentionSize=1.5B, Evaluation Protocol=Zero-shot2026.0688.2
GRASPruneModel=LLaMA-3.1-8B, Ratio=0.8, Evaluation Protocol=0-shot2026.0488.1
Depth-AttentionSize=3B, Evaluation Protocol=Zero-shot2026.0688
FLAPModel=LLaMA-2-7B, Ratio=0.8, Evaluation Protocol=0-shot2026.0487.8
DenseFormerSize=3B, Evaluation Protocol=Zero-shot2026.0686.9
Attention ResidualsSize=3B, Evaluation Protocol=Zero-shot2026.0686.7
mHCSize=1.5B, Evaluation Protocol=Zero-shot2026.0686.6
VanillaSize=3B, Evaluation Protocol=Zero-shot2026.0686.6
FLAPModel=LLaMA-3.1-8B, Ratio=0.8, Evaluation Protocol=0-shot2026.0486.5
X-GRAM-50%Backbone Scale=1.15B Scale, Cfg=4x, # Params=1.15B+7.47B, Evaluation Protocol=Zero/Few-shot2026.0486
Attention ResidualsSize=1.5B, Evaluation Protocol=Zero-shot2026.0685.9
X-GRAM-100%Backbone Scale=1.15B Scale, Cfg=4x, # Params=1.15B+14.94B, Evaluation Protocol=Zero/Few-shot2026.0485.8
mHCSize=3B, Evaluation Protocol=Zero-shot2026.0685.6
DenseFormerSize=1.5B, Evaluation Protocol=Zero-shot2026.0685.4
OracleModel=Llama3.1-8B, Black-Box Friendly=true2026.0485.1
X-GRAM-50%Backbone Scale=1.15B Scale, Cfg=2x, # Params=1.15B+3.73B, Evaluation Protocol=Zero/Few-shot2026.0484.8
OracleModel=Gemma2-9B, Black-Box Friendly=true2026.0484.6
X-GRAM-100%Backbone Scale=0.73B Scale, Cfg=4x, # Params=0.73B+9.33B, Evaluation Protocol=Zero/Few-shot2026.0484.4
X-GRAM-100%Backbone Scale=1.15B Scale, Cfg=2x, # Params=1.15B+7.47B, Evaluation Protocol=Zero/Few-shot2026.0484.2
VanillaSize=1.5B, Evaluation Protocol=Zero-shot2026.0684.1
RegMix-DProxy models=128, Mode=Online2026.0683.67
X-GRAM-50%Backbone Scale=0.73B Scale, Cfg=2x, # Params=0.73B+2.33B, Evaluation Protocol=Zero/Few-shot2026.0483.5
X-GRAM-100%Backbone Scale=0.73B Scale, Cfg=2x, # Params=0.73B+4.67B, Evaluation Protocol=Zero/Few-shot2026.0483.5
RegMix-DProxy models=512, Mode=Online2026.0683.45
X-GRAM-50%Backbone Scale=1.15B Scale, Cfg=1x, # Params=1.15B+1.87B, Evaluation Protocol=Zero/Few-shot2026.0483.4
RegMix-DProxy models=128, Mode=Offline2026.0683.35
X-GRAM-50%Backbone Scale=0.73B Scale, Cfg=1x, # Params=0.73B+1.17B, Evaluation Protocol=Zero/Few-shot2026.0482.9
X-GRAM-50%Backbone Scale=0.73B Scale, Cfg=4x, # Params=0.73B+4.67B, Evaluation Protocol=Zero/Few-shot2026.0482.8
X-GRAM-100%Backbone Scale=1.15B Scale, Cfg=1x, # Params=1.15B+3.73B, Evaluation Protocol=Zero/Few-shot2026.0482.8
MoRTBackbone Scale=1.15B Scale, Cfg=4x, # Params=1.15B+14.94B, Evaluation Protocol=Zero/Few-shot2026.0482.7
RegMix-DProxy models=512, Mode=Offline2026.0682.53
OracleModel=Qwen2.5-7B, Black-Box Friendly=true2026.0482
BaselineBackbone Scale=1.15B Scale, Cfg=N/A, # Params=1.15B, Evaluation Protocol=Zero/Few-shot2026.0482
RetokenBackbone Scale=1.15B Scale, Cfg=2x, # Params=1.15B+7.47B, Evaluation Protocol=Zero/Few-shot2026.0481.5
DoReMi2026.0681.42
RetokenBackbone Scale=1.15B Scale, Cfg=1x, # Params=1.15B+3.73B, Evaluation Protocol=Zero/Few-shot2026.0481.2
X-GRAM-100%Backbone Scale=0.73B Scale, Cfg=1x, # Params=0.73B+2.33B, Evaluation Protocol=Zero/Few-shot2026.0481
MoRTBackbone Scale=1.15B Scale, Cfg=1x, # Params=1.15B+3.73B, Evaluation Protocol=Zero/Few-shot2026.0481
RegMix2026.0680.98
OracleModel=Qwen2.5-3B, Black-Box Friendly=true2026.0480.8
MoRTBackbone Scale=1.15B Scale, Cfg=2x, # Params=1.15B+7.47B, Evaluation Protocol=Zero/Few-shot2026.0480.5
RetokenBackbone Scale=0.73B Scale, Cfg=4x, # Params=0.73B+9.33B, Evaluation Protocol=Zero/Few-shot2026.0480.4
OracleModel=Llama3.2-3B, Black-Box Friendly=true2026.0479.8
Human2026.0679.67
RetokenBackbone Scale=0.73B Scale, Cfg=2x, # Params=0.73B+4.67B, Evaluation Protocol=Zero/Few-shot2026.0478.8
EngramBackbone Scale=1.15B Scale, Cfg=2x, # Params=1.15B+7.47B, Evaluation Protocol=Zero/Few-shot2026.0478.7
RetokenBackbone Scale=1.15B Scale, Cfg=4x, # Params=1.15B+14.94B, Evaluation Protocol=Zero/Few-shot2026.0478.6
BaselineBackbone Scale=0.73B Scale, Cfg=N/A, # Params=0.73B, Evaluation Protocol=Zero/Few-shot2026.0478.4
MoRTBackbone Scale=0.73B Scale, Cfg=2x, # Params=0.73B+4.67B, Evaluation Protocol=Zero/Few-shot2026.0478.2
EngramBackbone Scale=0.73B Scale, Cfg=2x, # Params=0.73B+4.67B, Evaluation Protocol=Zero/Few-shot2026.0478.1
EngramBackbone Scale=1.15B Scale, Cfg=4x, # Params=1.15B+14.94B, Evaluation Protocol=Zero/Few-shot2026.0478
EngramBackbone Scale=0.73B Scale, Cfg=1x, # Params=0.73B+2.33B, Evaluation Protocol=Zero/Few-shot2026.0477.9
RetokenBackbone Scale=0.73B Scale, Cfg=1x, # Params=0.73B+2.33B, Evaluation Protocol=Zero/Few-shot2026.0477.5
EngramBackbone Scale=1.15B Scale, Cfg=1x, # Params=1.15B+3.73B, Evaluation Protocol=Zero/Few-shot2026.0477.5
MoRTBackbone Scale=0.73B Scale, Cfg=4x, # Params=0.73B+9.33B, Evaluation Protocol=Zero/Few-shot2026.0475
MoRTBackbone Scale=0.73B Scale, Cfg=1x, # Params=0.73B+2.33B, Evaluation Protocol=Zero/Few-shot2026.0474.3
RCSmedoidModel=Gemma2-9B, Black-Box Friendly=true2026.0474.2
RCSprobModel=Gemma2-9B, Black-Box Friendly=false2026.0474.2
GreedyModel=Gemma2-9B, Black-Box Friendly=true2026.0474.1
GRASPruneModel=LLaMA-3.1-8B, Ratio=0.6, Evaluation Protocol=0-shot2026.0474.1
RCSuniModel=Gemma2-9B, Black-Box Friendly=true2026.0474
SCModel=Gemma2-9B, Black-Box Friendly=true2026.0473.9
RCSfreqModel=Gemma2-9B, Black-Box Friendly=true2026.0473.9
EngramBackbone Scale=0.73B Scale, Cfg=4x, # Params=0.73B+9.33B, Evaluation Protocol=Zero/Few-shot2026.0473.9
CEModel=Gemma2-9B, Black-Box Friendly=false2026.0473.6
PionArchitecture=LLaMA-1.3B, Training Tokens=54B2026.0573.4
AC-ODM-410MSetting=Zero-shot, Pre-training Dataset=The Pile, Model Size=410M2025.0573
NLLModel=Gemma2-9B, Black-Box Friendly=false2026.0472.9
MuonArchitecture=LLaMA-1.3B, Training Tokens=54B2026.0571.6
GRASPruneModel=LLaMA-2-7B, Ratio=0.8, Evaluation Protocol=0-shot2026.0470.8
AdamWArchitecture=LLaMA-1.3B, Training Tokens=54B2026.0570.8
GreedyModel=Qwen2.5-7B, Black-Box Friendly=true2026.0470.5
CEModel=Qwen2.5-7B, Black-Box Friendly=false2026.0470.4
SCModel=Qwen2.5-7B, Black-Box Friendly=true2026.0470.3
RCSfreqModel=Qwen2.5-7B, Black-Box Friendly=true2026.0470.3
RCSuniModel=Qwen2.5-7B, Black-Box Friendly=true2026.0470.2
AC-ODMSetting=Zero-shot, Pre-training Dataset=The Pile, Model Size=1B2025.0570.2
RCSmedoidModel=Qwen2.5-7B, Black-Box Friendly=true2026.0470.1
RCSprobModel=Qwen2.5-7B, Black-Box Friendly=false2026.0470
FLAPModel=LLaMA-3.1-8B, Ratio=0.6, Evaluation Protocol=0-shot2026.0470
GreedyModel=Llama3.1-8B, Black-Box Friendly=true2026.0469
ODMSetting=Zero-shot, Pre-training Dataset=The Pile, Model Size=1B2025.0568.9
FLAPModel=LLaMA-3.1-8B, Ratio=0.5, Evaluation Protocol=0-shot2026.0468.8
RCSuniModel=Llama3.1-8B, Black-Box Friendly=true2026.0468.1
RCSmedoidModel=Llama3.1-8B, Black-Box Friendly=true2026.0468
RCSprobModel=Llama3.1-8B, Black-Box Friendly=false2026.0467.9
KALAVAI MoEModel Scale=Pythia-6.9B, Training Steps=10000 base, Freeze Layers=6, Seed=42, Number of Examples=5002026.0367.8
RCSfreqModel=Llama3.1-8B, Black-Box Friendly=true2026.0467.6