List your company
ResearchFormerly WizWand

Question Answering on BoolQ

90.9AccuracyPaLM 2-L
76.02879.88983.7587.611Dec 2, 2022Jun 16, 2023Dec 30, 2023Jul 14, 2024Jan 26, 2025Aug 11, 2025Feb 24, 2026
Updated 6mo ago

Evaluation results

MethodLinks
PaLM 2-Lprompting=1-shot2023.0590.9——
PaLM-2 Lprotocol=1-shot2023.1190.9——
AUSteerModel=Llama-3.3-70B-Instruct2026.0290.67——
FLAN-T5Model Variant=xlarge, Model Parameters=3B2023.0789.6——
VanillaModel=Llama-3.3-70B-Instruct2026.0289.54——
FP16Type=-, Backbone=Qwen3-14B2025.1189.4——
QTIPType=vector, Backbone=Qwen3-14B2025.1189.2——
MoNEPruning ratio=25%, Backbone=Qwen3-30B-A3B, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.0789.11——
AWQType=linear, Backbone=Qwen3-14B2025.1189.1——
PAROQType=linear, Backbone=Qwen3-14B2025.1189.1——
Falcon-180Bprotocol=1-shot2023.1189——
E-QATType=linear, Backbone=Qwen3-14B2025.1189——
MC-SMoEPruning ratio=25%, Backbone=Qwen3-30B-A3B, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.0788.75——
PaLMprompting=1-shot2023.0588.7——
PaLMprotocol=1-shot2023.1188.7——
AUSteerModel=Qwen3-30B-A3B2026.0288.69——
Qwen3-30B-A3BPruning ratio=0%, Backbone=Qwen3-30B-A3B, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.0788.69——
PaLM 2-Mprompting=1-shot2023.0588.6——
PaLM-2 Mprotocol=1-shot2023.1188.6——
RSPruning ratio=25%, Backbone=Qwen3-30B-A3B, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.0788.53——
PaLM 2-Sprompting=1-shot2023.0588.1——
PaLM-2 Sprotocol=1-shot2023.1188.1——
ALIGNModel Variant=large, Model Parameters=355M2023.0787.7——
Hard-routing MoEBackbone=Qwen2.5-7B-Instruct, # Params (%)=14.2G (100%), # Trans. (%)=14.2G (100%), Fine-tuning setting=DFL, Rank (rl)=322026.0287.67——
PAROQType=linear, Backbone=Qwen3-8B2025.1187——
FLAPPruning ratio=25%, Backbone=Qwen2-57B-A14B, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.0786.91——
QTIPType=vector, Backbone=Qwen3-8B2025.1186.9——
MoNEPruning ratio=25%, Backbone=Qwen2-57B-A14B, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.0786.88——
VanillaModel=Qwen3-30B-A3B2026.0286.82——
BF16Model Size=8B, Quantization Method=None (BF16), Adaptive Block Scaling (4/6)=false2025.1286.6——
FP16Type=-, Backbone=Qwen3-8B2025.1186.6——
GPTQ + 4/6Model Size=8B, Quantization Method=GPTQ, Adaptive Block Scaling (4/6)=true2025.1286.5——
AWQModel Size=8B, Quantization Method=AWQ, Adaptive Block Scaling (4/6)=false2025.1286.5——
AWQ + 4/6Model Size=8B, Quantization Method=AWQ, Adaptive Block Scaling (4/6)=true2025.1286.5——
Qwen2-57B-A14BPruning ratio=0%, Backbone=Qwen2-57B-A14B, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.0786.45——
MC-SMoEPruning ratio=25%, Backbone=Qwen2-57B-A14B, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.0786.45——
GPTQModel Size=8B, Quantization Method=GPTQ, Adaptive Block Scaling (4/6)=false2025.1286.4——
SmoothQuantModel Size=8B, Quantization Method=SmoothQuant, Adaptive Block Scaling (4/6)=false2025.1286.4——
AWQType=linear, Backbone=Qwen3-8B2025.1186.2——
E-QATType=linear, Backbone=Qwen3-8B2025.1186.1——
SmoothQuant + 4/6Model Size=8B, Quantization Method=SmoothQuant, Adaptive Block Scaling (4/6)=true2025.1285.9——
RTN + 4/6Model Size=8B, Quantization Method=RTN, Adaptive Block Scaling (4/6)=true2025.1285.8——
FLAPPruning ratio=25%, Backbone=Qwen3-30B-A3B, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.0785.72——
RTNModel Size=8B, Quantization Method=RTN, Adaptive Block Scaling (4/6)=false2025.1285.5——
PAROQType=linear, Backbone=Qwen3-4B2025.1185.3——
FP16Type=-, Backbone=Qwen3-4B2025.1185.1——
QTIPType=vector, Backbone=Qwen3-4B2025.1185——
AWQType=linear, Backbone=Qwen3-4B2025.1185——
FLAN-T5Model Variant=large, Model Parameters=780M2023.0784.9——
Krause-Qwen1.5-7BWindow size=32, top-k=162026.0284.78——
RSPruning ratio=25%, Backbone=Qwen2-57B-A14B, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.0784.77——
Sparse-and-Orthogonal LoRABackbone=Qwen2.5-7B-Instruct, # Params (%)=87M (0.60%), # Trans. (%)=42M (0.30%), Fine-tuning setting=DFL, Rank (rl)=322026.0284.76——
Krause-Qwen1.5-7BWindow size=48, top-k=242026.0284.66——
Krause-Qwen1.5-7BWindow size=18, top-k=122026.0284.63——
E-QATType=linear, Backbone=Qwen3-4B2025.1184.5——
QTIPType=vector, Backbone=LLaMA-3.1-8B-Instruct2025.1184.3——
FPFTBackbone=Qwen2.5-7B-Instruct, # Params (%)=14.2G (100%), # Trans. (%)=14.2G (100%), Fine-tuning setting=DFL, Rank (rl)=322026.0284.25——
FP16Type=-, Backbone=LLaMA-3.1-8B-Instruct2025.1184.1——
Qwen1.5-7BFine-tuning protocol=LoRA2026.0284.08——
PAROQType=linear, Backbone=LLaMA-3.1-8B-Instruct2025.1183.9——
Qwen1.5-7BFine-tuning protocol=None2026.0283.8——
AWQType=linear, Backbone=LLaMA-3.1-8B-Instruct2025.1183.5——
E-QATType=linear, Backbone=LLaMA-3.1-8B-Instruct2025.1183.5——
Sparse-and-Orthogonal LoRA (Single)Backbone=Qwen2.5-7B-Instruct, # Params (%)=87M (0.60%), # Trans. (%)=42M (0.30%), Fine-tuning setting=DFL, Rank (rl)=322026.0283.33——
Hard-routing MoEBackbone=Qwen2.5-1.5B-Instruct, # Params (%)=3.2G (100%), # Trans. (%)=3.2G (100%), Fine-tuning setting=DFL, Rank (rl)=322026.0283.23——
DenseBackbone=LLaMA-3.1-8B, Pruning Method=None, Evaluation Protocol=Zero-shot, FFN Sparsity=0%2026.0183.09——
LoRIBackbone=Qwen2.5-7B-Instruct, # Params (%)=168M (1.16%), # Trans. (%)=168M (1.16%), Fine-tuning setting=DFL, Rank (rl)=322026.0282.72——
MUPPET2022.1282.17——
Sparse-and-Orthogonal LoRABackbone=Qwen2.5-1.5B-Instruct, # Params (%)=16M (0.54%), # Trans. (%)=8M (0.27%), Fine-tuning setting=DFL, Rank (rl)=322026.0281.45——
ColD-Fusion2022.1281.39——
LoRABackbone=Qwen2.5-7B-Instruct, # Params (%)=362M (2.5%), # Trans. (%)=362M (2.5%), Fine-tuning setting=DFL, Rank (rl)=322026.0281.32——
ALIGNModel Variant=base, Model Parameters=125M2023.0781.1——
Krause-Llama3-8B2026.0280.59——
AngularPruning ratio=25%, Backbone=Qwen3-30B-A3B, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.0780.52——
Llama3-8Bfine-tuning=LoRA2026.0280.41——
MoonlightPruning ratio=0%, Backbone=Moonlight, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.0780.4——
Multitask2022.1280.27——
Sparse-and-Orthogonal LoRA (Single)Backbone=Qwen2.5-1.5B-Instruct, # Params (%)=16M (0.54%), # Trans. (%)=8M (0.27%), Fine-tuning setting=DFL, Rank (rl)=322026.0280.21——
Pythia-6.9B w/ Qwen2-7BShots=5, Teacher=Qwen2-7B2025.0680——
Deepseek-V2-LitePruning ratio=0%, Backbone=Deepseek-V2-Lite, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.0779.88——
Pythia-6.9B w/ Qwen2-7BShots=0, Teacher=Qwen2-7B2025.0679.85——
FPFTBackbone=Qwen2.5-1.5B-Instruct, # Params (%)=3.2G (100%), # Trans. (%)=3.2G (100%), Fine-tuning setting=DFL, Rank (rl)=322026.0279.81——
MoNEPruning ratio=25%, Backbone=Moonlight, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.0779.57——
Pythia-6.9B + Direct tuningShots=02025.0679.3——
LoRIBackbone=Qwen2.5-1.5B-Instruct, # Params (%)=32M (1.05%), # Trans. (%)=32M (1.05%), Fine-tuning setting=DFL, Rank (rl)=322026.0279.29——
S-MeZOBackbone=Mistral-7B, Evaluation Protocol=Fine-Tuning2024.0279.2——
Pythia-6.9B + Direct tuningShots=52025.0678.87——
Pythia-6.9B w/ LLAMA3-8BShots=5, Teacher=LLAMA3-8B2025.0678.78——
MC-SMoEPruning ratio=25%, Backbone=Moonlight, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.0778.72——
RSPruning ratio=25%, Backbone=Moonlight, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.0778.69——
MoNEPruning ratio=25%, Backbone=Deepseek-V2-Lite, Evaluation protocol=Zero-shot, Calibration samples=100, Calibration dataset=Zyda22025.0778.47——
LLaMA-13B (Touvron et al. 2023)Pruning Ratio=0%, Zero-shot=true2024.0378.1——
Pythia-6.9B w/ LLAMA3-8BShots=0, Teacher=LLAMA3-8B2025.0677.83——
DenseBackbone=Llama-2-7B, Sparsity=0%, Evaluation Protocol=Zero-shot2025.1077.68——
BF16Model Size=1.7B, Quantization Method=None (BF16), Adaptive Block Scaling (4/6)=false2025.1277.6——
Finetune2022.1277.14——
LoRABackbone=Qwen2.5-1.5B-Instruct, # Params (%)=65.5M (2.01%), # Trans. (%)=65.5M (2.01%), Fine-tuning setting=DFL, Rank (rl)=322026.0276.94——
Pythia-6.9B + Sentence distillShots=52025.0676.88——
Pythia-6.9B + Sentence distillShots=02025.0676.64——
MeZOBackbone=Mistral-7B, Evaluation Protocol=Fine-Tuning2024.0276.6——