List your company
ResearchFormerly WizWand

Boolean Question Answering on BoolQ (Accuracy and Speed)

85.9AccuracyTALE
44.622455.338766.05576.7713Jun 15, 2025Aug 10, 2025Oct 6, 2025Dec 1, 2025Jan 27, 2026Mar 24, 2026May 20, 2026
Updated 5mo ago

Evaluation results

MethodLinks
TALEBackbone=LLaMA 3.1 8B, Evaluation Protocol=0-shot, Layers Removed (#D)=32025.1085.912-8.8
BaselineBackbone=LLaMA 3.1 8B, Evaluation Protocol=0-shot, Layers Removed (#D)=02025.1085.4——
BSBABackbone=LLaMA 3.1 8B, Evaluation Protocol=0-shot, Layers Removed (#D)=72025.1085.4—-17.6
TALEBackbone=Mistral 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=42025.1084.416-18.5
TALEBackbone=Qwen 2.5 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=42025.1083.914-13.3
OriginalBackbone=Mistral-7B, Pruning Ratio=0%2026.0583.73——
BSBABackbone=Qwen 2.5 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=52025.1082.7—-23.2
TokAlign++Backbone=LLaMA38B, Initialization Method=TokAlign++, #GPU Hour=197.21, Shots=52026.0582.6——
AccBackbone=Mistral-7B, Pruning Ratio=25%, Relevance Estimation Source=Task-specific2026.0582.29——
BaselineBackbone=Qwen 2.5 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=02025.1081.9——
BaselineBackbone=Mistral 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=02025.1081.3——
BSBABackbone=Mistral 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=52025.1080.8—-27.7
TokAlign++Backbone=LLaMA38B, Initialization Method=TokAlign++, #GPU Hour=197.21, Shots=02026.0580.61——
DenseBase Model=LLaMA-2-7B, Sparsity=Dense2025.0677.68——
FIXED_E=128Experts=1282026.0574.89——
TALEBackbone=Lucie 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=52025.107430-17.2
DenseBase Model=DeepSeek-7B, Sparsity=Dense2025.0672.81——
FIXED_E=32Experts=322026.0571.71——
MaskProBase Model=LLaMA-2-7B, Sparsity=2:42025.0671.12——
SparsegptBase Model=LLaMA-2-7B, Sparsity=2:42025.0671.1——
EMO (Stage 4)E=32→642026.0570.52——
EMO (Stage 5)E=64→1282026.0570.4——
EMO (Stage 3)E=16→322026.0569.72——
Pruner-ZBase Model=LLaMA-2-7B, Sparsity=2:42025.0669.13——
EMO (Stage 2)E=8→162026.0567.8——
MaskProBase Model=DeepSeek-7B, Sparsity=2:42025.0667.77——
SparsegptBase Model=DeepSeek-7B, Sparsity=2:42025.0666.91——
EMO (Stage 1)E=82026.0566.85——
Cosine SimilarityBackbone=Mistral-7B, Pruning Ratio=25%, Relevance Estimation Source=Task-specific2026.0566.73——
Out. Norm-SimBackbone=Mistral-7B, Pruning Ratio=25%, Relevance Estimation Source=Task-specific2026.0566.54——
Pruner-ZBase Model=DeepSeek-7B, Sparsity=2:42025.0666.36——
PerplexityBackbone=Mistral-7B, Pruning Ratio=25%, Relevance Estimation Source=Task-specific2026.0564.86——
Out. Cosine-SimBackbone=Mistral-7B, Pruning Ratio=25%, Relevance Estimation Source=Task-specific2026.0564.62——
TokAlign++Backbone=Pythia2.8B, Initialization Method=TokAlign++, #GPU Hour=38.96, Shots=02026.0564.43——
BSBABackbone=Lucie 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=192025.1063—-54.2
Memory Grafting# Shots=0-shot, # Trainable Params=2.8B, # Activated (w/o token embed)=0.55B, # Trained Tokens=100B, # Experts (shared + routed, top-k)=1 + 47 (top-4)2026.0562.54——
TokAlign++Backbone=Pythia2.8B, Initialization Method=TokAlign++, #GPU Hour=38.96, Shots=52026.0560.98——
TokAlign++Backbone=Pythia1B, Initialization Method=TokAlign++, #GPU Hour=19.94, Shots=02026.0560.55——
Pythia1BBackbone=Pythia1B, Initialization Method=Vanilla, #GPU Hour=−, Shots=02026.0560.43——
Vanilla Engram# Shots=0-shot, # Trainable Params=2.8B, # Activated (w/o token embed)=0.55B, # Trained Tokens=100B, # Experts (shared + routed, top-k)=1 + 48 (top-4)2026.0560.24——
FIXED_E=16Experts=162026.0559.02——
Out. Divergence-SimBackbone=Mistral-7B, Pruning Ratio=25%, Relevance Estimation Source=Task-specific2026.0558.2——
MoE Baseline# Shots=0-shot, # Trainable Params=2.8B, # Activated (w/o token embed)=0.55B, # Trained Tokens=100B, # Experts (shared + routed, top-k)=1 + 64 (top-4)2026.0557.92——
PionArchitecture=LLaMA-1.3B, Training Tokens=54B2026.0557.58——
Pythia1BBackbone=Pythia1B, Initialization Method=Vanilla, #GPU Hour=−, Shots=52026.0557.37——
TaylorBackbone=Mistral-7B, Pruning Ratio=25%, Relevance Estimation Source=Task-specific2026.0555.72——
TokAlign++Backbone=Pythia1B, Initialization Method=TokAlign++, #GPU Hour=19.94, Shots=52026.0554.25——
BaselineBackbone=Lucie 7B, Evaluation Protocol=0-shot, Layers Removed (#D)=02025.1053——
MuonArchitecture=LLaMA-1.3B, Training Tokens=54B2026.0551.56——
CCDD-MMDiT# params.=144.1M, Evaluation Protocol=Zero-shot2025.1051.1——
GIDD# params.=92.1M, Evaluation Protocol=Zero-shot2025.1050.43——
CCDD-MoEDiT# params.=104.0M, Evaluation Protocol=Zero-shot2025.1050.21——
GIDD-base# params.=320M, Evaluation Protocol=Zero-shot2025.1049.57——
MDLM# params.=92.1M, Evaluation Protocol=Zero-shot2025.1049.42——
GPT2# params.=110M, Evaluation Protocol=Zero-shot2025.1048.72——
AdamWArchitecture=LLaMA-1.3B, Training Tokens=54B2026.0546.3——
Llama (retrain)# params.=117M, Evaluation Protocol=Zero-shot2025.1046.21——