List your company
ResearchFormerly WizWand

Commonsense Reasoning on HellaSwag (val)

95.3AccuracyGPT-4
27.471245.080662.6980.2994Mar 15, 2023Sep 30, 2023Apr 16, 2024Nov 1, 2024May 19, 2025Dec 4, 2025Jun 21, 2026
Updated 4mo ago

Evaluation results

MethodLinks
GPT-4number_of_shots=10-shot2023.0395.3——
D2-LoRABackbone=Qwen2.5-7B-Instruct, Training Epochs=1, Scoring Method=CLL2026.0291——
LoRABackbone=Qwen2.5-7B-Instruct, Training Epochs=1, Scoring Method=CLL2026.0290.2——
DoRABackbone=Qwen2.5-7B-Instruct, Training Epochs=1, Scoring Method=CLL2026.0290——
ALUMbenchmark-specific tuning=true2023.0385.6——
GPT-3.5number_of_shots=10-shot2023.0385.5——
LLaMAsplit=validation set2023.0384.2——
D2-LoRABackbone=Llama-3.2-3B-Instruct, Training Epochs=2, Scoring Method=CLL2026.0282.4——
DoRABackbone=Llama-3.2-3B-Instruct, Training Epochs=2, Scoring Method=CLL2026.0281.8——
LoRABackbone=Llama-3.2-3B-Instruct, Training Epochs=2, Scoring Method=CLL2026.0277.6——
Qwen3-4B2026.0668.4——
EvoGMBackbone=Qwen2.5-1.5B2026.0566——
PSO-MergingBackbone=Qwen2.5-1.5B2026.0565——
CMABase model=Qwen2.5-1.5B2026.0564.5——
EvoGMBase model=Qwen2.5-1.5B2026.0563.5——
Model SwarmBackbone=Qwen2.5-1.5B2026.0563.5——
Single BestBase model=Qwen2.5-1.5B2026.0562——
Single BestBackbone=Qwen2.5-1.5B2026.0562——
DAREBase model=Qwen2.5-1.5B2026.0561.5——
DAREBackbone=Qwen2.5-1.5B2026.0561.5——
Task ArithmeticBase model=Qwen2.5-1.5B2026.0560.5——
TABackbone=Qwen2.5-1.5B2026.0560.5——
CMABackbone=Qwen2.5-1.5B2026.0560——
BaseBase model=Qwen2.5-1.5B2026.0559——
Model SoupBase model=Qwen2.5-1.5B2026.0559——
BaseBackbone=Qwen2.5-1.5B2026.0559——
Model SoupBackbone=Qwen2.5-1.5B2026.0559——
Model SwarmBase model=Qwen2.5-1.5B2026.0558.5——
FedAvgBackbone=SmolLM-360M, Federated Partitioning=10-client non-IID, Evaluation Protocol=3-seed mean2026.0658.11.107897.77
Bank of Values (BoV)Output=Ev[i], Layers=last 1/3, Coeff.=γv, Model Scale=780M, Zero-shot=true2026.0657.4——
V1 variant (last 1/3)Output=V1, Layers=last 1/3, Coeff.=γv, Model Scale=780M, Zero-shot=true2026.0656.6——
Subspace-RegBackbone=SmolLM-360M, Federated Partitioning=10-client non-IID, Evaluation Protocol=3-seed mean2026.0656.51.125499.989
x0WV variantOutput=x0WV, Layers=last 1/3, Coeff.=γv, Model Scale=780M, Zero-shot=true2026.0656.2——
Standard AttentionOutput=V, Model Scale=780M, Zero-shot=true2026.0655.8——
DeepSeekMoE# Shot=0-shot, # Total Params=2.0B, # Activated Params=0.3B, FLOPs per 2K Tokens=4.3T, # Training Tokens=100B2024.0154.8——
SVDBackbone=SmolLM-360M, Federated Partitioning=10-client non-IID, Evaluation Protocol=3-seed mean2026.0653.61.170899.06
PSO-MergingBase model=Qwen2.5-1.5B2026.0553——
MTLBase model=Qwen2.5-1.5B2026.0552.5——
MTLBackbone=Qwen2.5-1.5B2026.0552.5——
FedSVDBackbone=SmolLM-360M, Federated Partitioning=10-client non-IID, Evaluation Protocol=3-seed mean2026.0652.11.184896.8
V1 variant (every layer)Output=V1, Layers=every, Coeff.=1, Model Scale=780M, Zero-shot=true2026.0651.6——
TIESBase model=Qwen2.5-1.5B2026.0551——
TIESBackbone=Qwen2.5-1.5B2026.0551——
GPT-2 XL2026.0650.9——
Mistral (Full-Attention)Model Scale=1.4B2024.0750.7——
GShard# Shot=0-shot, # Total Params=2.0B, # Activated Params=0.3B, FLOPs per 2K Tokens=4.3T, # Training Tokens=100B2024.0150.5——
Talkie-1930variant=base2026.0649.8——
Switch# Shot=0-shot, # Total Params=2.0B, # Activated Params=0.2B, FLOPs per 2K Tokens=2.9T, # Training Tokens=100B2024.0149.1——
BMoJo (Fading + Eidetic)Model Scale=1.4B2024.0746.8——
Hash Layer# Shot=0-shot, # Total Params=2.0B, # Activated Params=0.2B, FLOPs per 2K Tokens=2.9T, # Training Tokens=100B2024.0146.2——
BMoJo (Fading)Model Scale=1.4B2024.0746——
Mamba (SSM)Model Scale=1.4B2024.0745——
Dense# Shot=0-shot, # Total Params=0.2B, # Activated Params=0.2B, FLOPs per 2K Tokens=2.9T, # Training Tokens=100B2024.0138.8——
Hybrid (Sliding Attention + SSM)Model Scale=1.4B2024.0738.8——
AMOOptimizer=AMO, Model=Llama3.1-1.4B, Few-shot=02026.0537.46——
TYPEWRITERLMvariant=base2026.0635.9——
GPT-1900variant=base2026.0634.8——
Mistral (Full-Attention)Model Scale=370M2024.0733.8——
AMOOptimizer=AMO, Model=Llama3.1-760M, Few-shot=02026.0533.79——
Mamba (SSM)Model Scale=370M2024.0733.4——
BMoJo (Fading + Eidetic)Model Scale=370M2024.0733.3——
BMoJo (Fading)Model Scale=370M2024.0733.2——
TMMFormerBackbone=12-layer Transformer (d=768)2026.0531.82——
YuriiFormerBackbone=12-layer Transformer (d=768)2026.0531.58——
Hybrid (Sliding Attention + SSM)Model Scale=370M2024.0731.3——
AdamFormerBackbone=12-layer Transformer (d=768)2026.0530.96——
VanillaTransformerBackbone=12-layer Transformer (d=768)2026.0530.2——
AdamWFormerBackbone=12-layer Transformer (d=768)2026.0530.08——