List your company
ResearchFormerly WizWand

Multitask Language Understanding on MMLU (Average Accuracy)

79.64MMLU AccuracyFP16
32.694444.882257.0769.2578Jul 6, 2025Sep 3, 2025Nov 2, 2025Dec 31, 2025Mar 1, 2026Apr 29, 2026Jun 28, 2026
Updated 3mo ago

Evaluation results

MethodLinks
FP16Backbone=Qwen3-30B-A3B, Evaluation Mode=Few-shot2026.0679.64
SharQBackbone=Qwen3-30B-A3B, Evaluation Mode=Few-shot, Sparsity Pattern=4:8, Base Quantization=NVFP42026.0678.79
NVFP4Backbone=Qwen3-30B-A3B, Evaluation Mode=Few-shot2026.0677.72
FP16Backbone=Qwen2.5-7B, Evaluation Mode=Few-shot2026.0674.16
SharQBackbone=Qwen2.5-7B, Evaluation Mode=Few-shot, Sparsity Pattern=4:8, Base Quantization=NVFP42026.0672.83
NVFP4Backbone=Qwen2.5-7B, Evaluation Mode=Few-shot2026.0672.06
FP16Backbone=Llama-3.1-8B, Evaluation Mode=Few-shot2026.0665.24
SharQBackbone=Llama-3.1-8B, Evaluation Mode=Few-shot, Sparsity Pattern=4:8, Base Quantization=NVFP42026.0663.76
MoD-Single RoundCategory=Text, Backbone=Qwen2.5VL-3b-Instruct, Protocol=single-round inference2026.0663.73
Self-CorrectionCategory=Text, Backbone=Qwen2.5VL-3b-Instruct2026.0663.7
MoD-Multi RoundCategory=Text, Backbone=Qwen2.5VL-3b-Instruct, Protocol=multi-turn dialectical reasoning2026.0663.69
MoE-LoRACategory=Text, Backbone=Qwen2.5VL-3b-Instruct2026.0663.59
Multi-Agent DebateCategory=Text, Backbone=Qwen2.5VL-3b-Instruct2026.0663.21
Qwen2.5VL-3b-InstructCategory=Text2026.0662.84
NVFP4Backbone=Llama-3.1-8B, Evaluation Mode=Few-shot2026.0661.93
MoD-Multi RoundCategory=Text, Backbone=LLaVA-v1.6-13b, Protocol=multi-turn dialectical reasoning2026.0656.35
Self-CorrectionCategory=Text, Backbone=LLaVA-v1.6-13b2026.0656.06
Multi-Agent DebateCategory=Text, Backbone=LLaVA-v1.6-13b2026.0655.97
MoD-Single RoundCategory=Text, Backbone=LLaVA-v1.6-13b, Protocol=single-round inference2026.0655.95
MoE-LoRACategory=Text, Backbone=LLaVA-v1.6-13b2026.0655.62
LLaVA-v1.6-13bCategory=Text2026.0655.42
Qwen-VL-ChatCategory=Text2026.0650.7
Original ModelBackbone=Qwen-3-4B, Avg. Sparsity (L/H)=0% / 0%, Mem (GB)=5.80, Time (s)=41002026.0645.1
Learning to AllocateBackbone=Qwen-3-4B, Avg. Sparsity (L/H)=38% / 28%, Mem (GB)=5.30, Time (s)=28502026.0644.5
TTT+CT-KVTraining time per task (s)=342025.0744.1
CT-KVTraining time per task (s)=92025.0743.7
TTTTraining time per task (s)=302025.0743.6
CT-PromptTraining time per task (s)=332025.0743.6
FlexiDepthBackbone=Qwen-3-4B, Avg. Sparsity (L/H)=34% / –, Mem (GB)=5.75, Time (s)=29102026.0642.5
AdaSkipBackbone=Qwen-3-4B, Avg. Sparsity (L/H)=36% / –, Mem (GB)=5.65, Time (s)=28002026.0641.2
ICLTraining time per task (s)=0, Evaluation protocol=In-Context Learning2025.0741.2
DoRATraining time per task (s)=162025.0740.3
LoRATraining time per task (s)=162025.0740.1
Prefix TuningTraining time per task (s)=5, m=322025.0739.9
Oracle Static PruningBackbone=Qwen-3-4B, Avg. Sparsity (L/H)=33% / 22%, Mem (GB)=4.95, Time (s)=26802026.0639.8
Prompt TuningTraining time per task (s)=15, m=322025.0739.2
Rank-Stabilized LoRATraining time per task (s)=172025.0738.8
Prefix TuningTraining time per task (s)=8, m=# demo2025.0738.8
Prompt TuningTraining time per task (s)=29, m=# demo2025.0737.3
Zero-ShotTraining time per task (s)=0, Evaluation protocol=Zero-Shot2025.0735.8
Static PruningBackbone=Qwen-3-4B, Avg. Sparsity (L/H)=33% / 0%, Mem (GB)=5.10, Time (s)=27502026.0634.5