List your company
ResearchFormerly WizWand

Question Generation on SQuAD (F1 score)

90.68F1 ScoreFirst Order Adamw FT
6.044828.017449.9971.9626May 19, 2025Jul 21, 2025Sep 22, 2025Nov 25, 2025Jan 27, 2026Mar 31, 2026Jun 3, 2026
Updated 4mo ago

Evaluation results

MethodLinks
First Order Adamw FTModel Backbone=Llama2-7B, Fine-tuning Strategy=Full Fine-tuning, Optimization Algorithm=First Order Adamw, Number of training examples=10002026.0690.68
Dominant-layer ZO FTModel Backbone=Llama2-7B, Fine-tuning Strategy=Full Fine-tuning, Optimization Algorithm=Dominant-layer ZO, Number of training examples=10002026.0689.2
Dominant-layer ZO LoRAModel Backbone=Llama2-7B, Fine-tuning Strategy=LoRA, Optimization Algorithm=Dominant-layer ZO, Number of training examples=10002026.0688.58
QZOModel Backbone=Llama-3-8B, Model Precision=4 bits, Memory Profiling=6.3GB2025.0588.3
MeZO FTModel Backbone=Llama2-7B, Fine-tuning Strategy=Full Fine-tuning, Optimization Algorithm=MeZO, Number of training examples=10002026.0687.32
MeZOModel Backbone=Llama-3-8B, Model Precision=16 bits, Memory Profiling=20.5GB2025.0586.9
MeZO LoRAModel Backbone=Llama2-7B, Fine-tuning Strategy=LoRA, Optimization Algorithm=MeZO, Number of training examples=10002026.0686.26
QZOModel Backbone=Llama-2-7B, Model Precision=4 bits, Memory Profiling=5.0GB2025.0585.5
SubZero-GV (LoRA)Optimization Algorithm=SubZero, Tuning Strategy=LoRA, Guiding Vectors=true, Model=OPT-13B2026.0185.3
ZO-AdaMU (Prefix)Optimization Algorithm=ZO-AdaMU, Tuning Strategy=Prefix Tuning, Model=OPT-13B2026.0185.2
SubZero-GV (Prefix)Optimization Algorithm=SubZero, Tuning Strategy=Prefix Tuning, Guiding Vectors=true, Model=OPT-13B2026.0185.1
MeZO-GV (FT)Optimization Algorithm=MeZO, Tuning Strategy=Full Tuning, Guiding Vectors=true, Model=OPT-13B2026.0184.9
SubZero-GV (FT)Optimization Algorithm=SubZero, Tuning Strategy=Full Tuning, Guiding Vectors=true, Model=OPT-13B2026.0184.9
MeZO-GV (LoRA)Optimization Algorithm=MeZO, Tuning Strategy=LoRA, Guiding Vectors=true, Model=OPT-13B2026.0184.9
FTOptimization Algorithm=Adam, Tuning Strategy=Full Tuning, Model=OPT-13B2026.0184.9
Fine-tuningModel Backbone=Llama-3-8B, Model Precision=16 bits, Memory Profiling=31.9GB2025.0584.9
MeZO (FT)Optimization Algorithm=MeZO, Tuning Strategy=Full Tuning, Guiding Vectors=false, Model=OPT-13B2026.0184.7
SubZero (FT)Optimization Algorithm=SubZero, Tuning Strategy=Full Tuning, Guiding Vectors=false, Model=OPT-13B2026.0184.5
MeZO-GV (Prefix)Optimization Algorithm=MeZO, Tuning Strategy=Prefix Tuning, Guiding Vectors=true, Model=OPT-13B2026.0184.3
MeZO (Prefix)Optimization Algorithm=MeZO, Tuning Strategy=Prefix Tuning, Guiding Vectors=false, Model=OPT-13B2026.0184.2
HiZOO (LoRA)Optimization Algorithm=HiZOO, Tuning Strategy=LoRA, Model=OPT-13B2026.0183.8
MeZO (LoRA)Optimization Algorithm=MeZO, Tuning Strategy=LoRA, Guiding Vectors=false, Model=OPT-13B2026.0183.8
SubZero (LoRA)Optimization Algorithm=SubZero, Tuning Strategy=LoRA, Guiding Vectors=false, Model=OPT-13B2026.0183.7
SubZero (Prefix)Optimization Algorithm=SubZero, Tuning Strategy=Prefix Tuning, Guiding Vectors=false, Model=OPT-13B2026.0183.7
Fine-tuningModel Backbone=Llama-2-7B, Model Precision=16 bits, Memory Profiling=26.0GB2025.0583.7
HiZOO (Prefix)Optimization Algorithm=HiZOO, Tuning Strategy=Prefix Tuning, Model=OPT-13B2026.0183.2
ZO-AdaMU (2x)Optimization Algorithm=ZO-AdaMU, Tuning Strategy=Full Tuning, Training Steps=2x, Model=OPT-13B2026.0182.4
HiZOOOptimization Algorithm=HiZOO, Tuning Strategy=Full Tuning, Model=OPT-13B2026.0181.9
MeZOModel Backbone=Llama-2-7B, Model Precision=16 bits, Memory Profiling=14.8GB2025.0580.7
MeZOBackbone=Llama-2-7B, Precision=BF162026.0580.7
MeZOModel Backbone=OPT-6.7B, Model Precision=16 bits, Memory Profiling=14.8GB2025.0579.6
MeZO-GV(Prefix)Backbone=OPT-1.3B, Optimization Method=MeZO, Adaptation Technique=Prefix Tuning, Guiding Vectors (GV)=true2026.0178.8
MeZO-GV(LoRA)Backbone=OPT-1.3B, Optimization Method=MeZO, Adaptation Technique=LoRA, Guiding Vectors (GV)=true2026.0178.7
QZOModel Backbone=OPT-6.7B, Model Precision=4 bits, Memory Profiling=4.8GB2025.0578.5
Fine-tuningModel Backbone=OPT-6.7B, Model Precision=16 bits, Memory Profiling=26.8GB2025.0577.6
MeZO(LoRA)Backbone=OPT-1.3B, Optimization Method=MeZO, Adaptation Technique=LoRA2026.0177.5
ZO-AdaMU (LoRA)Optimization Algorithm=ZO-AdaMU, Tuning Strategy=LoRA, Model=OPT-13B2026.0176.8
MeZO(Prefix)Backbone=OPT-1.3B, Optimization Method=MeZO, Adaptation Technique=Prefix Tuning2026.0176
ICLEvaluation Protocol=In-context Learning, Model=OPT-13B2026.0175.9
MeZO-GV(FT)Backbone=OPT-1.3B, Optimization Method=MeZO, Adaptation Technique=Fine-tuning, Guiding Vectors (GV)=true2026.0175.2
MeZO(FT)Backbone=OPT-1.3B, Optimization Method=MeZO, Adaptation Technique=Fine-tuning2026.0172
Zero-ShotModel Backbone=Llama-3-8B, Model Precision=16 bits2025.0564.8
Zero-shot w/o finetuneModel Backbone=Llama2-7B, Fine-tuning Strategy=None, Optimization Algorithm=None, Number of training examples=10002026.0660.43
Zero-Shot-QModel Backbone=Llama-3-8B, Model Precision=4 bits2025.0559.2
ICLBackbone=OPT-1.3B, Learning Strategy=In-context Learning2026.0158.7
CAQ-ZOBackbone=Llama-2-7B, Precision=NF42026.0558.6
Zero-ShotModel Backbone=Llama-2-7B, Model Precision=16 bits2025.0555.6
Zero-Shot-QModel Backbone=Llama-2-7B, Model Precision=4 bits2025.0553.6
Zero-Shot-QBackbone=Llama-2-7B, Precision=NF42026.0553.6
Zero-shotEvaluation Protocol=Zero-shot, Model=OPT-13B2026.0146.2
MeZOBackbone=Qwen-2.5-1.5B, Precision=BF162026.0538.8
Zero-ShotModel Backbone=OPT-6.7B, Model Precision=16 bits2025.0536.5
Zero-Shot-QModel Backbone=OPT-6.7B, Model Precision=4 bits2025.0535.9
CAQ-ZOBackbone=Qwen-2.5-1.5B, Precision=NF42026.0533.3
QuZOBackbone=Llama-2-7B, Precision=NF42026.0530.5
Zero-shotBackbone=OPT-1.3B, Learning Strategy=Zero-shot2026.0127.2
Zero-Shot-QBackbone=Qwen-2.5-1.5B, Precision=NF42026.0519.1
QuZOBackbone=Qwen-2.5-1.5B, Precision=NF42026.059.3