Generation on DROP
48.74F1 ScoreFirst Order Adamw FT
Evaluation results
| Method | Links | |
|---|---|---|
| First Order Adamw FTModel Backbone=Llama2-7B, Fine-tuning Strategy=Full Fine-tuning, Optimization Algorithm=First Order Adamw, Number of training examples=10002026.06 | 48.74 | |
| Dominant-layer ZO LoRAModel Backbone=Llama2-7B, Fine-tuning Strategy=LoRA, Optimization Algorithm=Dominant-layer ZO, Number of training examples=10002026.06 | 42.1 | |
| Dominant-layer ZO FTModel Backbone=Llama2-7B, Fine-tuning Strategy=Full Fine-tuning, Optimization Algorithm=Dominant-layer ZO, Number of training examples=10002026.06 | 41.05 | |
| MeZO LoRAModel Backbone=Llama2-7B, Fine-tuning Strategy=LoRA, Optimization Algorithm=MeZO, Number of training examples=10002026.06 | 40.57 | |
| MeZO FTModel Backbone=Llama2-7B, Fine-tuning Strategy=Full Fine-tuning, Optimization Algorithm=MeZO, Number of training examples=10002026.06 | 39.85 | |
| SubZero-GV (Prefix)Optimization Algorithm=SubZero, Tuning Strategy=Prefix Tuning, Guiding Vectors=true, Model=OPT-13B2026.01 | 32.9 | |
| MeZO-GV (LoRA)Optimization Algorithm=MeZO, Tuning Strategy=LoRA, Guiding Vectors=true, Model=OPT-13B2026.01 | 32.7 | |
| ZO-AdaMU (LoRA)Optimization Algorithm=ZO-AdaMU, Tuning Strategy=LoRA, Model=OPT-13B2026.01 | 32.4 | |
| SubZero-GV (LoRA)Optimization Algorithm=SubZero, Tuning Strategy=LoRA, Guiding Vectors=true, Model=OPT-13B2026.01 | 32.4 | |
| bvLLM=OPT-6.7B, Params=0.02‰2025.09 | 32.4 | |
| ZO-AdaMU (2x)Optimization Algorithm=ZO-AdaMU, Tuning Strategy=Full Tuning, Training Steps=2x, Model=OPT-13B2026.01 | 32 | |
| SubZero (FT)Optimization Algorithm=SubZero, Tuning Strategy=Full Tuning, Guiding Vectors=false, Model=OPT-13B2026.01 | 32 | |
| SubZero (Prefix)Optimization Algorithm=SubZero, Tuning Strategy=Prefix Tuning, Guiding Vectors=false, Model=OPT-13B2026.01 | 32 | |
| MeZO-GV (FT)Optimization Algorithm=MeZO, Tuning Strategy=Full Tuning, Guiding Vectors=true, Model=OPT-13B2026.01 | 31.7 | |
| MeZO (LoRA)Optimization Algorithm=MeZO, Tuning Strategy=LoRA, Guiding Vectors=false, Model=OPT-13B2026.01 | 31.4 | |
| SubZero-GV (FT)Optimization Algorithm=SubZero, Tuning Strategy=Full Tuning, Guiding Vectors=true, Model=OPT-13B2026.01 | 31.3 | |
| SubZero (LoRA)Optimization Algorithm=SubZero, Tuning Strategy=LoRA, Guiding Vectors=false, Model=OPT-13B2026.01 | 31.3 | |
| FTOptimization Algorithm=Adam, Tuning Strategy=Full Tuning, Model=OPT-13B2026.01 | 31.3 | |
| LoRALLM=OPT-6.7B, Params=0.6‰2025.09 | 31.1 | |
| MeZO (FT)Optimization Algorithm=MeZO, Tuning Strategy=Full Tuning, Guiding Vectors=false, Model=OPT-13B2026.01 | 30.9 | |
| MeZO-GV (Prefix)Optimization Algorithm=MeZO, Tuning Strategy=Prefix Tuning, Guiding Vectors=true, Model=OPT-13B2026.01 | 30.9 | |
| PrefixLLM=OPT-6.7B, Params=0.2‰2025.09 | 30.5 | |
| ZO-AdaMU (Prefix)Optimization Algorithm=ZO-AdaMU, Tuning Strategy=Prefix Tuning, Model=OPT-13B2026.01 | 30.4 | |
| ICLEvaluation Protocol=In-context Learning, Model=OPT-13B2026.01 | 29.6 | |
| MeZO (Prefix)Optimization Algorithm=MeZO, Tuning Strategy=Prefix Tuning, Guiding Vectors=false, Model=OPT-13B2026.01 | 28.9 | |
| bvLLM=OPT-1.3B, Params=0.04‰2025.09 | 28.2 | |
| LoRALLM=OPT-1.3B, Params=1.2‰2025.09 | 27.9 | |
| bqLLM=OPT-6.7B, Params=0.02‰2025.09 | 27.9 | |
| ICLLLM=OPT-6.7B, Params=0‰2025.09 | 27.5 | |
| HiZOO (Prefix)Optimization Algorithm=HiZOO, Tuning Strategy=Prefix Tuning, Model=OPT-13B2026.01 | 25.3 | |
| PrefixLLM=OPT-1.3B, Params=0.4‰2025.09 | 25.2 | |
| HiZOO (LoRA)Optimization Algorithm=HiZOO, Tuning Strategy=LoRA, Model=OPT-13B2026.01 | 25.1 | |
| HiZOOOptimization Algorithm=HiZOO, Tuning Strategy=Full Tuning, Model=OPT-13B2026.01 | 25 | |
| MeZO-GV(Prefix)Backbone=OPT-1.3B, Optimization Method=MeZO, Adaptation Technique=Prefix Tuning, Guiding Vectors (GV)=true2026.01 | 24.8 | |
| MeZO-GV(LoRA)Backbone=OPT-1.3B, Optimization Method=MeZO, Adaptation Technique=LoRA, Guiding Vectors (GV)=true2026.01 | 24.4 | |
| MeZO-GV(FT)Backbone=OPT-1.3B, Optimization Method=MeZO, Adaptation Technique=Fine-tuning, Guiding Vectors (GV)=true2026.01 | 24.1 | |
| MeZO(Prefix)Backbone=OPT-1.3B, Optimization Method=MeZO, Adaptation Technique=Prefix Tuning2026.01 | 23.2 | |
| MeZO(LoRA)Backbone=OPT-1.3B, Optimization Method=MeZO, Adaptation Technique=LoRA2026.01 | 23.1 | |
| MeZO(FT)Backbone=OPT-1.3B, Optimization Method=MeZO, Adaptation Technique=Fine-tuning2026.01 | 21.9 | |
| bqLLM=OPT-1.3B, Params=0.04‰2025.09 | 21.4 | |
| ICLBackbone=OPT-1.3B, Learning Strategy=In-context Learning2026.01 | 20.5 | |
| ICLLLM=OPT-1.3B, Params=0‰2025.09 | 20.3 | |
| Zero-shot w/o finetuneModel Backbone=Llama2-7B, Fine-tuning Strategy=None, Optimization Algorithm=None, Number of training examples=10002026.06 | 19.73 | |
| Zero-ShotLLM=OPT-6.7B, Params=0‰2025.09 | 17.7 | |
| bkLLM=OPT-6.7B, Params=0.02‰2025.09 | 16.8 | |
| Zero-shotEvaluation Protocol=Zero-shot, Model=OPT-13B2026.01 | 14.6 | |
| Zero-shotBackbone=OPT-1.3B, Learning Strategy=Zero-shot2026.01 | 11.2 | |
| Zero-ShotLLM=OPT-1.3B, Params=0‰2025.09 | 11.1 | |
| bkLLM=OPT-1.3B, Params=0.04‰2025.09 | 10.4 |