Gradient Computation on VLM Gradient Computation 8-32B Models
1.87Speedup vs. PEFT DoRAFused
Evaluation results
| Method | Links | ||
|---|---|---|---|
| FusedModel=Mistral-Sm-24B, GPU=H200, r=384, Precision=bf16, Sequence Length=4096, Batch Size=1, Gradient Accumulation=8, Loss Tokens=10242026.03 | 1.87 | 1.21 | |
| FusedModel=Mistral-Sm-24B, GPU=B200, r=384, Precision=bf16, Sequence Length=4096, Batch Size=1, Gradient Accumulation=8, Loss Tokens=10242026.03 | 1.87 | 1.23 | |
| FusedModel=Qwen2.5-VL-32B, GPU=B200, r=384, Precision=bf16, Sequence Length=4096, Batch Size=1, Gradient Accumulation=8, Loss Tokens=10242026.03 | 1.74 | 1.22 | |
| FusedModel=Qwen2.5-VL-32B, GPU=H200, r=384, Precision=bf16, Sequence Length=4096, Batch Size=1, Gradient Accumulation=8, Loss Tokens=10242026.03 | 1.73 | 1.2 | |
| FusedModel=Qwen3-VL-32B, GPU=B200, r=384, Precision=bf16, Sequence Length=4096, Batch Size=1, Gradient Accumulation=8, Loss Tokens=10242026.03 | 1.67 | 1.21 | |
| FusedModel=Qwen3-VL-32B, GPU=H200, r=384, Precision=bf16, Sequence Length=4096, Batch Size=1, Gradient Accumulation=8, Loss Tokens=10242026.03 | 1.66 | 1.18 | |
| FusedModel=Mistral-Sm-24B, GPU=RTX, r=384, Precision=bf16, Sequence Length=4096, Batch Size=1, Gradient Accumulation=8, Loss Tokens=10242026.03 | 1.66 | 1.22 | |
| FusedModel=Gemma3-27B, GPU=H200, r=384, Precision=bf16, Sequence Length=4096, Batch Size=1, Gradient Accumulation=8, Loss Tokens=10242026.03 | 1.61 | 1.19 | |
| FusedModel=Qwen3.5-27B, GPU=H200, r=384, Precision=bf16, Sequence Length=4096, Batch Size=1, Gradient Accumulation=8, Loss Tokens=10242026.03 | 1.57 | 1.21 | |
| FusedModel=Qwen3.5-27B, GPU=B200, r=384, Precision=bf16, Sequence Length=4096, Batch Size=1, Gradient Accumulation=8, Loss Tokens=10242026.03 | 1.57 | 1.23 | |
| FusedModel=Gemma3-27B, GPU=B200, r=384, Precision=bf16, Sequence Length=4096, Batch Size=1, Gradient Accumulation=8, Loss Tokens=10242026.03 | 1.56 | 1.21 | |
| FusedModel=Gemma3-27B, GPU=RTX, r=384, Precision=bf16, Sequence Length=4096, Batch Size=1, Gradient Accumulation=8, Loss Tokens=10242026.03 | 1.53 | 1.2 | |
| FusedModel=Qwen3.5-27B, GPU=RTX, r=384, Precision=bf16, Sequence Length=4096, Batch Size=1, Gradient Accumulation=8, Loss Tokens=10242026.03 | 1.51 | 1.22 | |
| FusedModel=Qwen3-VL-8B, GPU=H200, r=384, Precision=bf16, Sequence Length=4096, Batch Size=1, Gradient Accumulation=8, Loss Tokens=10242026.03 | 1.5 | 1.21 | |
| FusedModel=Qwen3-VL-8B, GPU=B200, r=384, Precision=bf16, Sequence Length=4096, Batch Size=1, Gradient Accumulation=8, Loss Tokens=10242026.03 | 1.47 | 1.24 | |
| FusedModel=Qwen3-VL-8B, GPU=RTX, r=384, Precision=bf16, Sequence Length=4096, Batch Size=1, Gradient Accumulation=8, Loss Tokens=10242026.03 | 1.46 | 1.23 |