GPU Kernel Optimization on KernelBench Level 2
70.6Fast1 ScoredaVinci-kernel-14B
Evaluation results
| Method | Links | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| daVinci-kernel-14Bbackbone=Qwen3-14B-Base2026.06 | 70.6 | — | — | — | — | — | — | — | — | — | 27.1 | 11.4 | 6.2 | |
| Dr. Kernel-14B (reproduce)reproduce=true2026.06 | 58.5 | — | — | — | — | — | — | — | — | — | 18.1 | 7.5 | 4.1 | |
| Claude-4.5-Sonnet*mode=zero-shot, data_source=Dr. Kernel paper2026.06 | 50 | — | — | — | — | — | — | — | — | — | 26.7 | 9.2 | 1.8 | |
| Dr. Kernel-14B*data_source=Dr. Kernel paper2026.06 | 49.2 | — | — | — | — | — | — | — | — | — | 25.6 | 7.4 | 2.1 | |
| GPT-5*mode=zero-shot, data_source=Dr. Kernel paper2026.06 | 46.7 | — | — | — | — | — | — | — | — | — | 28.6 | 13.1 | 3 | |
| Dr. Kernel-8B*data_source=Dr. Kernel paper2026.06 | 46 | — | — | — | — | — | — | — | — | — | 20 | 5 | 1.5 | |
| daVinci-kernel-8Bbackbone=Qwen3-8B-Base2026.06 | 44.8 | — | — | — | — | — | — | — | — | — | 22.1 | 13.1 | 7.9 | |
| AutoTriton*data_source=Dr. Kernel paper2026.06 | 30.6 | — | — | — | — | — | — | — | — | — | 9.2 | 2.6 | 0.5 | |
| GLM-4.7*mode=zero-shot, data_source=Dr. Kernel paper2026.06 | 30 | — | — | — | — | — | — | — | — | — | 20.5 | 8.5 | 3.5 | |
| Dr. Kernel-8B (reproduce)reproduce=true2026.06 | 26.8 | — | — | — | — | — | — | — | — | — | 9.4 | 5.2 | 2 | |
| Cold-Start-14B (reproduce)reproduce=true2026.06 | 20.9 | — | — | — | — | — | — | — | — | — | 12.4 | 7.9 | 4.1 | |
| Qwen3-32B*mode=zero-shot, data_source=Dr. Kernel paper2026.06 | 14 | — | — | — | — | — | — | — | — | — | 9.4 | 2.4 | 0.2 | |
| Qwen3-8B*mode=zero-shot, data_source=Dr. Kernel paper2026.06 | 13 | — | — | — | — | — | — | — | — | — | 5.6 | 2 | 1.1 | |
| Qwen3-Coder-A30BA3*mode=zero-shot, data_source=Dr. Kernel paper2026.06 | 12.6 | — | — | — | — | — | — | — | — | — | 5 | 1.5 | 0.3 | |
| Cold-Start-8B (reproduce)reproduce=true2026.06 | 11.8 | — | — | — | — | — | — | — | — | — | 8.6 | 5.9 | 3.5 | |
| Deepseek-V3.2-Thinking*mode=zero-shot, data_source=Dr. Kernel paper2026.06 | 11 | — | — | — | — | — | — | — | — | — | 6.5 | 2.5 | 0.5 | |
| Cold-Start-8B*data_source=Dr. Kernel paper2026.06 | 8.8 | — | — | — | — | — | — | — | — | — | 5.6 | 1.8 | 0.4 | |
| AI CUDA EngineerHardware Setup=L40S, Agent's Models=O4-mini, Claude 3.7 Sonnet, Gemini 2.5 Pro, GPT-4.1, and O3, Hyperparameter=10 generations; 8 proposals sampled per generation; top 4 evaluated2026.02 | — | 83 | 3.865 | 1.695 | 1.574 | 0.328 | 123.39 | 79.52 | 20.48 | — | — | — | — | |
| AI CUDA EngineerHardware Setup=H100, Agent's Models=O4-mini, Claude 3.7 Sonnet, Gemini 2.5 Pro, GPT-4.1, and O3, Hyperparameter=10 generations; 8 proposals sampled per generation; top 4 evaluated2026.02 | — | 82 | 1.356 | 1.214 | 1.17 | 0.402 | 3.651 | 61.33 | 38.67 | — | — | — | — | |
| Astra2026.03 | — | — | — | — | — | — | — | — | — | 0.73 | — | — | — | |
| CudaForge2026.03 | — | — | — | — | — | — | — | — | — | 0.89 | — | — | — | |
| CudaForgeLLM=o3, GPU=RTX 6000, Metric Type=ArithMean (A), Tasks=100 tasks2026.06 | — | — | 2.1 | — | — | — | — | — | — | — | — | — | — | |
| IREEHardware Setup=L40S, Hyperparameter=-O3 optimization level with LLVMGPU passes2026.02 | — | 83 | 1.137 | 0.279 | 0.298 | 0.0255 | 38.5 | 16.87 | 83.13 | — | — | — | — | |
| KernelBlasterLLM=GPT-4.1/5.0, GPU=H100, Metric Type=GeoMean (G), Tasks=Full L22026.06 | — | — | 2.5 | — | — | — | — | — | — | — | — | — | — | |
| KERNELBLASTERHardware Setup=L40S, Agent's Models=GPT-4.1 and GPT-5.0, Hyperparameter=10 iterations, 10 rollout steps per iteration2026.02 | — | 95 | 9.419 | 2.214 | 2.074 | 0.0488 | 362.29 | 72.6 | 27.4 | — | — | — | — | |
| KERNELBLASTERHardware Setup=H100, Agent's Models=GPT-4.1 and GPT-5.0, Hyperparameter=10 iterations, 10 rollout steps per iteration2026.02 | — | 81 | 10.223 | 2.592 | 2.291 | 0.111 | 213.65 | 84.85 | 15.15 | — | — | — | — | |
| KernelFoundryLLM=GPT-4.1/o3, GPU=RTX A6000, Tasks=40-task subset, Method=MAP-Elites evolutionary search2026.06 | — | — | 2.1 | — | — | — | — | — | — | — | — | — | — | |
| KernelProLLM=Sonnet 4.6, GPU=A100, Metric Type=GeoMean (G), Tasks=100 tasks2026.06 | — | — | 4.69 | — | — | — | — | — | — | — | — | — | — | |
| KernelSkill2026.03 | — | — | — | — | — | — | — | — | — | 1 | — | — | — | |
| Kevin-32B2026.03 | — | — | — | — | — | — | — | — | — | 0.61 | — | — | — | |
| PRAGMA2026.03 | — | — | — | — | — | — | — | — | — | 0.74 | — | — | — | |
| QiMeng2026.03 | — | — | — | — | — | — | — | — | — | 0.66 | — | — | — | |
| STARK2026.03 | — | — | — | — | — | — | — | — | — | 1 | — | — | — | |
| StitchCUDALLM=Qwen3-32B (Coder), GPT-5.2 (Planner/Verifier), GPU=H200, Metric Type=ArithMean (A), Tasks=20 test tasks, Training Protocol=In-distribution (trained on 80% of KernelBench)2026.06 | — | — | 1.82 | — | — | — | — | — | — | — | — | — | — |