GPU Kernel Optimization on KernelBench Level 1
2.42Geometric MeanKernelPro
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| KernelProHardware=A1002026.06 | 2.42 | 1 | — | — | — | — | — | — | — | — | |
| KERNELBLASTERHardware Setup=H100, Agent's Models=GPT-4.1 and GPT-5.0, Hyperparameter=10 iterations, 10 rollout steps per iteration, Test Harness=Functionality Check against PyTorch baselines2026.02 | 1.497 | 0.86 | 2.817 | 1.137 | 0.0647 | 32.74 | 0.7037 | 0.2963 | — | — | |
| KernelBlasterHardware=A1002026.06 | 1.43 | — | — | — | — | — | — | — | — | — | |
| AI CUDA EngineerHardware Setup=L40S, Agent's Models=O4-mini, Claude 3.7 Sonnet, Gemini 2.5 Pro, GPT-4.1, and O3, Hyperparameter=10 generations; 8 proposals sampled per generation; top 4 evaluated, Test Harness=LLM Soft-Verification2026.02 | 1.101 | 0.82 | 1.728 | 1.202 | 0.0659 | 32.35 | 0.7439 | 0.2561 | — | — | |
| KERNELBLASTERHardware Setup=L40S, Agent's Models=GPT-4.1 and GPT-5.0, Hyperparameter=10 iterations, 10 rollout steps per iteration, Test Harness=Functionality Check against PyTorch baselines2026.02 | 1.08 | 0.93 | 4.502 | 1.037 | 0.0801 | 183.96 | 0.5972 | 0.4028 | — | — | |
| AI CUDA EngineerHardware Setup=H100, Agent's Models=O4-mini, Claude 3.7 Sonnet, Gemini 2.5 Pro, GPT-4.1, and O3, Hyperparameter=10 generations; 8 proposals sampled per generation; top 4 evaluated, Test Harness=LLM Soft-Verification2026.02 | 1.025 | 0.82 | 1.935 | 1.085 | 0.0492 | 54.4 | 0.642 | 0.358 | — | — | |
| IREEHardware Setup=L40S, Hyperparameter=-O3 optimization level with LLVMGPU passes, Test Harness=NCU profiling, iree-run-module2026.02 | 0.268 | 0.85 | 0.856 | 0.518 | 0.0035 | 9.01 | 0.2118 | 0.7882 | — | — | |
| Astra2026.03 | — | — | — | — | — | — | — | — | 0.43 | — | |
| CudaForge2026.03 | — | — | — | — | — | — | — | — | 0.54 | — | |
| CudaForgeLLM=o3, GPU=RTX 6000, Metric Type=ArithMean (A), Tasks=100 tasks2026.06 | — | — | — | — | — | — | — | — | — | 1.45 | |
| KernelBlasterLLM=GPT-4.1/5.0, GPU=H100, Metric Type=GeoMean (G), Tasks=Full L12026.06 | — | — | — | — | — | — | — | — | — | 1.43 | |
| KernelProLLM=Sonnet 4.6, GPU=A100, Metric Type=GeoMean (G), Tasks=100 tasks2026.06 | — | — | — | — | — | — | — | — | — | 2.42 | |
| KernelSkill2026.03 | — | — | — | — | — | — | — | — | 0.62 | — | |
| Kevin-32B2026.03 | — | — | — | — | — | — | — | — | 0.16 | — | |
| PRAGMA2026.03 | — | — | — | — | — | — | — | — | 0.45 | — | |
| QiMeng2026.03 | — | — | — | — | — | — | — | — | 0.59 | — | |
| STARK2026.03 | — | — | — | — | — | — | — | — | 0.71 | — | |
| StitchCUDALLM=Qwen3-32B (Coder), GPT-5.2 (Planner/Verifier), GPU=H200, Metric Type=ArithMean (A), Tasks=20 test tasks, Training Protocol=In-distribution (trained on 80% of KernelBench)2026.06 | — | — | — | — | — | — | — | — | — | 3.54 |