PDE Solving on CodePDE
0Advection ErrorAutoNumerics
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| AutoNumericsLLM=GPT-4.12026.02 | 0 | 0.0018 | 0.0001 | 0.0182 | 0 | 0 | — | |
| Gemini 2.0 FlashProtocol=Reasoning + Debugging + Refinement, Strategy=best of 122025.05 | 0.0824 | 0.0106 | 1.76 | 1.72 | 0.478 | 0.0026 | — | |
| Gemini 2.0 FlashMethod Category=CodePDE, Evaluation Protocol=Reasoning + Debugging + Refinement, Number of Samples (Best of N)=122025.05 | 0.0824 | 0.0106 | 1.76 | 1.72 | 0.478 | — | 0.0026 | |
| o3Protocol=Reasoning + Debugging, Strategy=best of 322025.05 | 0.0974 | 0.0369 | 19.8 | 2.8 | 0.492 | 0.0063 | — | |
| Gemini 2.0 ThinkingProtocol=Reasoning + Debugging + Refinement, Strategy=best of 122025.05 | 0.0974 | 0.027 | 1.74 | 2.41 | 0.48 | 0.0035 | — | |
| DeepSeek-V3Protocol=Reasoning + Debugging + Refinement, Strategy=best of 122025.05 | 0.0974 | 0.0259 | 1.74 | 1.56 | 0.48 | 0.0032 | — | |
| GPT-4oProtocol=Reasoning + Debugging + Refinement, Strategy=best of 122025.05 | 0.0974 | 0.0259 | 1.76 | 2.41 | 0.48 | 0.0035 | — | |
| o3Method Category=CodePDE, Evaluation Protocol=Reasoning + Debugging, Number of Samples (Best of N)=322025.05 | 0.0974 | 0.0369 | 19.8 | 2.8 | 0.492 | — | 0.0063 | |
| Gemini 2.0 ThinkingMethod Category=CodePDE, Evaluation Protocol=Reasoning + Debugging + Refinement, Number of Samples (Best of N)=122025.05 | 0.0974 | 0.027 | 1.74 | 2.41 | 0.48 | — | 0.0035 | |
| Gemini 2.5 ProMethod Category=CodePDE, Evaluation Protocol=Reasoning + Debugging + Refinement, Number of Samples (Best of N)=122025.05 | 0.0974 | 0.0315 | 9.94 | 2.77 | 0.492 | — | 0.0053 | |
| Qwen-2.5-MaxMethod Category=CodePDE, Evaluation Protocol=Reasoning + Debugging + Refinement, Number of Samples (Best of N)=122025.05 | 0.0974 | 0.026 | 9.13 | 1.49 | 0.48 | — | 0.0044 | |
| QwQ-PlusMethod Category=CodePDE, Evaluation Protocol=Reasoning + Debugging + Refinement, Number of Samples (Best of N)=122025.05 | 0.0974 | 0.0107 | 5.19 | 1.5 | 0.48 | — | 0.0033 | |
| Claude-3.7-sonnetMethod Category=CodePDE, Evaluation Protocol=Reasoning + Debugging + Refinement, Number of Samples (Best of N)=122025.05 | 0.0974 | 0.026 | 5.19 | 2.72 | 0.483 | — | 0.0044 | |
| DeepSeek-V3Method Category=CodePDE, Evaluation Protocol=Reasoning + Debugging + Refinement, Number of Samples (Best of N)=122025.05 | 0.0974 | 0.0259 | 1.74 | 1.56 | 0.48 | — | 0.0032 | |
| DeepSeek-R1Method Category=CodePDE, Evaluation Protocol=Reasoning + Debugging + Refinement, Number of Samples (Best of N)=122025.05 | 0.0974 | 0.0259 | 14.3 | 1.52 | 0.48 | — | 0.0048 | |
| GPT-4oMethod Category=CodePDE, Evaluation Protocol=Reasoning + Debugging + Refinement, Number of Samples (Best of N)=122025.05 | 0.0974 | 0.0259 | 1.76 | 2.41 | 0.48 | — | 0.0035 | |
| CodePDELLM=GPT-4.1, Setting=Reasoning + Debugging + Refinement, Evaluated=Best of 122026.02 | 0.101 | 0.0315 | 14.4 | 1.53 | 0.488 | 0.0051 | — | |
| Gemini 2.5 ProProtocol=Reasoning + Debugging, Strategy=best of 322025.05 | 0.101 | 0.0123 | 14.9 | 7.39 | 0.489 | 0.0058 | — | |
| o3Protocol=Reasoning + Debugging + Refinement, Strategy=best of 122025.05 | 0.101 | 0.0315 | 20.1 | 1.31 | 0.488 | 0.0053 | — | |
| Gemini 2.5 ProMethod Category=CodePDE, Evaluation Protocol=Reasoning + Debugging, Number of Samples (Best of N)=322025.05 | 0.101 | 0.0123 | 14.9 | 7.39 | 0.489 | — | 0.0058 | |
| Claude-3.5-haikuMethod Category=CodePDE, Evaluation Protocol=Reasoning + Debugging + Refinement, Number of Samples (Best of N)=122025.05 | 0.101 | 0.026 | 5.19 | 10.6 | 19.2 | — | 0.0122 | |
| GPT-4.1Method Category=CodePDE, Evaluation Protocol=Reasoning + Debugging + Refinement, Number of Samples (Best of N)=122025.05 | 0.101 | 0.0315 | 14.4 | 1.53 | 0.488 | — | 0.0051 | |
| o3Method Category=CodePDE, Evaluation Protocol=Reasoning + Debugging + Refinement, Number of Samples (Best of N)=122025.05 | 0.101 | 0.0315 | 20.1 | 1.31 | 0.488 | — | 0.0053 | |
| Reference Numerical Solvers2025.05 | 0.103 | 0.0355 | 0.229 | 1.89 | 0.48 | 0.0024 | — | |
| QwQ-PlusProtocol=Reasoning + Debugging, Strategy=best of 322025.05 | 0.103 | 0.0305 | 22 | 7.43 | 0.48 | 0.0076 | — | |
| Reference Numerical SolversMethod Category=Reference Numerical Solvers2025.05 | 0.103 | 0.0355 | 0.229 | 1.89 | 0.48 | — | 0.0024 | |
| QwQ-PlusMethod Category=CodePDE, Evaluation Protocol=Reasoning + Debugging, Number of Samples (Best of N)=322025.05 | 0.103 | 0.0305 | 22 | 7.43 | 0.48 | — | 0.0076 | |
| DeepSeek-R1Protocol=Reasoning + Debugging, Strategy=best of 322025.05 | 0.105 | 0.0276 | 22 | 7.46 | 0.48 | 0.0074 | — | |
| o3-miniProtocol=Reasoning + Debugging, Strategy=best of 322025.05 | 0.105 | 0.0276 | 21 | 4.48 | 0.483 | 0.0067 | — | |
| DeepSeek-R1Method Category=CodePDE, Evaluation Protocol=Reasoning + Debugging, Number of Samples (Best of N)=322025.05 | 0.105 | 0.0276 | 22 | 7.46 | 0.48 | — | 0.0074 | |
| Gemini 2.0 FlashProtocol=Reasoning + Debugging, Strategy=best of 322025.05 | 0.114 | 0.0297 | 21.9 | 4.5 | 0.48 | 0.0069 | — | |
| Gemini 2.0 FlashMethod Category=CodePDE, Evaluation Protocol=Reasoning + Debugging, Number of Samples (Best of N)=322025.05 | 0.114 | 0.0297 | 21.9 | 4.5 | 0.48 | — | 0.0069 | |
| Claude-3.7-sonnetProtocol=Reasoning + Debugging, Strategy=best of 322025.05 | 0.132 | 0.0259 | 5.19 | 4.26 | 0.48 | 0.0052 | — | |
| Claude-3.7-sonnetMethod Category=CodePDE, Evaluation Protocol=Reasoning + Debugging, Number of Samples (Best of N)=322025.05 | 0.132 | 0.0259 | 5.19 | 4.26 | 0.48 | — | 0.0052 | |
| GPT-4.1Protocol=Reasoning + Debugging, Strategy=best of 322025.05 | 0.15 | 0.0363 | 14.4 | 5.2 | 0.488 | 0.0072 | — | |
| GPT-4.1Method Category=CodePDE, Evaluation Protocol=Reasoning + Debugging, Number of Samples (Best of N)=322025.05 | 0.15 | 0.0363 | 14.4 | 5.2 | 0.488 | — | 0.0072 | |
| GPT-4oProtocol=Reasoning + Debugging, Strategy=best of 322025.05 | 0.155 | 0.0369 | 1.99 | 18.1 | 76.7 | 0.0174 | — | |
| GPT-4oMethod Category=CodePDE, Evaluation Protocol=Reasoning + Debugging, Number of Samples (Best of N)=322025.05 | 0.155 | 0.0369 | 1.99 | 18.1 | 76.7 | — | 0.0174 | |
| PyPDEtype=PDE Solving Software2025.05 | 0.203 | 0.0592 | 22.1 | 25.2 | — | — | — | |
| PyPDEMethod Category=PDE Solving Softwares2025.05 | 0.203 | 0.0592 | 22.1 | 25.2 | — | — | — | |
| UPS2026.02 | 0.22 | 3.73 | 5.57 | 0.45 | — | — | — | |
| UPStype=Neural Network baseline2025.05 | 0.22 | 3.73 | 5.57 | 0.45 | — | 0.012 | — | |
| UPSMethod Category=Neural Network & Foundation Model baselines2025.05 | 0.22 | 3.73 | 5.57 | 0.45 | — | — | — | |
| Claude-3.5-haikuProtocol=Reasoning + Debugging, Strategy=best of 322025.05 | 0.37 | 0.0323 | 22 | 23.4 | 100 | 0.0361 | — | |
| Claude-3.5-haikuMethod Category=CodePDE, Evaluation Protocol=Reasoning + Debugging, Number of Samples (Best of N)=322025.05 | 0.37 | 0.0323 | 22 | 23.4 | 100 | — | 0.0361 | |
| PDEformer2026.02 | 0.43 | 1.46 | — | — | — | — | — | |
| PDEformertype=Foundation Model baseline2025.05 | 0.43 | 0.46 | — | — | — | 0.0045 | — | |
| PDEformerMethod Category=Neural Network & Foundation Model baselines2025.05 | 0.43 | 1.46 | — | — | — | — | — | |
| Qwen-2.5-MaxProtocol=Reasoning + Debugging, Strategy=best of 322025.05 | 0.497 | 0.135 | 9.57 | 23.6 | 67.6 | 0.04 | — | |
| Qwen-2.5-MaxMethod Category=CodePDE, Evaluation Protocol=Reasoning + Debugging, Number of Samples (Best of N)=322025.05 | 0.497 | 0.135 | 9.57 | 23.6 | 67.6 | — | 0.04 | |
| GPT-4o miniProtocol=Reasoning + Debugging, Strategy=best of 322025.05 | 0.541 | 99.3 | 30.7 | 23.6 | 100 | 0.208 | — | |
| Gemini 2.0 ThinkingProtocol=Reasoning + Debugging, Strategy=best of 322025.05 | 0.554 | 0.0321 | 21.9 | 7.56 | 0.48 | 0.0107 | — | |
| Gemini 2.0 ThinkingMethod Category=CodePDE, Evaluation Protocol=Reasoning + Debugging, Number of Samples (Best of N)=322025.05 | 0.554 | 0.0321 | 21.9 | 7.56 | 0.48 | — | 0.0107 | |
| Dedalustype=PDE Solving Software2025.05 | 0.62 | 0.278 | 5.2 | 2.01 | — | — | — | |
| DedalusMethod Category=PDE Solving Softwares2025.05 | 0.62 | 0.278 | 5.2 | 2.01 | — | — | — | |
| o1-miniProtocol=Reasoning + Debugging, Strategy=best of 322025.05 | 0.645 | 0.034 | 15.5 | 23.6 | 82 | 0.0366 | — | |
| DeepSeek-V3Protocol=Reasoning + Debugging, Strategy=best of 322025.05 | 0.737 | 0.0602 | 21.8 | 2.41 | 100 | 0.0298 | — | |
| DeepSeek-V3Method Category=CodePDE, Evaluation Protocol=Reasoning + Debugging, Number of Samples (Best of N)=322025.05 | 0.737 | 0.0602 | 21.8 | 2.41 | 100 | — | 0.0298 | |
| FNO2026.02 | 0.77 | 0.78 | 0.14 | 9.5 | 0.98 | 0.0095 | — | |
| FNOtype=Neural Network baseline2025.05 | 0.77 | 0.78 | 0.14 | 9.5 | 0.98 | 0.0095 | — | |
| FNOMethod Category=Neural Network & Foundation Model baselines2025.05 | 0.77 | 0.78 | 0.14 | 9.5 | 0.98 | — | 0.0095 | |
| PINN2026.02 | 0.78 | 85 | 8 | — | — | — | — | |
| PINNtype=Neural Network baseline2025.05 | 0.78 | 85 | 8 | — | — | 0.0809 | — | |
| PINNMethod Category=Neural Network & Foundation Model baselines2025.05 | 0.78 | 85 | 8 | — | — | — | — | |
| ORCA2026.02 | 0.98 | 1.2 | 0.3 | 6.2 | — | — | — | |
| ORCAtype=Neural Network baseline2025.05 | 0.98 | 1.2 | 0.3 | 6.2 | — | 0.0122 | — | |
| ORCAMethod Category=Neural Network & Foundation Model baselines2025.05 | 0.98 | 1.2 | 0.3 | 6.2 | — | — | — | |
| o4-miniProtocol=Reasoning + Debugging, Strategy=best of 322025.05 | 1.21 | 0.0337 | 14.4 | 6.11 | 0.492 | 0.0112 | — | |
| U-Net2026.02 | 5 | 22 | 0.6 | 36 | — | — | — | |
| U-Nettype=Neural Network baseline2025.05 | 5 | 22 | 0.6 | 36 | — | 0.0698 | — | |
| U-NetMethod Category=Neural Network & Foundation Model baselines2025.05 | 5 | 22 | 0.6 | 36 | — | — | — | |
| Central Difference (Ill-designed)2026.02 | 705,000,000,000,000 | 1.64 | 12.3 | 385 | 23.4 | — | — |