Code distillation and example generation on GitHub Materials Science repositories 250 (subsampled)
75.6Execution Success Rate (Manual)CODEDISTILLER
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| CODEDISTILLERAgent Base Model=Claude 4-52025.11 | 75.6 | — | 75.6 | 74.1 | 19 | 21.9 | 1.9 | 1.71 | 2.26 | |
| CODEDISTILLERAgent Base Model=GPT-52025.11 | 69 | — | 69 | 60.5 | 20.3 | 47 | 2.2 | 0.7 | 1.68 | |
| CODEDISTILLERAgent Base Model=GPT-OSS-120B2025.11 | 29.6 | — | 29.6 | 25.9 | 13.8 | 16.3 | 2.4 | 0.09 | 0.13 |