Constraint discovery from data-code pairs on ICDBench
61Passed RateprismaDV
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| prismaDVLLM=gpt-5, Task-aware?=true, Generates test?=true, Avg. num. const.=2.62026.04 | 61 | 2 | 48 | 15 | 17 | 87.8 | |
| prismaDVLLM=gpt-5-mini, Task-aware?=true, Generates test?=true, Avg. num. const.=2.52026.04 | 60 | 3 | 46 | 17 | 20 | 85.7 | |
| prismaDVLLM=gemini-2.5-flash, Task-aware?=true, Generates test?=true, Avg. num. const.=2.12026.04 | 59 | 4 | 40 | 23 | 27 | 81.4 | |
| prismaDVLLM=gemini-2.5-pro, Task-aware?=true, Generates test?=true, Avg. num. const.=1.72026.04 | 59 | 4 | 42 | 21 | 25 | 82.5 | |
| prismaDVLLM=gpt-4.1, Task-aware?=true, Generates test?=true, Avg. num. const.=2.12026.04 | 56 | 6 | 38 | 25 | 31 | 77.8 | |
| deequTask-aware?=false, Generates test?=true, Avg. num. const.=1.92026.04 | 44 | 19 | 27 | 36 | 55 | 61.5 | |
| pocketflow-agentTask-aware?=true, Generates test?=true, Avg. num. const.=2.22026.04 | 38 | 25 | 32 | 31 | 56 | 57.6 | |
| few-shotLLM=gpt-5, Task-aware?=true, Generates test?=true, Avg. num. const.=2.52026.04 | 36 | 27 | 54 | 9 | 36 | 66.7 | |
| swe-agentTask-aware?=true, Generates test?=true, Avg. num. const.=2.22026.04 | 35 | 28 | 54 | 9 | 37 | 65.4 | |
| few-shotLLM=gemini-2.5-pro, Task-aware?=true, Generates test?=true, Avg. num. const.=1.92026.04 | 34 | 29 | 54 | 9 | 38 | 64.2 | |
| few-shotLLM=gpt-5-mini, Task-aware?=true, Generates test?=true, Avg. num. const.=2.42026.04 | 33 | 30 | 59 | 4 | 37 | 66 | |
| tensorflow-dvTask-aware?=false, Generates test?=true2026.04 | 32 | 31 | 32 | 31 | 62 | 50.8 | |
| zero-shotLLM=gpt-5-mini, Task-aware?=true, Generates test?=true, Avg. num. const.=3.52026.04 | 30 | 33 | 57 | 6 | 39 | 60.6 | |
| zero-shotLLM=gpt-5, Task-aware?=true, Generates test?=true, Avg. num. const.=3.12026.04 | 30 | 33 | 57 | 6 | 39 | 60.6 | |
| few-shotLLM=gpt-4.1, Task-aware?=true, Generates test?=true, Avg. num. const.=2.12026.04 | 29 | 34 | 52 | 11 | 45 | 56.3 | |
| few-shotLLM=gemini-2.5-flash, Task-aware?=true, Generates test?=true, Avg. num. const.=2.02026.04 | 27 | 36 | 51 | 12 | 48 | 52.9 | |
| zero-shotLLM=gemini-2.5-flash, Task-aware?=true, Generates test?=true, Avg. num. const.=2.42026.04 | 22 | 41 | 59 | 4 | 45 | 49.4 | |
| zero-shotLLM=gemini-2.5-pro, Task-aware?=true, Generates test?=true, Avg. num. const.=2.82026.04 | 19 | 44 | 60 | 3 | 47 | 44.7 | |
| zero-shotLLM=gpt-4.1, Task-aware?=true, Generates test?=true, Avg. num. const.=3.12026.04 | 19 | 44 | 56 | 7 | 51 | 42.7 | |
| isolation-forestTask-aware?=false, Generates test?=false2026.04 | 14 | 49 | 51 | 12 | 61 | 31.5 | |
| one-class-svmTask-aware?=false, Generates test?=false2026.04 | 0 | 63 | 61 | 2 | 65 | 0 | |
| stats-noveltyTask-aware?=false, Generates test?=false2026.04 | 0 | 63 | 63 | 0 | 63 | 0 |