Test Generation on TESTEVAL
95.64Syntax CorrectnessSAMPLING
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| SAMPLINGBackbone=GEMMA-1.1-2B-IT, Temperature=0.22026.01 | 95.64 | 1.36 | 1.36 | 1.44 | 1.41 | 1.37 | 1.42 | 1.27 | 1.35 | |
| SAMPLINGModel=GEMMA-1.1-2B-IT, Temperature=0.22026.01 | 95.64 | 1.36 | 1.36 | 1.44 | 1.41 | 1.37 | 1.42 | 1.27 | 1.35 | |
| SAMPLINGBackbone=GEMMA-1.1-2B-IT, Temperature=0.42026.01 | 95.4 | 1.36 | 1.26 | 2.33 | 2.31 | 2.29 | 2.32 | 2.2 | 2.27 | |
| SAMPLINGModel=GEMMA-1.1-2B-IT, Temperature=0.42026.01 | 95.4 | 1.36 | 1.26 | 2.33 | 2.31 | 2.29 | 2.32 | 2.2 | 2.27 | |
| NUCLEUS SAMPLINGTop-p=0.9, Temperature=0.8, Backbone=Gemma-1.1-2b-it2026.01 | 93.45 | 1.19 | 1.12 | 4.15 | 3.7 | 4.1 | 4.14 | 3.59 | 3.66 | |
| SAMPLINGBackbone=GEMMA-1.1-2B-IT, Temperature=0.62026.01 | 93.45 | 1.29 | 0 | 3.01 | 2.84 | 2.94 | 2.99 | 2.7 | 2.77 | |
| SAMPLINGModel=GEMMA-1.1-2B-IT, Temperature=0.62026.01 | 93.45 | 1.29 | 0 | 3.01 | 2.84 | 2.94 | 2.99 | 2.7 | 2.77 | |
| NUCLEUS SAMPLINGTop-p=0.95, Temperature=0.8, Backbone=Gemma-1.1-2b-it2026.01 | 92.57 | 1.21 | 1.19 | 5.09 | 4.88 | 5.07 | 5.09 | 4.73 | 4.84 | |
| SAMPLINGBackbone=GEMMA-1.1-2B-IT, Temperature=0.82026.01 | 91.79 | 1.19 | 0 | 3.54 | 3.37 | 3.46 | 3.51 | 3.21 | 3.3 | |
| SAMPLINGModel=GEMMA-1.1-2B-IT, Temperature=0.82026.01 | 91.79 | 1.19 | 0 | 3.54 | 3.37 | 3.46 | 3.51 | 3.21 | 3.3 | |
| STARS 0.1Backbone=GEMMA-1.1-2B-IT, Temperature=0.22026.01 | 91.52 | 1.45 | 1.36 | 7.28 | 6.5 | 7.23 | 7.25 | 6.36 | 6.4 | |
| STARSModel=GEMMA-1.1-2B-IT, Temperature=0.2, Scale=0.12026.01 | 91.52 | 1.45 | 1.36 | 7.28 | 6.5 | 7.23 | 7.25 | 6.36 | 6.4 | |
| STARS 0.1Backbone=GEMMA-1.1-2B-IT, Temperature=0.42026.01 | 90.43 | 1.48 | 1.43 | 8.44 | 7.69 | 8.41 | 8.44 | 7.58 | 7.63 | |
| STARSModel=GEMMA-1.1-2B-IT, Temperature=0.4, Scale=0.12026.01 | 90.43 | 1.48 | 1.43 | 8.44 | 7.69 | 8.41 | 8.44 | 7.58 | 7.63 | |
| SAMPLINGBackbone=GEMMA-1.1-2B-IT, Temperature=1.02026.01 | 89.9 | 1.02 | 0.95 | 5.36 | 5.06 | 5.32 | 5.35 | 4.98 | 5.04 | |
| SAMPLINGModel=GEMMA-1.1-2B-IT, Temperature=12026.01 | 89.9 | 1.02 | 0.95 | 5.36 | 5.06 | 5.32 | 5.35 | 4.98 | 5.04 | |
| STARS 0.1Backbone=GEMMA-1.1-2B-IT, Temperature=0.62026.01 | 89.19 | 1.71 | 1.43 | 12.1 | 11.22 | 12.03 | 12.1 | 10.99 | 11.15 | |
| STARSModel=GEMMA-1.1-2B-IT, Temperature=0.6, Scale=0.12026.01 | 89.19 | 1.71 | 1.43 | 12.1 | 11.22 | 12.03 | 12.1 | 10.99 | 11.15 | |
| SAMPLINGBackbone=GEMMA-1.1-2B-IT, Temperature=1.22026.01 | 88.48 | 1.36 | 1.19 | 8.7 | 7.85 | 8.63 | 8.66 | 7.69 | 7.78 | |
| STARS 0.1Backbone=GEMMA-1.1-2B-IT, Temperature=0.82026.01 | 87.83 | 1.45 | 1.12 | 10.77 | 9.88 | 10.64 | 10.72 | 9.65 | 9.78 | |
| STARSModel=GEMMA-1.1-2B-IT, Temperature=0.8, Scale=0.12026.01 | 87.83 | 1.45 | 1.12 | 10.77 | 9.88 | 10.64 | 10.72 | 9.65 | 9.78 | |
| STARS 0.1Backbone=GEMMA-1.1-2B-IT, Temperature=1.02026.01 | 86.48 | 1.43 | 1.17 | 10.03 | 9.32 | 9.97 | 10.02 | 9.12 | 9.25 | |
| STARSModel=GEMMA-1.1-2B-IT, Temperature=1, Scale=0.12026.01 | 86.48 | 1.43 | 1.17 | 10.03 | 9.32 | 9.97 | 10.02 | 9.12 | 9.25 | |
| SAMPLINGBackbone=GEMMA-1.1-2B-IT, Temperature=1.42026.01 | 85.5 | 1.26 | 1.14 | 8.76 | 8 | 8.69 | 8.76 | 7.85 | 7.94 | |
| STARS 0.1Backbone=GEMMA-1.1-2B-IT, Temperature=1.22026.01 | 85.29 | 1.43 | 1.26 | 11.32 | 10.43 | 10.99 | 11.28 | 9.89 | 10.33 | |
| STARS 0.1Backbone=GEMMA-1.1-2B-IT, Temperature=1.42026.01 | 83.64 | 1.45 | 1.31 | 13.91 | 13.31 | 13.82 | 13.9 | 13.04 | 13.25 | |
| SAMPLINGBackbone=GEMMA-1.1-2B-IT, Temperature=1.62026.01 | 82.21 | 1.1 | 0.95 | 9.3 | 8.68 | 9.21 | 9.29 | 8.51 | 8.63 | |
| STARS 0.1Backbone=GEMMA-1.1-2B-IT, Temperature=1.62026.01 | 80.9 | 1.26 | 1.07 | 11.84 | 10.85 | 11.72 | 11.79 | 10.54 | 10.71 | |
| STARS 0.5Backbone=GEMMA-1.1-2B-IT, Temperature=0.42026.01 | 80.31 | 4.38 | 4 | 40.03 | 35.96 | 38.39 | 39.36 | 33.73 | 35.07 | |
| STARSModel=GEMMA-1.1-2B-IT, Temperature=0.4, Scale=0.52026.01 | 80.31 | 4.38 | 4 | 40.03 | 35.96 | 38.39 | 39.36 | 33.73 | 35.07 | |
| STARS 0.5Backbone=GEMMA-1.1-2B-IT, Temperature=1.02026.01 | 79.76 | 3.83 | 3.45 | 34.76 | 30.93 | 33.81 | 34.45 | 29.58 | 30.4 | |
| STARSModel=GEMMA-1.1-2B-IT, Temperature=1, Scale=0.52026.01 | 79.76 | 3.83 | 3.45 | 34.76 | 30.93 | 33.81 | 34.45 | 29.58 | 30.4 | |
| STARS 0.5Backbone=GEMMA-1.1-2B-IT, Temperature=0.82026.01 | 79.43 | 3.79 | 3.45 | 32.96 | 30.02 | 31.7 | 32.59 | 28.23 | 29.43 | |
| STARSModel=GEMMA-1.1-2B-IT, Temperature=0.8, Scale=0.52026.01 | 79.43 | 3.79 | 3.45 | 32.96 | 30.02 | 31.7 | 32.59 | 28.23 | 29.43 | |
| STARS_0.5Top-p=0.95, Temperature=0.8, Backbone=Gemma-1.1-2b-it2026.01 | 79.36 | 4.4 | 3.74 | 35.69 | 31.65 | 34.13 | 34.99 | 29.47 | 30.65 | |
| STARS 0.5Backbone=GEMMA-1.1-2B-IT, Temperature=0.62026.01 | 79.24 | 3.86 | 3.24 | 34.07 | 30.38 | 32.04 | 33.22 | 28 | 29.46 | |
| STARSModel=GEMMA-1.1-2B-IT, Temperature=0.6, Scale=0.52026.01 | 79.24 | 3.86 | 3.24 | 34.07 | 30.38 | 32.04 | 33.22 | 28 | 29.46 | |
| STARS 0.5Backbone=GEMMA-1.1-2B-IT, Temperature=0.22026.01 | 78.93 | 4.38 | 4.02 | 39.03 | 35.05 | 37.38 | 38.67 | 32.94 | 34.39 | |
| STARSModel=GEMMA-1.1-2B-IT, Temperature=0.2, Scale=0.52026.01 | 78.93 | 4.38 | 4.02 | 39.03 | 35.05 | 37.38 | 38.67 | 32.94 | 34.39 | |
| STARS_0.5Top-p=0.9, Temperature=0.8, Backbone=Gemma-1.1-2b-it2026.01 | 78.81 | 3.93 | 3.55 | 35.73 | 31.62 | 34.25 | 34.88 | 29.79 | 30.47 | |
| STARS 0.1Backbone=GEMMA-1.1-2B-IT, Temperature=1.82026.01 | 78 | 0.93 | 0.81 | 11.86 | 10.71 | 11.81 | 11.86 | 10.54 | 10.68 | |
| STARS 0.5Backbone=GEMMA-1.1-2B-IT, Temperature=1.22026.01 | 76.95 | 3.93 | 3.6 | 37.04 | 33.78 | 34.72 | 36.16 | 30.91 | 32.66 | |
| SAMPLINGBackbone=GEMMA-1.1-2B-IT, Temperature=1.82026.01 | 75.76 | 0.95 | 0.83 | 12.45 | 11.49 | 12.38 | 12.43 | 11.31 | 11.45 | |
| STARS 0.5Backbone=GEMMA-1.1-2B-IT, Temperature=1.42026.01 | 74.69 | 2.88 | 2.52 | 28.94 | 25.91 | 27.02 | 28.13 | 23.47 | 24.87 | |
| STARSModel=QWEN3-1.7B, Temperature=0.4, Scale=0.52026.01 | 73.88 | 42.79 | 30.74 | 91.3 | 87.01 | 82.95 | 87.37 | 74.4 | 81 | |
| STARSModel=QWEN3-1.7B, Temperature=0.2, Scale=0.52026.01 | 73.4 | 41.81 | 29.95 | 91.35 | 87.13 | 82.95 | 87.61 | 74.28 | 81.32 | |
| STARSModel=QWEN3-1.7B, Temperature=0.6, Scale=0.52026.01 | 72.79 | 43.02 | 31.05 | 90.73 | 86.35 | 82.39 | 87.25 | 73.77 | 80.88 | |
| STARSModel=QWEN3-1.7B, Temperature=1, Scale=0.52026.01 | 72.21 | 41.86 | 30.79 | 91.05 | 86.81 | 82.18 | 87.31 | 73.59 | 81.02 | |
| STARSModel=QWEN3-1.7B, Temperature=0.8, Scale=0.52026.01 | 71.55 | 41.31 | 29.07 | 92.22 | 87.86 | 83 | 88.42 | 74.39 | 82.14 | |
| STARS 0.5Backbone=GEMMA-1.1-2B-IT, Temperature=1.62026.01 | 71.26 | 2.48 | 2.17 | 27.23 | 24.11 | 25.26 | 26.57 | 21.91 | 23.34 | |
| SAMPLINGBackbone=GEMMA-1.1-2B-IT, Temperature=2.02026.01 | 68.86 | 1.05 | 0.69 | 11.74 | 10.34 | 11.41 | 11.57 | 9.89 | 10.1 | |
| STARS 0.1Backbone=GEMMA-1.1-2B-IT, Temperature=2.02026.01 | 68.71 | 1.19 | 1.07 | 13.23 | 11.91 | 13 | 13.18 | 11.53 | 11.82 | |
| STARS 0.5Backbone=GEMMA-1.1-2B-IT, Temperature=1.82026.01 | 66.76 | 2.29 | 1.9 | 24.51 | 21.48 | 23.11 | 23.97 | 19.79 | 20.76 | |
| STARSModel=QWEN3-1.7B, Temperature=0.2, Scale=0.12026.01 | 62.93 | 41.11 | 18.81 | 86.25 | 80.22 | 80.67 | 83.6 | 71.72 | 75.92 | |
| STARSModel=QWEN3-1.7B, Temperature=0.4, Scale=0.12026.01 | 62.69 | 41.6 | 19.74 | 87.14 | 81.17 | 81.82 | 84.46 | 73.15 | 77.13 | |
| STARSModel=QWEN3-1.7B, Temperature=0.6, Scale=0.12026.01 | 61.57 | 41.1 | 19.43 | 86.84 | 80.95 | 81.08 | 83.99 | 72.24 | 76.53 | |
| STARSModel=QWEN3-1.7B, Temperature=1, Scale=0.12026.01 | 61.1 | 39.9 | 19.17 | 86.77 | 81.1 | 80.54 | 83.62 | 71.71 | 76.36 | |
| STARSModel=QWEN3-1.7B, Temperature=0.8, Scale=0.12026.01 | 61.1 | 39.9 | 19.17 | 86.77 | 81.1 | 80.54 | 83.62 | 71.71 | 76.36 | |
| STARS 0.5Backbone=GEMMA-1.1-2B-IT, Temperature=2.02026.01 | 58.26 | 1.64 | 1.36 | 16.51 | 14.44 | 16.04 | 16.15 | 13.82 | 13.98 | |
| SAMPLINGModel=QWEN3-1.7B, Temperature=12026.01 | 18.07 | 11.88 | 4.19 | 59.95 | 55.05 | 57.18 | 58.66 | 51.31 | 53.39 | |
| SAMPLINGModel=QWEN3-1.7B, Temperature=0.82026.01 | 13.86 | 8.17 | 2.83 | 37.06 | 33.73 | 36.15 | 36.47 | 32.43 | 33.04 | |
| SAMPLINGModel=QWEN3-1.7B, Temperature=0.62026.01 | 11.69 | 6.19 | 2.26 | 23.92 | 21.79 | 22.76 | 23.24 | 20.35 | 20.89 | |
| SAMPLINGModel=QWEN3-1.7B, Temperature=0.42026.01 | 9 | 3.83 | 1.52 | 13.56 | 12.26 | 13.29 | 13.43 | 11.86 | 12.08 | |
| SAMPLINGModel=QWEN3-1.7B, Temperature=0.22026.01 | 8.17 | 2.69 | 1.17 | 4.71 | 4.16 | 4.7 | 4.71 | 4.13 | 4.14 |