Code Generation on HumanEval (Accuracy, NFE, T/F)
98.27AccSASFT
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| SASFTModel=Qwen3-8B-Base2025.07 | 98.27 | — | — | |
| SFT+PenaltyModel=Gemma-2-9B2025.07 | 97.02 | — | — | |
| SASFTModel=Qwen3-8B-Base, Training Set=Chinese 210k2025.07 | 96.92 | — | — | |
| SFTModel=Qwen3-8B-Base, Training Set=Chinese 210k2025.07 | 96.83 | — | — | |
| SFT+PenaltyModel=Qwen3-8B-Base, Training Set=Chinese 210k2025.07 | 96.83 | — | — | |
| SASFTModel=Gemma-2-9B, Dataset=Russian 110k2025.07 | 96.68 | — | — | |
| SFT+GRPOModel=Qwen3-8B-Base2025.07 | 96.44 | — | — | |
| SFT+PenaltyModel=Gemma-2-9B, Dataset=Russian 110k2025.07 | 96.44 | — | — | |
| SFT+GRPOModel=Qwen3-8B-Base, Training Set=Chinese 210k2025.07 | 96.3 | — | — | |
| SFTModel=Qwen3-8B-Base2025.07 | 95.87 | — | — | |
| SFT+GRPOModel=Gemma-2-9B2025.07 | 95.72 | — | — | |
| SASFTModel=Gemma-2-9B2025.07 | 95.67 | — | — | |
| SFTModel=Gemma-2-9B2025.07 | 95.62 | — | — | |
| SASFTModel=Llama-3.1-8B, Training Set=Chinese 210k2025.07 | 95.38 | — | — | |
| SFT+PenaltyModel=Qwen3-8B-Base, Dataset=Russian 110k2025.07 | 95 | — | — | |
| SFT+PenaltyModel=Qwen3-8B-Base2025.07 | 94.71 | — | — | |
| SFT+GRPOModel=Qwen3-8B-Base, Dataset=Russian 110k2025.07 | 94.33 | — | — | |
| SFTModel=Gemma-2-9B, Dataset=Russian 110k2025.07 | 94.23 | — | — | |
| SFTModel=Qwen3-8B-Base, Dataset=Russian 110k2025.07 | 94.13 | — | — | |
| SFT+GRPOModel=Gemma-2-9B, Training Set=Chinese 210k2025.07 | 94.09 | — | — | |
| SFT+GRPOModel=Gemma-2-9B, Dataset=Russian 110k2025.07 | 93.89 | — | — | |
| SFTModel=Gemma-2-9B, Training Set=Chinese 210k2025.07 | 93.7 | — | — | |
| SFTModel=Qwen3-1.7B-Base, Training Set=Chinese 210k2025.07 | 93.22 | — | — | |
| SFT+GRPOModel=Qwen3-1.7B-Base, Training Set=Chinese 210k2025.07 | 93.12 | — | — | |
| SASFTModel=Llama-3.1-8B, Dataset=Russian 110k2025.07 | 92.98 | — | — | |
| SFT+PenaltyModel=Qwen3-1.7B-Base, Training Set=Chinese 210k2025.07 | 92.69 | — | — | |
| SASFTModel=Gemma-2-9B, Training Set=Chinese 210k2025.07 | 92.5 | — | — | |
| SFTModel=Llama-3.1-8B, Dataset=Russian 110k2025.07 | 92.4 | — | — | |
| SASFTModel=Qwen3-1.7B-Base, Training Set=Chinese 210k2025.07 | 92.12 | — | — | |
| SASFTModel=Llama-3.1-8B2025.07 | 91.88 | — | — | |
| SFT+PenaltyModel=Gemma-2-9B, Training Set=Chinese 210k2025.07 | 91.73 | — | — | |
| SFT+GRPOModel=Llama-3.1-8B, Dataset=Russian 110k2025.07 | 91.49 | — | — | |
| SFTModel=Llama-3.1-8B, Training Set=Chinese 210k2025.07 | 91.35 | — | — | |
| SFTModel=Gemma-2-2B, Training Set=Chinese 210k2025.07 | 91.25 | — | — | |
| SASFTModel=Qwen3-1.7B-Base, Dataset=Russian 110k2025.07 | 91.11 | — | — | |
| SFT+GRPOModel=Gemma-2-2B, Training Set=Chinese 210k2025.07 | 90.87 | — | — | |
| SASFTModel=Gemma-2-2B, Training Set=Chinese 210k2025.07 | 90.82 | — | — | |
| SFT+PenaltyModel=Gemma-2-2B, Training Set=Chinese 210k2025.07 | 90.77 | — | — | |
| SFT+GRPOModel=Qwen3-1.7B-Base, Dataset=Russian 110k2025.07 | 90.72 | — | — | |
| SFT+GRPOModel=Qwen3-1.7B-Base2025.07 | 90.48 | — | — | |
| SFTModel=Qwen3-1.7B-Base2025.07 | 90.29 | — | — | |
| SFT+PenaltyModel=Qwen3-1.7B-Base, Dataset=Russian 110k2025.07 | 90.05 | — | — | |
| SFTModel=Qwen3-1.7B-Base, Dataset=Russian 110k2025.07 | 90 | — | — | |
| SFT+GRPOModel=Llama-3.1-8B, Training Set=Chinese 210k2025.07 | 89.33 | — | — | |
| SFT+PenaltyModel=Llama-3.1-8B, Dataset=Russian 110k2025.07 | 89.23 | — | — | |
| SFT+PenaltyModel=Qwen3-1.7B-Base2025.07 | 89.13 | — | — | |
| SASFTModel=Qwen3-1.7B-Base2025.07 | 89.04 | — | — | |
| SASFTModel=Qwen3-8B-Base, Dataset=Russian 110k2025.07 | 88.89 | — | — | |
| SFT+PenaltyModel=Llama-3.1-8B, Training Set=Chinese 210k2025.07 | 88.51 | — | — | |
| SFTModel=Llama-3.1-8B2025.07 | 87.74 | — | — | |
| SFT+GRPOModel=Llama-3.1-8B2025.07 | 85.58 | — | — | |
| SFT+PenaltyModel=Llama-3.1-8B2025.07 | 85.43 | — | — | |
| SFT+PenaltyModel=Gemma-2-2B, Dataset=Russian 110k2025.07 | 85.1 | — | — | |
| SFT+GRPOModel=Gemma-2-2B, Dataset=Russian 110k2025.07 | 83.85 | — | — | |
| SFTModel=Gemma-2-2B, Dataset=Russian 110k2025.07 | 82.45 | — | — | |
| SASFTModel=Gemma-2-2B, Dataset=Russian 110k2025.07 | 80.96 | — | — | |
| SFT+PenaltyModel=Gemma-2-2B2025.07 | 80.62 | — | — | |
| SASFTModel=Gemma-2-2B2025.07 | 79.09 | — | — | |
| SFTModel=Gemma-2-2B2025.07 | 76.63 | — | — | |
| SFT+GRPOModel=Gemma-2-2B2025.07 | 76.35 | — | — | |
| d3LLMGeneration length=256, Block size=322025.12 | 37.2 | 33.41 | 6.86 | |
| dUltra-codingGeneration length=256, Block size=32, Planner=Enabled2025.12 | 35.85 | 24.29 | 7.23 | |
| Fast-dLLMGeneration length=256, Block size=322025.12 | 33.54 | 62.65 | 3.94 | |
| dParallelGeneration length=256, Block size=322025.12 | 31.71 | 30.68 | 5.87 | |
| dUltra-coding (w/o planner)Generation length=256, Block size=32, Planner=Disabled2025.12 | 23.78 | 16.18 | 10.19 |