List your company
ResearchFormerly WizWand

Code Generation on HumanEval (Accuracy, NFE, T/F)

98.27AccSASFT
20.800440.912761.02581.1373Jul 20, 2025Aug 15, 2025Sep 10, 2025Oct 6, 2025Nov 1, 2025Nov 27, 2025Dec 24, 2025
Updated 8mo ago

Evaluation results

MethodLinks
SASFTModel=Qwen3-8B-Base2025.0798.27——
SFT+PenaltyModel=Gemma-2-9B2025.0797.02——
SASFTModel=Qwen3-8B-Base, Training Set=Chinese 210k2025.0796.92——
SFTModel=Qwen3-8B-Base, Training Set=Chinese 210k2025.0796.83——
SFT+PenaltyModel=Qwen3-8B-Base, Training Set=Chinese 210k2025.0796.83——
SASFTModel=Gemma-2-9B, Dataset=Russian 110k2025.0796.68——
SFT+GRPOModel=Qwen3-8B-Base2025.0796.44——
SFT+PenaltyModel=Gemma-2-9B, Dataset=Russian 110k2025.0796.44——
SFT+GRPOModel=Qwen3-8B-Base, Training Set=Chinese 210k2025.0796.3——
SFTModel=Qwen3-8B-Base2025.0795.87——
SFT+GRPOModel=Gemma-2-9B2025.0795.72——
SASFTModel=Gemma-2-9B2025.0795.67——
SFTModel=Gemma-2-9B2025.0795.62——
SASFTModel=Llama-3.1-8B, Training Set=Chinese 210k2025.0795.38——
SFT+PenaltyModel=Qwen3-8B-Base, Dataset=Russian 110k2025.0795——
SFT+PenaltyModel=Qwen3-8B-Base2025.0794.71——
SFT+GRPOModel=Qwen3-8B-Base, Dataset=Russian 110k2025.0794.33——
SFTModel=Gemma-2-9B, Dataset=Russian 110k2025.0794.23——
SFTModel=Qwen3-8B-Base, Dataset=Russian 110k2025.0794.13——
SFT+GRPOModel=Gemma-2-9B, Training Set=Chinese 210k2025.0794.09——
SFT+GRPOModel=Gemma-2-9B, Dataset=Russian 110k2025.0793.89——
SFTModel=Gemma-2-9B, Training Set=Chinese 210k2025.0793.7——
SFTModel=Qwen3-1.7B-Base, Training Set=Chinese 210k2025.0793.22——
SFT+GRPOModel=Qwen3-1.7B-Base, Training Set=Chinese 210k2025.0793.12——
SASFTModel=Llama-3.1-8B, Dataset=Russian 110k2025.0792.98——
SFT+PenaltyModel=Qwen3-1.7B-Base, Training Set=Chinese 210k2025.0792.69——
SASFTModel=Gemma-2-9B, Training Set=Chinese 210k2025.0792.5——
SFTModel=Llama-3.1-8B, Dataset=Russian 110k2025.0792.4——
SASFTModel=Qwen3-1.7B-Base, Training Set=Chinese 210k2025.0792.12——
SASFTModel=Llama-3.1-8B2025.0791.88——
SFT+PenaltyModel=Gemma-2-9B, Training Set=Chinese 210k2025.0791.73——
SFT+GRPOModel=Llama-3.1-8B, Dataset=Russian 110k2025.0791.49——
SFTModel=Llama-3.1-8B, Training Set=Chinese 210k2025.0791.35——
SFTModel=Gemma-2-2B, Training Set=Chinese 210k2025.0791.25——
SASFTModel=Qwen3-1.7B-Base, Dataset=Russian 110k2025.0791.11——
SFT+GRPOModel=Gemma-2-2B, Training Set=Chinese 210k2025.0790.87——
SASFTModel=Gemma-2-2B, Training Set=Chinese 210k2025.0790.82——
SFT+PenaltyModel=Gemma-2-2B, Training Set=Chinese 210k2025.0790.77——
SFT+GRPOModel=Qwen3-1.7B-Base, Dataset=Russian 110k2025.0790.72——
SFT+GRPOModel=Qwen3-1.7B-Base2025.0790.48——
SFTModel=Qwen3-1.7B-Base2025.0790.29——
SFT+PenaltyModel=Qwen3-1.7B-Base, Dataset=Russian 110k2025.0790.05——
SFTModel=Qwen3-1.7B-Base, Dataset=Russian 110k2025.0790——
SFT+GRPOModel=Llama-3.1-8B, Training Set=Chinese 210k2025.0789.33——
SFT+PenaltyModel=Llama-3.1-8B, Dataset=Russian 110k2025.0789.23——
SFT+PenaltyModel=Qwen3-1.7B-Base2025.0789.13——
SASFTModel=Qwen3-1.7B-Base2025.0789.04——
SASFTModel=Qwen3-8B-Base, Dataset=Russian 110k2025.0788.89——
SFT+PenaltyModel=Llama-3.1-8B, Training Set=Chinese 210k2025.0788.51——
SFTModel=Llama-3.1-8B2025.0787.74——
SFT+GRPOModel=Llama-3.1-8B2025.0785.58——
SFT+PenaltyModel=Llama-3.1-8B2025.0785.43——
SFT+PenaltyModel=Gemma-2-2B, Dataset=Russian 110k2025.0785.1——
SFT+GRPOModel=Gemma-2-2B, Dataset=Russian 110k2025.0783.85——
SFTModel=Gemma-2-2B, Dataset=Russian 110k2025.0782.45——
SASFTModel=Gemma-2-2B, Dataset=Russian 110k2025.0780.96——
SFT+PenaltyModel=Gemma-2-2B2025.0780.62——
SASFTModel=Gemma-2-2B2025.0779.09——
SFTModel=Gemma-2-2B2025.0776.63——
SFT+GRPOModel=Gemma-2-2B2025.0776.35——
d3LLMGeneration length=256, Block size=322025.1237.233.416.86
dUltra-codingGeneration length=256, Block size=32, Planner=Enabled2025.1235.8524.297.23
Fast-dLLMGeneration length=256, Block size=322025.1233.5462.653.94
dParallelGeneration length=256, Block size=322025.1231.7130.685.87
dUltra-coding (w/o planner)Generation length=256, Block size=32, Planner=Disabled2025.1223.7816.1810.19