Code Generation on HumanEval (Accuracy only)
90AccuracyGemini 2.0 Flash Lite
Evaluation results
| Method | Links | |
|---|---|---|
| Gemini 2.0 Flash LiteSystem Optimization=true, Throughput (TPS)=201, # Shots=02025.12 | 90 | |
| Mercury Coder SmallSystem Optimization=true, Throughput (TPS)=737, # Shots=02025.12 | 90 | |
| Gemini DiffusionSystem Optimization=true, Throughput (TPS)=1479, # Shots=02025.12 | 89.6 | |
| GPT 4o MiniSystem Optimization=true, Throughput (TPS)=59, # Shots=02025.12 | 88 | |
| Mercury Coder MiniSystem Optimization=true, Throughput (TPS)=1109, # Shots=02025.12 | 88 | |
| Seed DiffusionSystem Optimization=true, Throughput (TPS)=1600, # Shots=02025.12 | 82.8 | |
| Nova MicroSystem Optimization=true, Throughput (TPS)=148, # Shots=02025.12 | 79.3 | |
| REFUSION# Total Params=8B, # Activated Params=8B, System Optimization=false, Throughput (TPS)=98, # Shots=02025.12 | 78.66 | |
| LLaDA-MoE# Total Params=7B, # Activated Params=1B, System Optimization=true, Throughput (TPS)=884, # Shots=02025.12 | 62.2 |