Code Generation on Cangjie-bench
81.94AccuracyILA-agent
Evaluation results
| Method | Links | ||
|---|---|---|---|
| ILA-agentModels=Claude-Sonnet-4.52026.01 | 81.94 | 86.45 | |
| ILA-agentModels=Qwen3-Max2026.01 | 73.55 | 80.65 | |
| Iterative RAGModels=Claude-Sonnet-4.52026.01 | 65.81 | 70.32 | |
| ILA-agentModels=DeepSeek-V3.22026.01 | 63.23 | 70.97 | |
| Single-time RAGModels=Claude-Sonnet-4.52026.01 | 59.35 | 63.23 | |
| Iterative RAGModels=Qwen3-Max2026.01 | 52.9 | 57.42 | |
| Iterative RAGModels=DeepSeek-V3.22026.01 | 47.1 | 52.23 | |
| Single-time RAGModels=Qwen3-Max2026.01 | 46.45 | 50.97 | |
| Zero-shotModels=Claude-Sonnet-4.52026.01 | 45.16 | 49.68 | |
| Single-time RAGModels=DeepSeek-V3.22026.01 | 40 | 46.45 | |
| Cangjie GeneratorModels=Specialized2026.01 | 34.84 | 47.1 | |
| Zero-shotModels=Qwen3-Max2026.01 | 7.74 | 9.03 | |
| Zero-shotModels=DeepSeek-V3.22026.01 | 3.23 | 5.16 |