# DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence

Source: [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](<https://www.sota2.com/research/paper/68fa71f21e209131cea53b4b>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T03:58:46.043Z

Authors: Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y.K. Li, Fuli Luo, Yingfei Xiong, Wenfeng Liang

Published: 2024-01-25T14:17:53.000Z

## Abstract

The rapid development of large language models has revolutionized code intelligence in software development. However, the predominance of closed-source models has restricted extensive research and development. To address this, we introduce the DeepSeek-Coder series, a range of open-source code models with sizes from 1.3B to 33B, trained from scratch on 2 trillion tokens. These models are pre-trained on a high-quality project-level code corpus and employ a fill-in-the-blank task with a 16K window to enhance code generation and infilling. Our extensive evaluations demonstrate that DeepSeek-Coder not only achieves state-of-the-art performance among open-source code models across multiple benchmarks but also surpasses existing closed-source models like Codex and GPT-3.5. Furthermore, DeepSeek-Coder models are under a permissive license that allows for both research and unrestricted commercial use.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Cross-file code completion on CrossCodeEval Python (test)](<https://www.sota2.com/research/sota/cross-file-code-completion-on-crosscodeeval-python-test.md>): Cross-file code completion; CrossCodeEval Python (test); Code ES: 78.78.

- [Cross-file code completion on CrossCodeEval TypeScript (test)](<https://www.sota2.com/research/sota/cross-file-code-completion-on-crosscodeeval-typescript-test.md>): Cross-file code completion; CrossCodeEval TypeScript (test); Code ES: 70.31.

- [Code Generation on ClassEval](<https://www.sota2.com/research/sota/code-generation-on-classeval.md>): Code Generation; ClassEval; Pass@1 (Class-level): 27.

- [Code Completion on DS-1000 (test)](<https://www.sota2.com/research/sota/code-completion-on-ds-1000-test.md>): Code Completion; DS-1000 (test); Matplotlib Success Rate: 56.1.

- [Program-aided math reasoning on TabMWP](<https://www.sota2.com/research/sota/program-aided-math-reasoning-on-tabmwp.md>): Program-aided math reasoning; TabMWP; Accuracy: 75.3.

- [Code Efficiency Evaluation on EvalPerf](<https://www.sota2.com/research/sota/code-efficiency-evaluation-on-evalperf.md>): Code Efficiency Evaluation; EvalPerf; DPS: 83.6.

- [Arithmetical puzzle solving on Arithmetical Puzzle 1.0 (OOD-F)](<https://www.sota2.com/research/sota/arithmetical-puzzle-solving-on-arithmetical-puzzle-1-0-ood-f.md>): Arithmetical puzzle solving; Arithmetical Puzzle 1.0 (OOD-F); Pass@1: 0.1.

- [Code Generation on APPS stdin-style Plus](<https://www.sota2.com/research/sota/code-generation-on-apps-stdin-style-plus.md>): Code Generation; APPS stdin-style Plus; Syntax Validity: 83.4.

- [Code Generation on HumanEval Multilingual (test)](<https://www.sota2.com/research/sota/code-generation-on-humaneval-multilingual-test.md>): Code Generation; HumanEval Multilingual (test); Average Score: 69.2.

- [Code Generation on BigCodeBench-Completion Full](<https://www.sota2.com/research/sota/code-generation-on-bigcodebench-completion-full.md>): Code Generation; BigCodeBench-Completion Full; pass@1: 59.7.

- [Diagram Generation on Code Agent](<https://www.sota2.com/research/sota/diagram-generation-on-code-agent.md>): Diagram Generation; Code Agent; Pass@1: 55.56.

- [Cross-file code completion on CrossCodeEval C# (test)](<https://www.sota2.com/research/sota/cross-file-code-completion-on-crosscodeeval-c-sharp-test.md>): Cross-file code completion; CrossCodeEval C# (test); Code ES: 75.04.

- [Data science code generation on DS-1000](<https://www.sota2.com/research/sota/data-science-code-generation-on-ds-1000.md>): Data science code generation; DS-1000; Matplotlib Score: 56.1.

- [Code Generation on ReCode Format](<https://www.sota2.com/research/sota/code-generation-on-recode-format.md>): Code Generation; ReCode Format; Pass@1: 62.37.

- [Code Generation on ReCode Syntax](<https://www.sota2.com/research/sota/code-generation-on-recode-syntax.md>): Code Generation; ReCode Syntax; Pass@1: 55.43.

- [Code Generation on ReCode Function](<https://www.sota2.com/research/sota/code-generation-on-recode-function.md>): Code Generation; ReCode Function; pass@1: 36.73.

- [Code Summarization on CodeXGLUE Code-Text](<https://www.sota2.com/research/sota/code-summarization-on-codexglue-code-text.md>): Code Summarization; CodeXGLUE Code-Text; BLEU-4: 11.8.

- [Instruction Following on HumanEvalFixDocs](<https://www.sota2.com/research/sota/instruction-following-on-humanevalfixdocs.md>): Instruction Following; HumanEvalFixDocs; pass@1: 48.21.

- [Multilingual Code Completion on Multipl-E (Pass@k)](<https://www.sota2.com/research/sota/multilingual-code-completion-on-multipl-e-pass-at-k.md>): Multilingual Code Completion; MultiPL-E; Pass@1: 30.47.

- [Code Bug Fixing on HumanEvalFix](<https://www.sota2.com/research/sota/code-bug-fixing-on-humanevalfix.md>): Code Bug Fixing; HumanEvalFix; Python Fix Rate: 47.5.

- [Arithmetical puzzle solving on Arithmetical Puzzle 1.0 (ID)](<https://www.sota2.com/research/sota/arithmetical-puzzle-solving-on-arithmetical-puzzle-1-0-id.md>): Arithmetical puzzle solving; Arithmetical Puzzle 1.0 (ID); Pass@1: 0.1.

- [Logical Reasoning on PrOntoQA (Acc, NLDD)](<https://www.sota2.com/research/sota/logical-reasoning-on-prontoqa-acc-nldd.md>): Logical reasoning; PrOntoQA; Accuracy: 100.

- [Mathematical Reasoning on GSM8K (Acc & NLDD)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-gsm8k-acc-and-nldd.md>): Mathematical Reasoning; GSM8K; Accuracy: 100.

- [Sequence Modeling on Dyck-n](<https://www.sota2.com/research/sota/sequence-modeling-on-dyck-n.md>): Sequence Modeling; Dyck-n; Acc: 47.2.

- [Code Completion on HumanEval+](<https://www.sota2.com/research/sota/code-completion-on-humaneval-plus.md>): Code Completion; HumanEval+; Pass@1: 46.3.

- [Agentic Task Completion on AppWorld LeaderBoard](<https://www.sota2.com/research/sota/agentic-task-completion-on-appworld-leaderboard.md>): Agentic Task Completion; AppWorld LeaderBoard; Greedy Success Rate: 44.6.

- [Cross-file code completion on CrossCodeEval Java (test)](<https://www.sota2.com/research/sota/cross-file-code-completion-on-crosscodeeval-java-test.md>): Cross-file code completion; CrossCodeEval Java (test); Code ES: 73.41.

- [Code Generation on Asleep at the Keyboard](<https://www.sota2.com/research/sota/code-generation-on-asleep-at-the-keyboard.md>): Code Generation; Asleep at the Keyboard; Valid Rate: 92.1.

- [Code Generation on HumanEval compile (L0)](<https://www.sota2.com/research/sota/code-generation-on-humaneval-compile-l0.md>): Code Generation; HumanEval compile (L0); Pass@1: 10.28.

- [Code Generation on HumanEval compile (L1)](<https://www.sota2.com/research/sota/code-generation-on-humaneval-compile-l1.md>): Code Generation; HumanEval compile (L1); Pass@1: 0.0689.

- [Code Generation on HumanEval compile (L2)](<https://www.sota2.com/research/sota/code-generation-on-humaneval-compile-l2.md>): Code Generation; HumanEval compile (L2); Pass@1: 7.94.

- [Code Generation on HumanEval compile (L3)](<https://www.sota2.com/research/sota/code-generation-on-humaneval-compile-l3.md>): Code Generation; HumanEval compile (L3); Pass@1: 6.17.

- [Fill-in-the-Middle Code Completion on FIM-Tasks Single-Line Infilling (test)](<https://www.sota2.com/research/sota/fill-in-the-middle-code-completion-on-fim-tasks-single-line-infilling-test.md>): Fill-in-the-Middle Code Completion; FIM-Tasks Single-Line Infilling (test); Python Score: 66.6.

- [Arithmetical puzzle solving on Arithmetical Puzzle 1.0 (OOD-N)](<https://www.sota2.com/research/sota/arithmetical-puzzle-solving-on-arithmetical-puzzle-1-0-ood-n.md>): Arithmetical puzzle solving; Arithmetical Puzzle 1.0 (OOD-N); Pass@1: 0.

- [Arithmetical puzzle solving on Arithmetical Puzzle OOD-1000 1.0](<https://www.sota2.com/research/sota/arithmetical-puzzle-solving-on-arithmetical-puzzle-ood-1000-1-0.md>): Arithmetical puzzle solving; Arithmetical Puzzle OOD-1000 1.0; pass@1: 0.

- [Code Generation on MBPP (Pass@1 (%))](<https://www.sota2.com/research/sota/code-generation-on-mbpp-pass-at-1-percent.md>): Code Generation; MBPP; Pass@1: 89.4.

- [Code Generation on DS-1000 1.0 (test)](<https://www.sota2.com/research/sota/code-generation-on-ds-1000-1-0-test.md>): Code Generation; DS-1000 1.0 (test); Matplotlib: 58.4.

- [Code Completion on BigCodeBench](<https://www.sota2.com/research/sota/code-completion-on-bigcodebench.md>): Code Completion; BigCodeBench; Full Score: 41.1.

- [Multi-turn Code Generation on MT-Bench Coding](<https://www.sota2.com/research/sota/multi-turn-code-generation-on-mt-bench-coding.md>): Multi-turn Code Generation; MT-Bench Coding; First Turn Score: 6.7.

- [Code Generation on EvoEval semantic-altering](<https://www.sota2.com/research/sota/code-generation-on-evoeval-semantic-altering.md>): Code Generation; EvoEval semantic-altering; Pass@1 Average: 41.4.

- [Code Generation on LiveCodeBench May 2023 to February 2024 (start date 2023-07-01)](<https://www.sota2.com/research/sota/code-generation-on-livecodebench-may-2023-to-february-2024-start-date-2023-07-01.md>): Code Generation; LiveCodeBench May 2023 to February 2024 (start date 2023-07-01); Pass@1: 20.8.

- [Code Generation on MultiPL-E (pass@1 Lua, Julia, R)](<https://www.sota2.com/research/sota/code-generation-on-multipl-e-pass-at-1-lua-julia-r.md>): Code Generation; MultiPL-E; Pass@1 (Lua): 40.

- [Text2Cypher generation on Crime](<https://www.sota2.com/research/sota/text2cypher-generation-on-crime.md>): Text2Cypher generation; Crime; Q@1 Accuracy: 46.

- [Text2Cypher generation on Fraud](<https://www.sota2.com/research/sota/text2cypher-generation-on-fraud.md>): Text2Cypher generation; Fraud; Q@1: 44.

- [Text2Cypher generation on Healthcare](<https://www.sota2.com/research/sota/text2cypher-generation-on-healthcare.md>): Text2Cypher generation; Healthcare; Q@1: 37.

- [Code Translation on Code Translation dataset](<https://www.sota2.com/research/sota/code-translation-on-code-translation-dataset.md>): Code Translation; Code Translation dataset; CB: 64.4.

- [Diagram Editing on DiagramGenBenchmark 1.0 (test)](<https://www.sota2.com/research/sota/diagram-editing-on-diagramgenbenchmark-1-0-test.md>): Diagram Editing; DiagramGenBenchmark 1.0 (test); Pass@1: 90.5.

- [Code Completion on MBPP (test)](<https://www.sota2.com/research/sota/code-completion-on-mbpp-test.md>): Code Completion; MBPP (test); 3-Shot Pass Rate: 60.6.

- [Code Infilling on SantaCoder FIM](<https://www.sota2.com/research/sota/code-infilling-on-santacoder-fim.md>): Code Infilling; SantaCoder FIM; EM: 76.3.

- [Code generation on LiveCodeBench start date 2023-09-01 May 2023 to February 2024](<https://www.sota2.com/research/sota/code-generation-on-livecodebench-start-date-2023-09-01-may-2023-to-february-2024.md>): Code Generation; LiveCodeBench start date 2023-09-01 May 2023 to February 2024; Pass@1: 19.2.

- [Code generation on LiveCodeBench May 2023 to February 2024 (start date 2023-05-01)](<https://www.sota2.com/research/sota/code-generation-on-livecodebench-may-2023-to-february-2024-start-date-2023-05-01.md>): Code Generation; LiveCodeBench May 2023 to February 2024 (start date 2023-05-01); Pass@1: 21.6.

- [Data Science Code Completion on DS-1000](<https://www.sota2.com/research/sota/data-science-code-completion-on-ds-1000.md>): Data Science Code Completion; DS-1000; Pandas (Pass@1): 21.3.

- [Code Generation on MobileKernelBench](<https://www.sota2.com/research/sota/code-generation-on-mobilekernelbench.md>): Code Generation; MobileKernelBench; CSR (%): 15.3.

- [Code Generation on BigCodeBench-Instruct (Full)](<https://www.sota2.com/research/sota/code-generation-on-bigcodebench-instruct-full.md>): Code Generation; BigCodeBench-Instruct (Full); Pass@1: 0.482.

- [Code Completion on MBPP+ (Pass@1)](<https://www.sota2.com/research/sota/code-completion-on-mbpp-plus-pass-at-1.md>): Code Completion; MBPP+; Pass@1: 59.1.

- [Code Generation on MBPP subset of 500 (test)](<https://www.sota2.com/research/sota/code-generation-on-mbpp-subset-of-500-test.md>): Code Generation; MBPP subset of 500 (test); Pass@1: 60.6.

- [Code Completion on HumanEval](<https://www.sota2.com/research/sota/code-completion-on-humaneval.md>): Code Completion; HumanEval; HE Score: 47.6.

- [Code Completion on MBPP Sanitized](<https://www.sota2.com/research/sota/code-completion-on-mbpp-sanitized.md>): Code Completion; MBPP Sanitized; MBPP Pass Rate: 70.2.

- [Code Generation on NaturalCodeBench](<https://www.sota2.com/research/sota/code-generation-on-naturalcodebench.md>): Code Generation; NaturalCodeBench; NCB (zh) Python Score: 44.3.

- [Code Generation on HumanEval HE, HE+](<https://www.sota2.com/research/sota/code-generation-on-humaneval-he-he-plus.md>): Code Generation; HumanEval HE, HE+; HE Pass Rate: 78.6.

- [Code Infilling on HumanEval single-line](<https://www.sota2.com/research/sota/code-infilling-on-humaneval-single-line.md>): Code Infilling; HumanEval single-line; Pass@1: 73.

- [Code Editing on CanItEdit original (test)](<https://www.sota2.com/research/sota/code-editing-on-canitedit-original-test.md>): Code Editing; CanItEdit original (test); Pass@1 (Average): 36.3.

- [Code Question Answering on CodeSimpleQA Chinese](<https://www.sota2.com/research/sota/code-question-answering-on-codesimpleqa-chinese.md>): Code Question Answering; CodeSimpleQA Chinese; ADS: 75.

- [Code Output Prediction on CRUXEval-O](<https://www.sota2.com/research/sota/code-output-prediction-on-cruxeval-o.md>): Code Output Prediction; CRUXEval-O; Pass@1: 48.6.

- [Code Generation on BigCodeBench-Instruct Hard](<https://www.sota2.com/research/sota/code-generation-on-bigcodebench-instruct-hard.md>): Code Generation; BigCodeBench-Instruct Hard; Pass@1: 24.3.

- [Code Generation on BigCodeBench-Completion Hard](<https://www.sota2.com/research/sota/code-generation-on-bigcodebench-completion-hard.md>): Code Generation; BigCodeBench-Completion Hard; pass@1: 29.7.

- [Code Generation on HumanEval-X](<https://www.sota2.com/research/sota/code-generation-on-humaneval-x.md>): Code Generation; HumanEval-X; Pass@1 (C++): 45.1.

- [Code Editing on Aider whole format](<https://www.sota2.com/research/sota/code-editing-on-aider-whole-format.md>): Code Editing; Aider whole format; Average Tries: 44.4.

- [Code Generation on MBPP MBPP+](<https://www.sota2.com/research/sota/code-generation-on-mbpp-mbpp-plus.md>): Code Generation; MBPP MBPP+; MBPP Score: 75.1.

- [Code Generation on BigCodeBench instruct](<https://www.sota2.com/research/sota/code-generation-on-bigcodebench-instruct.md>): Code Generation; BigCodeBench instruct; Full Score: 0.355.

- [Code Generation on MultiPL-E (Language Scores)](<https://www.sota2.com/research/sota/code-generation-on-multipl-e-language-scores.md>): Code Generation; MultiPL-E; Average Score: 66.1.

- [Execution-based Output Prediction on CRUXEval-O (200-sample, seed 42, test)](<https://www.sota2.com/research/sota/execution-based-output-prediction-on-cruxeval-o-200-sample-seed-42-test.md>): Execution-based output prediction; CRUXEval-O 200-sample (seed 42) (test); Accuracy: 45.5.

- [Code Editing on CanItEdit](<https://www.sota2.com/research/sota/code-editing-on-canitedit.md>): Code Editing; CanItEdit; Pass@1: 46.2.

- [Commit Message Generation on Commit Chronicle](<https://www.sota2.com/research/sota/commit-message-generation-on-commit-chronicle.md>): Commit Message Generation; Commit Chronicle; ROUGE-2 Score: 13.77.

- [Vulnerability Detection on Smart Contract Audit Dataset](<https://www.sota2.com/research/sota/vulnerability-detection-on-smart-contract-audit-dataset.md>): Vulnerability Detection; Smart Contract Audit Dataset; Precision: 96.3.

- [Code Generation on MBPP (MBPP, MBPP+)](<https://www.sota2.com/research/sota/code-generation-on-mbpp-mbpp-mbpp-plus.md>): Code Generation; MBPP; MBPP: 74.2.

- [Code-to-text on HumanEvalExplain (test)](<https://www.sota2.com/research/sota/code-to-text-on-humanevalexplain-test.md>): Code-to-text; HumanEvalExplain (test); Python Acc: 43.9.

- [URL Vulnerability Evaluation on MalURLBench 1.0 (test)](<https://www.sota2.com/research/sota/url-vulnerability-evaluation-on-malurlbench-1-0-test.md>): URL Vulnerability Evaluation; MalURLBench 1.0 (test); Success Rate (Commercial): 33.

- [Algorithmic Program Execution on FHP Algorithmic Task n=100 (test)](<https://www.sota2.com/research/sota/algorithmic-program-execution-on-fhp-algorithmic-task-n-100-test.md>): Algorithmic Program Execution; FHP Algorithmic Task n=100 (test); Accuracy: 50.5.

- [Code Generation on MBPP (Pass@1)](<https://www.sota2.com/research/sota/code-generation-on-mbpp-pass-at-1.md>): Code Generating; MBPP; Pass@1: 73.2.

- [Code Input Prediction on CRUXEval-I](<https://www.sota2.com/research/sota/code-input-prediction-on-cruxeval-i.md>): Code Input Prediction; CRUXEval-I; Pass@1: 46.5.

- [CUDA Kernel Generation on KernelBench Level 2](<https://www.sota2.com/research/sota/cuda-kernel-generation-on-kernelbench-level-2.md>): CUDA Kernel Generation; KernelBench Level 2; Execution Count: 7.

- [Algorithmic Program Execution on FP Algorithmic Task n=100 (test)](<https://www.sota2.com/research/sota/algorithmic-program-execution-on-fp-algorithmic-task-n-100-test.md>): Algorithmic Program Execution; FP Algorithmic Task n=100 (test); Accuracy: 52.2.

- [Severity Classification on Smart Contract Audit Dataset](<https://www.sota2.com/research/sota/severity-classification-on-smart-contract-audit-dataset.md>): Severity classification; Smart Contract Audit Dataset; Precision: 36.78.

- [RTL code generation on RTLLM v2.0 (test)](<https://www.sota2.com/research/sota/rtl-code-generation-on-rtllm-v2-0-test.md>): RTL code generation; RTLLM v2.0 (test); Syntax Pass@1: 59.3.

- [CUDA Kernel Generation on KernelBench Level 1](<https://www.sota2.com/research/sota/cuda-kernel-generation-on-kernelbench-level-1.md>): CUDA Kernel Generation; KernelBench Level 1; Exec Count: 13.

- [Cross-file code completion on CrossCodeEval (test)](<https://www.sota2.com/research/sota/cross-file-code-completion-on-crosscodeeval-test.md>): Cross-file code completion; CrossCodeEval (test); Python EM: 16.14.

- [Code Generation on Cangjie-bench](<https://www.sota2.com/research/sota/code-generation-on-cangjie-bench.md>): Code Generation; Cangjie-bench; Accuracy: 34.84.

- [Code Generation on LiveCodeBench pass@1 v5](<https://www.sota2.com/research/sota/code-generation-on-livecodebench-pass-at-1-v5.md>): Code Generation; LiveCodeBench pass@1 v5; pass@1: 14.5.

- [Aggregate Outcome Classification on CUE-Bench six tasks combined](<https://www.sota2.com/research/sota/aggregate-outcome-classification-on-cue-bench-six-tasks-combined.md>): Aggregate Outcome Classification; CUE-Bench six tasks combined; NB (%): 3.2.

- [Code Generation on Ag-LiveCodeBench-X](<https://www.sota2.com/research/sota/code-generation-on-ag-livecodebench-x.md>): Code Generation; Ag-LiveCodeBench-X; Lua Pass@1: 8.

- [Code Generation on CodeForces (pass@1)](<https://www.sota2.com/research/sota/code-generation-on-codeforces-pass-at-1.md>): Code Generation; CodeForces; pass@1: 2.56.

- [Text-to-SQL on BIRD (Accuracy)](<https://www.sota2.com/research/sota/text-to-sql-on-bird-accuracy.md>): Text-to-SQL; Bird; Accuracy: 62.5.

- [Binary Outcome Discrimination on Resolved vs. Rest (test)](<https://www.sota2.com/research/sota/binary-outcome-discrimination-on-resolved-vs-rest-test.md>): Binary Outcome Discrimination; Resolved vs. Rest (test); AUC: 84.1.

- [Code Reasoning on CUE-Bench](<https://www.sota2.com/research/sota/code-reasoning-on-cue-bench.md>): Code Reasoning; CUE-Bench; VT Score: 0.1.

- [Function-level Code Generation on MBPP+ augmented (test)](<https://www.sota2.com/research/sota/function-level-code-generation-on-mbpp-plus-augmented-test.md>): Function-level Code Generation; MBPP+ augmented (test); Pass@1: 70.1.

- [Code Generation on BigCodeBench (pass@1)](<https://www.sota2.com/research/sota/code-generation-on-bigcodebench-pass-at-1.md>): Code Generation; BigCodeBench; pass@1: 27.02.

- [RTL code understanding on RTL code understanding (test)](<https://www.sota2.com/research/sota/rtl-code-understanding-on-rtl-code-understanding-test.md>): RTL code understanding; RTL code understanding (test); BLEU-4: 2.56.

- [Code Generation on LeetCode Contest Benchmark](<https://www.sota2.com/research/sota/code-generation-on-leetcode-contest-benchmark.md>): Code Generation; LeetCode Contest Benchmark; Easy Accuracy: 57.8.

- [Code Repair on HumanEvalFix (test)](<https://www.sota2.com/research/sota/code-repair-on-humanevalfix-test.md>): Code Repair; HumanEvalFix (test); Success Rate (Python): 29.9.

- [Verilog Code Generation on VerilogEval Machine v1](<https://www.sota2.com/research/sota/verilog-code-generation-on-verilogeval-machine-v1.md>): Verilog Code Generation; VerilogEval Machine v1; Pass@1: 52.2.

- [Verilog Code Generation on RTLLM v1](<https://www.sota2.com/research/sota/verilog-code-generation-on-rtllm-v1.md>): Verilog Code Generation; RTLLM v1; Pass@1: 30.2.

- [Code Generation on HumanEval+ v1 (test)](<https://www.sota2.com/research/sota/code-generation-on-humaneval-plus-v1-test.md>): Code Generation; HumanEval+ v1 (test); Pass Rate: 70.1.

- [CUDA Kernel Generation on KernelBench Level 3](<https://www.sota2.com/research/sota/cuda-kernel-generation-on-kernelbench-level-3.md>): CUDA Kernel Generation; KernelBench Level 3; Executions Count: 0.

- [Code Generation on HumanEval+ augmented (test)](<https://www.sota2.com/research/sota/code-generation-on-humaneval-plus-augmented-test.md>): Function-level Code Generation; HumanEval+ augmented (test); Pass@1: 75.

- [Code Generation on FullStackBench](<https://www.sota2.com/research/sota/code-generation-on-fullstackbench.md>): Code Generation; FullStackBench; Pass@1: 56.37.

- [Verilog Code Generation on RTLLM v2](<https://www.sota2.com/research/sota/verilog-code-generation-on-rtllm-v2.md>): Verilog Code Generation; RTLLM v2; Pass@1: 41.8.

- [Reasoning on Big-Bench Hard (test)](<https://www.sota2.com/research/sota/reasoning-on-big-bench-hard-test.md>): Reasoning; BBH (test); Accuracy: 55.2.

- [Factual code question answering on CodeSimpleQA English 1.0](<https://www.sota2.com/research/sota/factual-code-question-answering-on-codesimpleqa-english-1-0.md>): Factual code question answering; CodeSimpleQA English 1.0; ADS: 40.

- [Text-to-SQL on Spider Lite 2.0](<https://www.sota2.com/research/sota/text-to-sql-on-spider-lite-2-0.md>): Text-to-SQL; Spider Lite 2.0; Execution Accuracy (EX): 15.6.

- [Verilog Code Generation on VerilogEval SR v2](<https://www.sota2.com/research/sota/verilog-code-generation-on-verilogeval-sr-v2.md>): Verilog Code Generation; VerilogEval SR v2; Pass@1: 36.8.

- [Code Generation on MBPP+ (pass@1)](<https://www.sota2.com/research/sota/code-generation-on-mbpp-plus-pass-at-1.md>): Code Generation; MBPP+; Pass@1: 75.1.

- [Code Generation on MBPP Plus (test)](<https://www.sota2.com/research/sota/code-generation-on-mbpp-plus-test.md>): Code Generation; MBPP Plus (test); Accuracy: 65.9.

- [Code Generation on MBPP (pass@1 accuracy)](<https://www.sota2.com/research/sota/code-generation-on-mbpp-pass-at-1-accuracy.md>): Code Generation; MBPP; Pass@1 Accuracy: 65.

- [Verilog Code Generation on VerilogEval CC v2](<https://www.sota2.com/research/sota/verilog-code-generation-on-verilogeval-cc-v2.md>): Verilog Code Generation; VerilogEval CC v2; Pass@1: 24.4.

- [Code Generation on LiveCodeBench (test)](<https://www.sota2.com/research/sota/code-generation-on-livecodebench-test.md>): Code Generation; LiveCodeBench (test); Pass@1 Overall: 18.1.

- [Code Reasoning on CRUXEval (CoT Metrics)](<https://www.sota2.com/research/sota/code-reasoning-on-cruxeval-cot-metrics.md>): Code Reasoning; CRUXEval; Input-CoT Accuracy: 50.6.

- [Unlearning on Syntax-preserving dataset (forget set)](<https://www.sota2.com/research/sota/unlearning-on-syntax-preserving-dataset-forget-set.md>): Unlearning; Syntax-preserving dataset (forget set); PrivLeak: 0.97.

- [Code Generation on CodeContests (test)](<https://www.sota2.com/research/sota/code-generation-on-codecontests-test.md>): Code Generation; CodeContests (test); Pass@1: 21.3.

- [Code Generation on LiveCodeBench (Avg)](<https://www.sota2.com/research/sota/code-generation-on-livecodebench-avg.md>): Code Generation; LiveCodeBench; Average Score: 20.5.

- [Code Generation on LiveCodeBench (pass@1, pass@3, pass@5)](<https://www.sota2.com/research/sota/code-generation-on-livecodebench-pass-at-1-pass-at-3-pass-at-5.md>): Code Generation; LiveCodeBench; Pass@1: 8.24.

- [Verilog Code Generation on VerilogEval v1 (Human)](<https://www.sota2.com/research/sota/verilog-code-generation-on-verilogeval-v1-human.md>): Verilog Code Generation; VerilogEval v1 (Human); Pass@1: 30.2.

- [Code Generation on MBPP (pass@1, pass@80)](<https://www.sota2.com/research/sota/code-generation-on-mbpp-pass-at-1-pass-at-80.md>): Code Generation; MBPP; Pass@1: 60.6.

- [Code Generation on LiveCodeBench](<https://www.sota2.com/research/sota/code-generation-on-livecodebench.md>): Code Generation; LiveCodeBench; Pass@1: 27.9.

- [Code Generation on HumanEval+](<https://www.sota2.com/research/sota/code-generation-on-humaneval-plus.md>): Code Generation; HumanEval+; Pass@1: 82.3.

- [Code Generation on MBPP (test)](<https://www.sota2.com/research/sota/code-generation-on-mbpp-test.md>): Code Generation; MBPP (test); Pass@1: 71.9.

- [Mathematical Reasoning on GSM-Hard](<https://www.sota2.com/research/sota/mathematical-reasoning-on-gsm-hard.md>): Mathematical Reasoning; GSM-Hard; Solve Rate: 54.1.

- [Language Understanding on MMLU (test)](<https://www.sota2.com/research/sota/language-understanding-on-mmlu-test.md>): Language Understanding; MMLU (test); MMLU Average Accuracy: 49.1.

- [Mathematical Reasoning on MAWPS](<https://www.sota2.com/research/sota/mathematical-reasoning-on-mawps.md>): Mathematical Reasoning; MAWPS; Accuracy: 93.3.

- [Code Generation on HumanEval 1.0 (test)](<https://www.sota2.com/research/sota/code-generation-on-humaneval-1-0-test.md>): Code Generation; HumanEval 1.0 (test); Pass@1: 0.774.

- [Code Generation on HumanEval (test)](<https://www.sota2.com/research/sota/code-generation-on-humaneval-test.md>): Code Generation; HumanEval (test); Pass@1: 79.3.

- [Mathematical Reasoning on ASDiv (Accuracy)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-asdiv-accuracy.md>): Mathematical Reasoning; ASDIV; Accuracy: 0.767.

- [Text-to-SQL on Spider (test)](<https://www.sota2.com/research/sota/text-to-sql-on-spider-test.md>): Text-to-SQL; Spider (test); Execution Accuracy: 58.4.

- [Code Generation on HumanEval](<https://www.sota2.com/research/sota/code-generation-on-humaneval.md>): Code Generation; HumanEval; Pass@1: 85.4.

- [Mathematical Reasoning on SVAMP](<https://www.sota2.com/research/sota/mathematical-reasoning-on-svamp.md>): Mathematical Reasoning; SVAMP; Accuracy: 71.6.

- [Multitask Language Understanding on MMLU (test)](<https://www.sota2.com/research/sota/multitask-language-understanding-on-mmlu-test.md>): Multitask Language Understanding; MMLU (test); Accuracy: 34.

- [Mathematical Reasoning on GSM8K (test)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-gsm8k-test.md>): Mathematical Reasoning; GSM8K (test); Accuracy: 58.7.

- [Text-to-SQL on BIRD (dev)](<https://www.sota2.com/research/sota/text-to-sql-on-bird-dev.md>): Text-to-SQL; BIRD (dev); Execution Accuracy (EA): 22.5.

- [Mathematical Reasoning on GSM8K](<https://www.sota2.com/research/sota/mathematical-reasoning-on-gsm8k.md>): Mathematical Reasoning; GSM8K; Accuracy: 60.7.

- [Mathematical Reasoning on MATH](<https://www.sota2.com/research/sota/mathematical-reasoning-on-math.md>): Mathematical Reasoning; MATH; Accuracy: not recorded.

- [Text-to-SQL on Spider (dev)](<https://www.sota2.com/research/sota/text-to-sql-on-spider-dev.md>): Text-to-SQL; Spider (dev); EX: not recorded.

- [Code Infilling on HumanEval multi-line code infilling](<https://www.sota2.com/research/sota/code-infilling-on-humaneval-multi-line-code-infilling.md>): Code Infilling; HumanEval multi-line code infilling; Pass Rate: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2401.14196>)

[PDF](<https://arxiv.org/pdf/2401.14196.pdf>)

[Implementation](<https://github.com/deepseek-ai/DeepSeek-Coder>)
