Technical engineering calculation on TPS-CalcBench (test)
91.4Delta KPIdeepseek-v3.1
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| deepseek-v3.1strategy=FT + RAG-EQ + PA-CoT, fine-tuning-dataset=DFA-TPS2026.04 | 91.4 | — | — | — | — | — | — | — | — | |
| aws.claude-sonnet-4.5strategy=FT + RAG-EQ + PA-CoT, fine-tuning-dataset=DFA-TPS2026.04 | 88.7 | — | — | — | — | — | — | — | — | |
| deepseek-v3.1strategy=RAG-EQ + PA-CoT2026.04 | 87.2 | — | — | — | — | — | — | — | — | |
| deepseek-v3.1strategy=FT only, fine-tuning-dataset=DFA-TPS2026.04 | 86.3 | — | — | — | — | — | — | — | — | |
| aws.claude-sonnet-4.5strategy=RAG-EQ + PA-CoT2026.04 | 83.4 | — | — | — | — | — | — | — | — | |
| aws.claude-sonnet-4.5strategy=FT only, fine-tuning-dataset=DFA-TPS2026.04 | 81.5 | — | — | — | — | — | — | — | — | |
| deepseek-v3.1strategy=Zero-shot2026.04 | 79.22 | — | — | — | — | — | — | — | — | |
| gemini-3-flash-previewstrategy=FT + RAG-EQ + PA-CoT, fine-tuning-dataset=DFA-TPS2026.04 | 72.3 | — | — | — | — | — | — | — | — | |
| aws.claude-sonnet-4.5strategy=Zero-shot2026.04 | 71.75 | — | — | — | — | — | — | — | — | |
| gemini-3-flash-previewstrategy=RAG-EQ + PA-CoT2026.04 | 62.1 | — | — | — | — | — | — | — | — | |
| gemini-3-flash-previewstrategy=FT only, fine-tuning-dataset=DFA-TPS2026.04 | 54.8 | — | — | — | — | — | — | — | — | |
| gemini-3-flash-previewstrategy=Zero-shot2026.04 | 42 | — | — | — | — | — | — | — | — | |
| PA-CoTBackbone Model=gemini-3-flash-preview, Strategy=PA-CoT2026.04 | 10.8 | 1.14 | 1.45 | 1.52 | 1.25 | 1.48 | 1.22 | 1.41 | 1.54 | |
| PA-CoTBackbone Model=aws.claude-sonnet-4.5, Strategy=PA-CoT2026.04 | 7.2 | 1.64 | 1.8 | 1.78 | 1.58 | 1.74 | 1.61 | 1.72 | 1.81 | |
| PA-CoTBackbone Model=deepseek-v3.1, Strategy=PA-CoT2026.04 | 6.1 | 1.73 | 1.83 | 1.8 | 1.68 | 1.79 | 1.68 | 1.78 | 1.83 | |
| BaseBackbone Model=aws.claude-sonnet-4.5, Strategy=Base2026.04 | — | 1.62 | 1.78 | 1.71 | 1.55 | 1.41 | 0.92 | 1.33 | 1.74 | |
| BaseBackbone Model=deepseek-v3.1, Strategy=Base2026.04 | — | 1.71 | 1.82 | 1.74 | 1.66 | 1.48 | 1.03 | 1.41 | 1.77 | |
| BaseBackbone Model=gemini-3-flash-preview, Strategy=Base2026.04 | — | 1.1 | 1.42 | 1.38 | 1.21 | 1.05 | 0.61 | 0.88 | 1.35 |