Incident Management on OPENRCA (test)
40Telecom Correct ScoreOpsAgent
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| OpsAgentSeed LLM=gpt-oss-20b, s/case=53.482025.10 | 40 | 55 | 5.56 | 11.11 | 6.78 | 16.95 | 11.28 | 20.3 | |
| OpsAgentSeed LLM=Qwen2.5-14B-Instruct-1M, s/case=71.182025.10 | 30 | 45 | 18.52 | 40.74 | 10.17 | 40.68 | 16.54 | 41.35 | |
| RCA-AgentSeed LLM=Claude 3.5 Sonnet, s/case=287.712025.10 | 20 | 35 | 16.67 | 35.19 | 3.39 | 28.81 | 11.28 | 32.33 | |
| CoTSeed LLM=Qwen2.5-14B-Instruct-1M, s/case=16.832025.10 | 10 | 20 | 0 | 1.85 | 5.08 | 13.55 | 1.5 | 9.77 | |
| OpsAgentSeed LLM=Phi-3-medium-128k-instruct, s/case=57.942025.10 | 10 | 35 | 0 | 9.26 | 3.39 | 10.17 | 3.01 | 13.53 | |
| ReActSeed LLM=Qwen2.5-14B-Instruct-1M, s/case=164.242025.10 | 5 | 20 | 0 | 1.85 | 0 | 5.08 | 0.75 | 6.01 | |
| ReflexionSeed LLM=Qwen2.5-14B-Instruct-1M, s/case=241.552025.10 | 0 | 10 | 3.7 | 11.11 | 1.69 | 5.08 | 2.26 | 8.27 | |
| CoTSeed LLM=gpt-oss-20b, s/case=27.422025.10 | 0 | 5 | 0 | 0 | 0 | 0 | 0 | 0.75 | |
| ReActSeed LLM=gpt-oss-20b, s/case=81.092025.10 | 0 | 0 | 0 | 0 | 1.69 | 1.69 | 0.75 | 0.75 | |
| ReflexionSeed LLM=gpt-oss-20b, s/case=165.742025.10 | 0 | 0 | 0 | 0 | 1.69 | 3.38 | 0.75 | 1.5 | |
| CoTSeed LLM=Phi-3-medium-128k-instruct, s/case=20.332025.10 | 0 | 10 | 0 | 0 | 0 | 0 | 0 | 1.5 | |
| ReActSeed LLM=Phi-3-medium-128k-instruct, s/case=48.422025.10 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
| ReflexionSeed LLM=Phi-3-medium-128k-instruct, s/case=100.052025.10 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
| ARTSeed LLM=Claude 3.5 Sonnet, s/case=2.272025.10 | 0 | 15 | 0 | 11.11 | 1.69 | 23.73 | 0.75 | 17.29 |