Quantitative evaluation of LLM feedback against human gold standards on 50 SOC analysis reports (test)
0.66Spearman Correlation (ρ)MESSALA
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| MESSALAModel=gpt-4.1 mini, Model Category=Closed-models2026.01 | 0.66 | 0.53 | 0.66 | 0.89 | |
| Method 1Model=gpt-4.1 mini, Model Category=Closed-models2026.01 | 0.65 | 0.54 | 0.64 | 1.24 | |
| Method 3Model=gpt-4.1 mini, Model Category=Closed-models2026.01 | 0.65 | 0.51 | 0.63 | 0.91 | |
| MESSALAModel=gpt-4.1, Model Category=Closed-models2026.01 | 0.64 | 0.53 | 0.64 | 0.88 | |
| Method 1Model=gpt-4.1, Model Category=Closed-models2026.01 | 0.63 | 0.53 | 0.61 | 1.27 | |
| MESSALAModel=gpt-4o, Model Category=Closed-models2026.01 | 0.6 | 0.49 | 0.59 | 1.03 | |
| Method 2Model=gpt-4.1, Model Category=Closed-models2026.01 | 0.6 | 0.5 | 0.57 | 1.07 | |
| Method 4Model=gpt-4.1, Model Category=Closed-models2026.01 | 0.6 | 0.5 | 0.6 | 0.97 | |
| MESSALAModel=gpt-oss, Parameter count=20B, Model Category=Open-models2026.01 | 0.59 | 0.46 | 0.59 | 1.13 | |
| Method 4Model=gpt-4.1 mini, Model Category=Closed-models2026.01 | 0.58 | 0.45 | 0.58 | 1.04 | |
| MESSALAModel=qwen3, Parameter count=14B, Model Category=Open-models2026.01 | 0.58 | 0.46 | 0.56 | 1.02 | |
| Method 4Model=qwen3, Parameter count=14B, Model Category=Open-models2026.01 | 0.57 | 0.44 | 0.56 | 0.93 | |
| Method 1Model=gpt-oss, Parameter count=20B, Model Category=Open-models2026.01 | 0.56 | 0.44 | 0.55 | 1.27 | |
| Method 3Model=gpt-oss, Parameter count=20B, Model Category=Open-models2026.01 | 0.56 | 0.43 | 0.54 | 1.16 | |
| Method 1Model=qwen3, Parameter count=14B, Model Category=Open-models2026.01 | 0.56 | 0.44 | 0.55 | 1.01 | |
| Method 4Model=gpt-4o, Model Category=Closed-models2026.01 | 0.55 | 0.45 | 0.53 | 1.01 | |
| Method 1Model=gpt-4o, Model Category=Closed-models2026.01 | 0.54 | 0.46 | 0.54 | 1.16 | |
| Method 2Model=gpt-4.1 mini, Model Category=Closed-models2026.01 | 0.54 | 0.42 | 0.55 | 1.23 | |
| Method 2Model=qwen3, Parameter count=14B, Model Category=Open-models2026.01 | 0.52 | 0.4 | 0.51 | 0.97 | |
| Method 2Model=gpt-4o, Model Category=Closed-models2026.01 | 0.51 | 0.42 | 0.51 | 1.05 | |
| Method 3Model=qwen3, Parameter count=14B, Model Category=Open-models2026.01 | 0.5 | 0.39 | 0.49 | 1 | |
| Method 3Model=gpt-4o, Model Category=Closed-models2026.01 | 0.49 | 0.4 | 0.49 | 1.27 | |
| Method 3Model=gpt-4.1, Model Category=Closed-models2026.01 | 0.49 | 0.4 | 0.49 | 1.3 | |
| Method 4Model=gpt-oss, Parameter count=20B, Model Category=Open-models2026.01 | 0.46 | 0.36 | 0.44 | 1.34 | |
| Method 2Model=gpt-oss, Parameter count=20B, Model Category=Open-models2026.01 | 0.41 | 0.33 | 0.39 | 1.56 | |
| MESSALAModel=gemma3, Parameter count=12B, Model Category=Open-models2026.01 | 0.38 | 0.3 | 0.35 | 1.06 | |
| Method 2Model=gemma3, Parameter count=12B, Model Category=Open-models2026.01 | 0.36 | 0.3 | 0.36 | 1.12 | |
| Method 4Model=gemma3, Parameter count=12B, Model Category=Open-models2026.01 | 0.3 | 0.23 | 0.28 | 1.17 | |
| Method 1Model=gemma3, Parameter count=12B, Model Category=Open-models2026.01 | 0.28 | 0.23 | 0.28 | 1.52 | |
| Method 3Model=gemma3, Parameter count=12B, Model Category=Open-models2026.01 | 0.22 | 0.17 | 0.23 | 1.34 |