LLM-Judge Evaluation on Humanities Rubric on Shared Human-Rated Question Set n=100 (test)
4.88Answer AccuracySPIRE
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| SPIREGenerator Model=DEEPSEEK-V4-FLASH, Rater (Judge Model)=Qwen3.52026.05 | 4.88 | 4.86 | 4.87 | 4.79 | 4.85 | |
| SPIREGenerator Model=GEMINI-3-FLASH, Rater (Judge Model)=Qwen3.52026.05 | 4.59 | 4.56 | 4.6 | 4.55 | 4.58 | |
| SPIREGenerator Model=DEEPSEEK-V4-FLASH, Rater (Judge Model)=GPT-5.42026.05 | 4.49 | 4.44 | 4.75 | 4.33 | 4.5 | |
| Text RAGGenerator Model=DEEPSEEK-V4-FLASH, Rater (Judge Model)=Qwen3.52026.05 | 4.29 | 3.45 | 4.22 | 3.2 | 3.79 | |
| SPIREGenerator Model=GEMINI-3-FLASH, Rater (Judge Model)=GPT-5.42026.05 | 4.25 | 4.03 | 4.34 | 4.2 | 4.21 | |
| Text RAGGenerator Model=DEEPSEEK-V4-FLASH, Rater (Judge Model)=GPT-5.42026.05 | 4.08 | 3.69 | 4.21 | 3.51 | 3.87 | |
| Naive LLMGenerator Model=GEMINI-3-FLASH, Rater (Judge Model)=GPT-5.42026.05 | 3.9 | 3.13 | 3.96 | 2.31 | 3.33 | |
| Naive LLMGenerator Model=DEEPSEEK-V4-FLASH, Rater (Judge Model)=Qwen3.52026.05 | 3.87 | 2.1 | 3.01 | 1.9 | 2.72 | |
| GraphRAGGenerator Model=DEEPSEEK-V4-FLASH, Rater (Judge Model)=Qwen3.52026.05 | 3.77 | 2.92 | 3.71 | 2.32 | 3.18 | |
| Naive LLMGenerator Model=DEEPSEEK-V4-FLASH, Rater (Judge Model)=GPT-5.42026.05 | 3.58 | 2.5 | 3.29 | 2.1 | 2.87 | |
| GraphRAGGenerator Model=DEEPSEEK-V4-FLASH, Rater (Judge Model)=GPT-5.42026.05 | 3.43 | 3.07 | 3.76 | 2.68 | 3.23 | |
| Text RAGGenerator Model=GEMINI-3-FLASH, Rater (Judge Model)=Qwen3.52026.05 | 3.37 | 3.15 | 3.25 | 3.17 | 3.24 | |
| Naive LLMGenerator Model=GEMINI-3-FLASH, Rater (Judge Model)=Qwen3.52026.05 | 3.35 | 3.32 | 3.15 | 2.5 | 3.08 | |
| Text RAGGenerator Model=GEMINI-3-FLASH, Rater (Judge Model)=GPT-5.42026.05 | 3.17 | 3.09 | 3.12 | 2.68 | 3.02 | |
| GraphRAGGenerator Model=GEMINI-3-FLASH, Rater (Judge Model)=Qwen3.52026.05 | 2.67 | 2.04 | 2.64 | 2.24 | 2.4 | |
| GraphRAGGenerator Model=GEMINI-3-FLASH, Rater (Judge Model)=GPT-5.42026.05 | 2.6 | 2.24 | 2.86 | 1.42 | 2.28 |