Citation-augmented Question Answering on bar-GT, PK 1.0 (test)
78.59AccuracyPOSTCITE
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| POSTCITEModel=GPT-4o2025.04 | 78.59 | 39.83 | 77.26 | 53.72 | 3.53 | 2.94 | 0.15 | |
| GUIDED-RAELModel=DeepSeek-R12025.04 | 76.64 | 43.05 | 57.14 | 41.89 | 3.26 | 4 | 0.18 | |
| FOOTNOTEModel=DeepSeek-R12025.04 | 76.02 | 41.66 | 40.78 | 40.96 | 3.35 | 3.82 | 0.18 | |
| INTRALIGNModel=Llama-3.1-70B-Instruct2025.04 | 75.64 | 62.01 | 89.82 | 75.71 | 3.67 | 4.42 | 0.09 | |
| POSTCITEModel=GPT-4o2025.04 | 75 | 24.44 | 80.67 | 52.03 | 3.78 | 2.67 | 0.18 | |
| GUIDED-RAELModel=DeepSeek-R12025.04 | 73.29 | 36.89 | 50.32 | 42.31 | 3.2 | 3.94 | 0.13 | |
| FOOTNOTEModel=GPT-4o2025.04 | 72.89 | 40.06 | 53.78 | 49.51 | 3.42 | 4 | 0.15 | |
| Guided-RAELModel=GPT-4o2025.04 | 72.27 | 43.97 | 69.61 | 54.93 | 3.64 | 4.04 | 0.22 | |
| FOOTNOTEModel=DeepSeek-R12025.04 | 71.66 | 36.41 | 50.82 | 40.4 | 3.3 | 3.86 | 0.18 | |
| INTRALIGNModel=Llama-3.1-70B-Instruct2025.04 | 71.25 | 54.42 | 82.07 | 60.32 | 3.6 | 4.47 | 0.13 | |
| Guided-RAELModel=GPT-4o2025.04 | 69.51 | 41 | 83.33 | 42.94 | 3.47 | 4.09 | 0.18 | |
| INTRALIGNModel=Llama-3.1-8B-Instruct2025.04 | 69.05 | 44.53 | 87.95 | 51.84 | 3.64 | 3.78 | 0.11 | |
| FOOTNOTEModel=Llama-3.1-70B-Instruct2025.04 | 67.74 | 68.18 | 33.33 | 30.92 | 3.71 | 4.4 | 0.18 | |
| FOOTNOTEModel=GPT-4o2025.04 | 66.03 | 47.88 | 52.87 | 32.38 | 3.55 | 3.85 | 0.2 | |
| FRONTModel=Llama-3.1-70B-Instruct, Knowledge Restriction=External knowledge only2025.04 | 65.24 | 72.1 | — | 53.78 | 3.48 | 2.32 | — | |
| FOOTNOTEModel=o1-mini2025.04 | 64.61 | 47.61 | 43.06 | 45.51 | 3.6 | 4.05 | 0.15 | |
| FRONTModel=Llama-3.1-8B-Instruct, Knowledge Restriction=External knowledge only2025.04 | 64.29 | 68.62 | — | 56.25 | 1.8 | 1.86 | — | |
| Guided-RAELModel=Llama-3.1-70B-Instruct2025.04 | 63.4 | 40.44 | 50 | 40.61 | 3.54 | 3.18 | 0.1 | |
| POSTCITEModel=Llama-3.1-8B-Instruct2025.04 | 63.33 | 22.03 | 35.93 | 24.24 | 3.6 | 2.38 | 0.13 | |
| GUIDED-RAELModel=o1-mini2025.04 | 63.23 | 69.23 | 51.02 | 45.35 | 3.09 | 4.07 | 0.13 | |
| CONTEXTCITEModel=Llama-3.1-70B-Instruct, Knowledge Restriction=External knowledge only2025.04 | 63.07 | 38.89 | — | 38.86 | 3.32 | 3.29 | — | |
| INTRALIGNModel=Llama-3.1-8B-Instruct2025.04 | 62.24 | 59.11 | 72.65 | 59.8 | 3.65 | 3.83 | 0.11 | |
| RECITEModel=Llama-3.1-8B-Instruct, Knowledge Restriction=Parameter knowledge only2025.04 | 61.49 | — | 66.13 | 59.81 | 3.32 | 3.32 | 0.3 | |
| FOOTNOTEModel=o1-mini2025.04 | 61.23 | 39.82 | 42.5 | 40.83 | 3.01 | 3.81 | 0.13 | |
| POSTCITEModel=Llama-3.1-70B-Instruct2025.04 | 58.13 | 21.93 | 68.9 | 42.56 | 3.86 | 1.37 | 0.15 | |
| Guided-RAELModel=Llama-3.1-70B-Instruct2025.04 | 57.32 | 46.15 | 81.94 | 51.64 | 3.33 | 3.18 | 0.12 | |
| CONTEXTCITEModel=Llama-3.1-8B-Instruct, Knowledge Restriction=External knowledge only2025.04 | 56.67 | 35.56 | — | 35.55 | 3.51 | 3.01 | — | |
| FRONTModel=Llama-3.1-70B-Instruct, Knowledge Restriction=External knowledge only2025.04 | 56.06 | 47.46 | — | 42.28 | 3.32 | 2.24 | — | |
| FOOTNOTEModel=Llama-3.1-70B-Instruct2025.04 | 54.66 | 41.98 | 26.23 | 20.62 | 3.53 | 4.1 | 0.18 | |
| POSTCITEModel=Llama-3.1-70B-Instruct2025.04 | 53.88 | 22.63 | 74.21 | 45.13 | 3.66 | 1.74 | 0.19 | |
| CONTEXTCITEModel=Llama-3.1-70B-Instruct, Knowledge Restriction=External knowledge only2025.04 | 52.91 | 15.81 | — | 15.81 | 3.55 | 3.86 | — | |
| GUIDED-RAELModel=o1-mini2025.04 | 51.58 | 57 | 40.23 | 38.53 | 3.53 | 4.51 | 0.15 | |
| FOOTNOTEModel=Llama-3.1-8B-Instruct2025.04 | 50.95 | 56.52 | 45.9 | 14.16 | 3.55 | 3.47 | 0.17 | |
| Guided-RAELModel=Llama-3.1-8B-Instruct2025.04 | 49.3 | 42.77 | 72.34 | 47.54 | 3.05 | 3.59 | 0.13 | |
| FRONTModel=Llama-3.1-8B-Instruct, Knowledge Restriction=External knowledge only2025.04 | 48.6 | 54.61 | — | 31.6 | 3.3 | 1.86 | — | |
| POSTCITEModel=Llama-3.1-8B-Instruct2025.04 | 48.6 | 24.7 | 33.84 | 28.59 | 3.62 | 2.51 | 0.2 | |
| Guided-RAELModel=Llama-3.1-8B-Instruct2025.04 | 48.1 | 51.61 | 43.75 | 43.56 | 3.48 | 3.8 | 0.13 | |
| RECITEModel=Llama-3.1-70B-Instruct, Knowledge Restriction=Parameter knowledge only2025.04 | 44.43 | — | 82.12 | 72.86 | 3.44 | 3.72 | 0.22 | |
| CONTEXTCITEModel=Llama-3.1-8B-Instruct, Knowledge Restriction=External knowledge only2025.04 | 44.41 | 12.46 | — | 12.46 | 3.37 | 3.81 | — | |
| FOOTNOTEModel=Llama-3.1-8B-Instruct2025.04 | 35.44 | 35.46 | 31.15 | 18.87 | 3.59 | 3.29 | 0.17 | |
| RECITEModel=Llama-3.1-70B-Instruct, Knowledge Restriction=Parameter knowledge only2025.04 | 1.95 | — | 80.18 | 78.73 | 3.82 | 3.09 | 0.23 | |
| RECITEModel=Llama-3.1-8B-Instruct, Knowledge Restriction=Parameter knowledge only2025.04 | 0.95 | — | 56.51 | 48.49 | 3.05 | 2.04 | 0.23 |