Long-form Question Answering with Citations on ASQA
45.01EMCaLM
Evaluation results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| CaLMMain LM=GPT-3.5-Turbo-1106, Retriever=GTR, Method Type=Iterative, # Main LM Call=≤ 4, # Verifier LM Call=≤ 32024.06 | 45.01 | — | — | — | — | 83.98 | 72.59 | 78.03 | 68.98 | — | — | |
| CaLMMain LM=text-unicorn, Retriever=GTR, Method Type=Iterative, # Main LM Call=≤ 4, # Verifier LM Call=≤ 32024.06 | 44.21 | — | — | — | — | 82.08 | 70.55 | 72.37 | 67.3 | — | — | |
| CaLMMain LM=GPT-3.5-Turbo-1106, Retriever=DPR, Method Type=Iterative, # Main LM Call=≤ 4, # Verifier LM Call=≤ 32024.06 | 43.56 | — | — | — | — | 81.35 | 66 | 69.95 | 64.71 | — | — | |
| ICLCite + USCMain LM=GPT-3.5-Turbo-1106, Retriever=GTR, Method Type=Postprocess, # Main LM Call=10, # Verifier LM Call=02024.06 | 42.75 | — | — | — | — | 77.31 | 67.08 | 71.64 | 64.69 | — | — | |
| Summ + ICLCiteMain LM=text-unicorn, Retriever=GTR, Method Type=Preprocess, # Main LM Call=10, # Verifier LM Call=02024.06 | 42.65 | — | — | — | — | 75.68 | 61.18 | 68.91 | 62.11 | — | — | |
| CaLMMain LM=text-unicorn, Retriever=DPR, Method Type=Iterative, # Main LM Call=≤ 4, # Verifier LM Call=≤ 32024.06 | 42.24 | — | — | — | — | 77.18 | 63.71 | 64.99 | 62.03 | — | — | |
| ICLCiteMain LM=GPT-3.5-Turbo-1106, Retriever=GTR, Method Type=Single Run, # Main LM Call=1, # Verifier LM Call=02024.06 | 41.92 | — | — | — | — | 71.85 | 73.14 | 77.9 | 66.2 | — | — | |
| Summ + ICLCiteMain LM=GPT-3.5-Turbo-1106, Retriever=GTR, Method Type=Preprocess, # Main LM Call=10, # Verifier LM Call=02024.06 | 41.11 | — | — | — | — | 68.01 | 66.04 | 74.43 | 62.4 | — | — | |
| Snippet + ICLCiteMain LM=text-unicorn, Retriever=GTR, Method Type=Preprocess, # Main LM Call=10, # Verifier LM Call=02024.06 | 40.97 | — | — | — | — | 72.5 | 60.88 | 68.02 | 60.59 | — | — | |
| ICLCite + USCMain LM=text-unicorn, Retriever=GTR, Method Type=Postprocess, # Main LM Call=10, # Verifier LM Call=02024.06 | 40.75 | — | — | — | — | 63.27 | 68.9 | 67.6 | 60.13 | — | — | |
| ICLCite + USCMain LM=GPT-3.5-Turbo-1106, Retriever=DPR, Method Type=Postprocess, # Main LM Call=10, # Verifier LM Call=02024.06 | 40.71 | — | — | — | — | 77.5 | 61.2 | 64.07 | 60.87 | — | — | |
| Ret-ChatGPTRetrieval=Yes, Proprietary=true2023.10 | 40.7 | 39.9 | 79.7 | 65.1 | 76.6 | — | — | — | — | — | — | |
| Snippet + ICLCiteMain LM=GPT-3.5-Turbo-1106, Retriever=GTR, Method Type=Preprocess, # Main LM Call=10, # Verifier LM Call=02024.06 | 39.89 | — | — | — | — | 68.84 | 62.05 | 71.06 | 60.46 | — | — | |
| ICLCiteMain LM=text-unicorn, Retriever=GTR, Method Type=Single Run, # Main LM Call=1, # Verifier LM Call=02024.06 | 39.83 | — | — | — | — | 63.25 | 69.39 | 67.98 | 60.11 | — | — | |
| ICLCiteMain LM=GPT-3.5-Turbo-1106, Retriever=DPR, Method Type=Single Run, # Main LM Call=1, # Verifier LM Call=02024.06 | 39.32 | — | — | — | — | 74.73 | 67.36 | 69.48 | 62.72 | — | — | |
| Summ + ICLCiteMain LM=text-unicorn, Retriever=DPR, Method Type=Preprocess, # Main LM Call=10, # Verifier LM Call=02024.06 | 38.67 | — | — | — | — | 63.21 | 52.41 | 59.45 | 53.43 | — | — | |
| Snippet + ICLCiteMain LM=text-unicorn, Retriever=DPR, Method Type=Preprocess, # Main LM Call=10, # Verifier LM Call=02024.06 | 37.69 | — | — | — | — | 59.03 | 54.62 | 59.44 | 52.69 | — | — | |
| ICLCite + USCMain LM=text-unicorn, Retriever=DPR, Method Type=Postprocess, # Main LM Call=10, # Verifier LM Call=02024.06 | 37.16 | — | — | — | — | 57.92 | 62.05 | 60 | 54.28 | — | — | |
| ICLCiteMain LM=text-unicorn, Retriever=DPR, Method Type=Single Run, # Main LM Call=1, # Verifier LM Call=02024.06 | 37.09 | — | — | — | — | 62.01 | 62.42 | 60.35 | 55.46 | — | — | |
| ChatGPTRetrieval=No, Proprietary=true2023.10 | 35.3 | 36.2 | 68.8 | — | — | — | — | — | — | — | — | |
| AlpacaScale=13B, Retrieval=Yes2023.10 | 34.8 | 36.7 | 56.6 | 2 | 3.8 | — | — | — | — | — | — | |
| Ret-Llama2-ChatScale=13B, Retrieval=Yes, Proprietary=true2023.10 | 32.8 | 34.8 | 43.8 | 19.8 | 36.1 | — | — | — | — | — | — | |
| SELF-RAGScale=13B, Retrieval=Yes2023.10 | 31.7 | 37 | 71.6 | 70.3 | 71.3 | — | — | — | — | — | — | |
| Self-RAG (13B)Main LM=Llama-2-13B, Retriever=GTR, Method Type=Finetune LM2024.06 | 31.7 | — | — | — | — | 71.6 | 70.3 | 71.3 | 61.2 | — | — | |
| Llama2-FTScale=7B, Retrieval=Yes, Fine-tuned=true2023.10 | 31 | 35.8 | 51.2 | 5 | 7.5 | — | — | — | — | — | — | |
| AlpacaScale=7B, Retrieval=Yes2023.10 | 30.9 | 33.3 | 57.9 | 5.5 | 7.2 | — | — | — | — | — | — | |
| SELF-RAGScale=7B, Retrieval=Yes2023.10 | 30 | 35.7 | 74.3 | 66.9 | 67.8 | — | — | — | — | — | — | |
| Self-RAG (7B)Main LM=Llama-2-7B, Retriever=GTR, Method Type=Finetune LM2024.06 | 30 | — | — | — | — | 74.3 | 66.9 | 67.8 | 59.8 | — | — | |
| Snippet + ICLCiteMain LM=GPT-3.5-Turbo-1106, Retriever=DPR, Method Type=Preprocess, # Main LM Call=10, # Verifier LM Call=02024.06 | 29.48 | — | — | — | — | 48.56 | 59.52 | 53.84 | 47.85 | — | — | |
| Summ + ICLCiteMain LM=GPT-3.5-Turbo-1106, Retriever=DPR, Method Type=Preprocess, # Main LM Call=10, # Verifier LM Call=02024.06 | 29.3 | — | — | — | — | 48.95 | 60.14 | 54.52 | 48.23 | — | — | |
| AlpacaScale=13B, Retrieval=No2023.10 | 22.9 | 32 | 70.6 | — | — | — | — | — | — | — | — | |
| Llama2-ChatScale=13B, Retrieval=No, Proprietary=true2023.10 | 22.4 | 29.6 | 28.6 | — | — | — | — | — | — | — | — | |
| AlpacaScale=7B, Retrieval=No2023.10 | 18.8 | 29.4 | 61.7 | — | — | — | — | — | — | — | — | |
| Llama2Scale=13B, Retrieval=Yes2023.10 | 16.3 | 20.5 | 24.7 | 2.3 | 3.6 | — | — | — | — | — | — | |
| Llama2Scale=7B, Retrieval=Yes2023.10 | 15.2 | 22.1 | 32 | 2.9 | 4 | — | — | — | — | — | — | |
| Llama2Scale=7B, Retrieval=No2023.10 | 7.9 | 15.3 | 19 | — | — | — | — | — | — | — | — | |
| Llama2Scale=13B, Retrieval=No2023.10 | 7.2 | 12.4 | 16 | — | — | — | — | — | — | — | — | |
| ChatGPT Closed BookBase Model=ChatGPT, Pipeline Strategy=Closed Book2024.04 | — | — | — | — | — | — | — | — | — | 38.3 | 26.7 | |
| ChatGPT Inline SearchBase Model=ChatGPT, Pipeline Strategy=Inline Search2024.04 | — | — | — | — | — | — | — | — | — | 32.4 | 58.2 | |
| ChatGPT SnippetBase Model=ChatGPT, Pipeline Strategy=Snippet, Number of Passages=102024.04 | — | — | — | — | — | — | — | — | — | 41.4 | 61.1 | |
| ChatGPT SummarizationBase Model=ChatGPT, Pipeline Strategy=Summarization, Number of Passages=102024.04 | — | — | — | — | — | — | — | — | — | 43.3 | 65.2 | |
| ChatGPT VanillaBase Model=ChatGPT, Pipeline Strategy=Vanilla, Number of Passages=52024.04 | — | — | — | — | — | — | — | — | — | 40.4 | 73 | |
| LLaMA-13B VanillaBase Model=LLaMA-13B, Pipeline Strategy=Vanilla2024.04 | — | — | — | — | — | — | — | — | — | 26.9 | 12.6 | |
| LongT5 3B +Blueprint +AttributionBase Model=LongT5 3B, Pipeline Strategy=Abstractive Blueprint, Number of Passages=102024.04 | — | — | — | — | — | — | — | — | — | 33.8 | 77.8 | |
| Vicuna-13B VanillaBase Model=Vicuna-13B, Pipeline Strategy=Vanilla2024.04 | — | — | — | — | — | — | — | — | — | 31.9 | 50.6 |