Question Answering on TriviaQA (test)
85.18AccuracyRankCoT
Evaluation Results
| Method | Links | |
|---|---|---|
| RankCoT2025.02 | 85.18 | |
| Vanilla RAG2025.02 | 82.85 | |
| Recomp2025.02 | 79.04 | |
| Self-RAGBackbone=Llama3-8B-Instruct2025.02 | 78.35 | |
| SEGENCBackbone=BART2025.02 | 76.73 | |
| RPOGeneration LLM=LLaMA3-8B-instruct, Adaptive Category=Integrated-Eval, #API/LM calls=12025.01 | 74.4 | |
| Self-RAGGeneration LLM=LLaMA3-8B-instruct, Adaptive Category=Post-Eval, #API/LM calls=2-112025.01 | 71.4 | |
| RTNPrecision (W-A-KV)=8-8-8, Size (GB)=30.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 70.4 | |
| RTNPrecision (W-A-KV)=8-8-16, Size (GB)=30.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 70.3 | |
| SmoothQuantPrecision (W-A-KV)=8-8-16, Size (GB)=30.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 70.3 | |
| SmoothQuantPrecision (W-A-KV)=8-8-8, Size (GB)=30.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 70.2 | |
| LLM-QATPrecision (W-A-KV)=8-8-16, Size (GB)=30.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 70 | |
| LLaMA-30BPrecision (W-A-KV)=16-16-16, Size (GB)=60.6, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 69.9 | |
| LLM-QATPrecision (W-A-KV)=8-8-8, Size (GB)=30.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 69.4 | |
| Self-RAGGeneration LLM=LLaMA2-7B, Adaptive Category=Post-Eval, #API/LM calls=2-112025.01 | 68.9 | |
| SMOOTHIE-GLOBALEnsemble Size=7B2024.12 | 68.3 | |
| LLM-QATPrecision (W-A-KV)=8-8-4, Size (GB)=30.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 67.9 | |
| Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.3, Lambda (λ)=0.1, Alpha (α)=1.52025.05 | 67.8 | |
| top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.3, k derived from λ=0.1, k derived from α=1.52025.05 | 67.58 | |
| top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.3, k derived from λ=0.1, k derived from α=22025.05 | 67.58 | |
| top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.7, k derived from λ=0.1, k derived from α=1.52025.05 | 67.58 | |
| top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.7, k derived from λ=0.1, k derived from α=22025.05 | 67.58 | |
| top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1, k derived from λ=0.1, k derived from α=1.52025.05 | 67.58 | |
| top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1, k derived from λ=0.1, k derived from α=22025.05 | 67.58 | |
| top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1.5, k derived from λ=0.1, k derived from α=1.52025.05 | 67.58 | |
| top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1.5, k derived from λ=0.1, k derived from α=22025.05 | 67.58 | |
| Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.3, Lambda (λ)=0.1, Alpha (α)=22025.05 | 67.47 | |
| RTNPrecision (W-A-KV)=4-8-16, Size (GB)=15.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 67.1 | |
| LLM-QATPrecision (W-A-KV)=4-8-16, Size (GB)=15.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 67.1 | |
| Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1.5, Lambda (λ)=0.1, Alpha (α)=22025.05 | 66.96 | |
| RTNPrecision (W-A-KV)=4-8-8, Size (GB)=15.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 66.8 | |
| top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.3, k derived from λ=0.01, k derived from α=1.52025.05 | 66.74 | |
| top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.3, k derived from λ=0.01, k derived from α=22025.05 | 66.74 | |
| Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.3, Lambda (λ)=0.01, Alpha (α)=22025.05 | 66.69 | |
| Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1, Lambda (λ)=0.1, Alpha (α)=22025.05 | 66.69 | |
| Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.3, Lambda (λ)=0.01, Alpha (α)=1.52025.05 | 66.57 | |
| Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.7, Lambda (λ)=0.1, Alpha (α)=22025.05 | 66.35 | |
| LLM-QATPrecision (W-A-KV)=4-8-8, Size (GB)=15.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 65.9 | |
| RAGGeneration LLM=LLaMA3-8B-instruct, #API/LM calls=12025.01 | 65.8 | |
| BEST-ON-VALEnsemble Size=7B2024.12 | 65.8 | |
| RAGGeneration LLM=ChatGPT, #API/LM calls=12025.01 | 65.7 | |
| Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.7, Lambda (λ)=0.1, Alpha (α)=1.52025.05 | 65.68 | |
| Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1, Lambda (λ)=0.1, Alpha (α)=1.52025.05 | 65.63 | |
| InstructRAGGeneration LLM=LLaMA3-8B-instruct, #API/LM calls=12025.01 | 65.1 | |
| top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.7, k derived from λ=0.01, k derived from α=1.52025.05 | 65.01 | |
| top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.7, k derived from λ=0.01, k derived from α=22025.05 | 65.01 | |
| Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1.5, Lambda (λ)=0.1, Alpha (α)=1.52025.05 | 64.85 | |
| Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.7, Lambda (λ)=0.01, Alpha (α)=1.52025.05 | 64.23 | |
| LLM-QATPrecision (W-A-KV)=4-6-16, Size (GB)=15.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 64.2 | |
| Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.7, Lambda (λ)=0.01, Alpha (α)=22025.05 | 63.84 | |
| LLaMA-13BPrecision (W-A-KV)=16-16-16, Size (GB)=24.2, Evaluation=5-shot, Backbone=LLaMA-13B2023.05 | 63.7 | |
| LLM-QATPrecision (W-A-KV)=4-8-4, Size (GB)=15.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 63.5 | |
| SmoothQuantPrecision (W-A-KV)=8-8-8, Size (GB)=12.4, Evaluation=5-shot, Backbone=LLaMA-13B2023.05 | 63.4 | |
| SmoothQuantPrecision (W-A-KV)=8-8-16, Size (GB)=12.4, Evaluation=5-shot, Backbone=LLaMA-13B2023.05 | 63.4 | |
| LLM-QATPrecision (W-A-KV)=8-8-16, Size (GB)=12.4, Evaluation=5-shot, Backbone=LLaMA-13B2023.05 | 63.4 | |
| LLM-QATPrecision (W-A-KV)=8-8-8, Size (GB)=12.4, Evaluation=5-shot, Backbone=LLaMA-13B2023.05 | 63.3 | |
| RTNPrecision (W-A-KV)=8-8-16, Size (GB)=12.4, Evaluation=5-shot, Backbone=LLaMA-13B2023.05 | 63.1 | |
| RTNPrecision (W-A-KV)=8-8-8, Size (GB)=12.4, Evaluation=5-shot, Backbone=LLaMA-13B2023.05 | 62.9 | |
| top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1, k derived from λ=0.01, k derived from α=1.52025.05 | 62.67 | |
| top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1, k derived from λ=0.01, k derived from α=22025.05 | 62.67 | |
| Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1, Lambda (λ)=0.01, Alpha (α)=22025.05 | 61.17 | |
| Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1, Lambda (λ)=0.01, Alpha (α)=1.52025.05 | 61.06 | |
| Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1.5, Lambda (λ)=0.01, Alpha (α)=22025.05 | 60.84 | |
| top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1.5, k derived from λ=0.01, k derived from α=1.52025.05 | 60.84 | |
| top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1.5, k derived from λ=0.01, k derived from α=22025.05 | 60.84 | |
| LLM-QATPrecision (W-A-KV)=4-8-16, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-13B2023.05 | 59.8 | |
| Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1.5, Lambda (λ)=0.01, Alpha (α)=1.52025.05 | 59.78 | |
| CRAGGeneration LLM=LLaMA2-7B, Adaptive Category=Pre-Eval, #API/LM calls=62025.01 | 59.6 | |
| LLM-QATPrecision (W-A-KV)=4-8-8, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-13B2023.05 | 59.3 | |
| RANDOMEnsemble Size=7B2024.12 | 59.3 | |
| LLM-QATPrecision (W-A-KV)=8-8-4, Size (GB)=12.4, Evaluation=5-shot, Backbone=LLaMA-13B2023.05 | 58.8 | |
| SmoothQuantPrecision (W-A-KV)=4-8-16, Size (GB)=15.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 58.5 | |
| RTNPrecision (W-A-KV)=4-8-8, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-13B2023.05 | 57.9 | |
| RTNPrecision (W-A-KV)=4-8-16, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-13B2023.05 | 57.9 | |
| SmoothQuantPrecision (W-A-KV)=4-8-8, Size (GB)=15.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 57.9 | |
| LLM-QATPrecision (W-A-KV)=4-6-16, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-13B2023.05 | 57.7 | |
| RPOGeneration LLM=LLaMA2-7B, Adaptive Category=Integrated-Eval, #API/LM calls=12025.01 | 57.6 | |
| LLaMA-7BPrecision (W-A-KV)=16-16-16, Size (GB)=12.6, Evaluation=5-shot, Backbone=LLaMA-7B2023.05 | 57 | |
| SmoothQuantPrecision (W-A-KV)=8-8-16, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-7B2023.05 | 56.8 | |
| SmoothQuantPrecision (W-A-KV)=8-8-8, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-7B2023.05 | 56.7 | |
| SmoothQuantPrecision (W-A-KV)=4-6-16, Size (GB)=15.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.05 | 56.7 | |
| RTNPrecision (W-A-KV)=8-8-8, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-7B2023.05 | 56.6 | |
| RTNPrecision (W-A-KV)=8-8-16, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-7B2023.05 | 56.6 | |
| LLM-QATPrecision (W-A-KV)=8-8-8, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-7B2023.05 | 56.1 | |
| LLM-QATPrecision (W-A-KV)=8-8-16, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-7B2023.05 | 56.1 | |
| RAG + SFTGeneration LLM=LLaMA2-7B, #API/LM calls=12025.01 | 54.3 | |
| LLM-QATPrecision (W-A-KV)=4-8-4, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-13B2023.05 | 54.3 | |
| RAGGeneration LLM=LLaMA2-7B, #API/LM calls=12025.01 | 52.5 | |
| RAG + DPOGeneration LLM=LLaMA2-7B, #API/LM calls=12025.01 | 51.7 | |
| LLM-QATPrecision (W-A-KV)=4-8-8, Size (GB)=3.5, Evaluation=5-shot, Backbone=LLaMA-7B2023.05 | 50.8 | |
| LLM-QATPrecision (W-A-KV)=4-8-16, Size (GB)=3.5, Evaluation=5-shot, Backbone=LLaMA-7B2023.05 | 50.8 | |
| SmoothQuantPrecision (W-A-KV)=8-8-4, Size (GB)=12.4, Evaluation=5-shot, Backbone=LLaMA-13B2023.05 | 49.6 | |
| LLM-QATPrecision (W-A-KV)=4-6-16, Size (GB)=3.5, Evaluation=5-shot, Backbone=LLaMA-7B2023.05 | 49 | |
| AstuteRAGGeneration LLM=ChatGPT, Adaptive Category=Pre-Eval, #API/LM calls=2-42025.01 | 47.6 | |
| SmoothQuantPrecision (W-A-KV)=4-8-8, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-13B2023.05 | 46.6 | |
| SmoothQuantPrecision (W-A-KV)=4-8-16, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-13B2023.05 | 46.6 | |
| LLM-QATPrecision (W-A-KV)=8-8-4, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-7B2023.05 | 46.2 | |
| RTNPrecision (W-A-KV)=4-8-16, Size (GB)=3.5, Evaluation=5-shot, Backbone=LLaMA-7B2023.05 | 44.9 | |
| SmoothQuantPrecision (W-A-KV)=4-6-16, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-13B2023.05 | 44.8 | |
| RTNPrecision (W-A-KV)=4-8-8, Size (GB)=3.5, Evaluation=5-shot, Backbone=LLaMA-7B2023.05 | 44.5 |