List your company
ResearchFormerly WizWand

Question Answering on TriviaQA (test)

85.18AccuracyRankCoT
42.872853.856464.8475.8236May 29, 2023Sep 27, 2023Jan 26, 2024May 26, 2024Sep 24, 2024Jan 23, 2025May 25, 2025
Updated 8mo ago

Evaluation results

MethodLinks
RankCoT2025.0285.18
Vanilla RAG2025.0282.85
Recomp2025.0279.04
Self-RAGBackbone=Llama3-8B-Instruct2025.0278.35
SEGENCBackbone=BART2025.0276.73
RPOGeneration LLM=LLaMA3-8B-instruct, Adaptive Category=Integrated-Eval, #API/LM calls=12025.0174.4
Self-RAGGeneration LLM=LLaMA3-8B-instruct, Adaptive Category=Post-Eval, #API/LM calls=2-112025.0171.4
RTNPrecision (W-A-KV)=8-8-8, Size (GB)=30.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.0570.4
RTNPrecision (W-A-KV)=8-8-16, Size (GB)=30.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.0570.3
SmoothQuantPrecision (W-A-KV)=8-8-16, Size (GB)=30.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.0570.3
SmoothQuantPrecision (W-A-KV)=8-8-8, Size (GB)=30.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.0570.2
LLM-QATPrecision (W-A-KV)=8-8-16, Size (GB)=30.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.0570
LLaMA-30BPrecision (W-A-KV)=16-16-16, Size (GB)=60.6, Evaluation=5-shot, Backbone=LLaMA-30B2023.0569.9
LLM-QATPrecision (W-A-KV)=8-8-8, Size (GB)=30.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.0569.4
Self-RAGGeneration LLM=LLaMA2-7B, Adaptive Category=Post-Eval, #API/LM calls=2-112025.0168.9
SMOOTHIE-GLOBALEnsemble Size=7B2024.1268.3
LLM-QATPrecision (W-A-KV)=8-8-4, Size (GB)=30.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.0567.9
Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.3, Lambda (λ)=0.1, Alpha (α)=1.52025.0567.8
top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.3, k derived from λ=0.1, k derived from α=1.52025.0567.58
top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.3, k derived from λ=0.1, k derived from α=22025.0567.58
top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.7, k derived from λ=0.1, k derived from α=1.52025.0567.58
top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.7, k derived from λ=0.1, k derived from α=22025.0567.58
top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1, k derived from λ=0.1, k derived from α=1.52025.0567.58
top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1, k derived from λ=0.1, k derived from α=22025.0567.58
top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1.5, k derived from λ=0.1, k derived from α=1.52025.0567.58
top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1.5, k derived from λ=0.1, k derived from α=22025.0567.58
Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.3, Lambda (λ)=0.1, Alpha (α)=22025.0567.47
RTNPrecision (W-A-KV)=4-8-16, Size (GB)=15.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.0567.1
LLM-QATPrecision (W-A-KV)=4-8-16, Size (GB)=15.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.0567.1
Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1.5, Lambda (λ)=0.1, Alpha (α)=22025.0566.96
RTNPrecision (W-A-KV)=4-8-8, Size (GB)=15.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.0566.8
top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.3, k derived from λ=0.01, k derived from α=1.52025.0566.74
top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.3, k derived from λ=0.01, k derived from α=22025.0566.74
Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.3, Lambda (λ)=0.01, Alpha (α)=22025.0566.69
Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1, Lambda (λ)=0.1, Alpha (α)=22025.0566.69
Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.3, Lambda (λ)=0.01, Alpha (α)=1.52025.0566.57
Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.7, Lambda (λ)=0.1, Alpha (α)=22025.0566.35
LLM-QATPrecision (W-A-KV)=4-8-8, Size (GB)=15.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.0565.9
RAGGeneration LLM=LLaMA3-8B-instruct, #API/LM calls=12025.0165.8
BEST-ON-VALEnsemble Size=7B2024.1265.8
RAGGeneration LLM=ChatGPT, #API/LM calls=12025.0165.7
Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.7, Lambda (λ)=0.1, Alpha (α)=1.52025.0565.68
Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1, Lambda (λ)=0.1, Alpha (α)=1.52025.0565.63
InstructRAGGeneration LLM=LLaMA3-8B-instruct, #API/LM calls=12025.0165.1
top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.7, k derived from λ=0.01, k derived from α=1.52025.0565.01
top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.7, k derived from λ=0.01, k derived from α=22025.0565.01
Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1.5, Lambda (λ)=0.1, Alpha (α)=1.52025.0564.85
Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.7, Lambda (λ)=0.01, Alpha (α)=1.52025.0564.23
LLM-QATPrecision (W-A-KV)=4-6-16, Size (GB)=15.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.0564.2
Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=0.7, Lambda (λ)=0.01, Alpha (α)=22025.0563.84
LLaMA-13BPrecision (W-A-KV)=16-16-16, Size (GB)=24.2, Evaluation=5-shot, Backbone=LLaMA-13B2023.0563.7
LLM-QATPrecision (W-A-KV)=4-8-4, Size (GB)=15.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.0563.5
SmoothQuantPrecision (W-A-KV)=8-8-8, Size (GB)=12.4, Evaluation=5-shot, Backbone=LLaMA-13B2023.0563.4
SmoothQuantPrecision (W-A-KV)=8-8-16, Size (GB)=12.4, Evaluation=5-shot, Backbone=LLaMA-13B2023.0563.4
LLM-QATPrecision (W-A-KV)=8-8-16, Size (GB)=12.4, Evaluation=5-shot, Backbone=LLaMA-13B2023.0563.4
LLM-QATPrecision (W-A-KV)=8-8-8, Size (GB)=12.4, Evaluation=5-shot, Backbone=LLaMA-13B2023.0563.3
RTNPrecision (W-A-KV)=8-8-16, Size (GB)=12.4, Evaluation=5-shot, Backbone=LLaMA-13B2023.0563.1
RTNPrecision (W-A-KV)=8-8-8, Size (GB)=12.4, Evaluation=5-shot, Backbone=LLaMA-13B2023.0562.9
top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1, k derived from λ=0.01, k derived from α=1.52025.0562.67
top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1, k derived from λ=0.01, k derived from α=22025.0562.67
Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1, Lambda (λ)=0.01, Alpha (α)=22025.0561.17
Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1, Lambda (λ)=0.01, Alpha (α)=1.52025.0561.06
Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1.5, Lambda (λ)=0.01, Alpha (α)=22025.0560.84
top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1.5, k derived from λ=0.01, k derived from α=1.52025.0560.84
top-k decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1.5, k derived from λ=0.01, k derived from α=22025.0560.84
LLM-QATPrecision (W-A-KV)=4-8-16, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-13B2023.0559.8
Bregman primal decodingBackbone=LLaMA 3.1 8B, Temperature (Temp)=1.5, Lambda (λ)=0.01, Alpha (α)=1.52025.0559.78
CRAGGeneration LLM=LLaMA2-7B, Adaptive Category=Pre-Eval, #API/LM calls=62025.0159.6
LLM-QATPrecision (W-A-KV)=4-8-8, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-13B2023.0559.3
RANDOMEnsemble Size=7B2024.1259.3
LLM-QATPrecision (W-A-KV)=8-8-4, Size (GB)=12.4, Evaluation=5-shot, Backbone=LLaMA-13B2023.0558.8
SmoothQuantPrecision (W-A-KV)=4-8-16, Size (GB)=15.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.0558.5
RTNPrecision (W-A-KV)=4-8-8, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-13B2023.0557.9
RTNPrecision (W-A-KV)=4-8-16, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-13B2023.0557.9
SmoothQuantPrecision (W-A-KV)=4-8-8, Size (GB)=15.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.0557.9
LLM-QATPrecision (W-A-KV)=4-6-16, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-13B2023.0557.7
RPOGeneration LLM=LLaMA2-7B, Adaptive Category=Integrated-Eval, #API/LM calls=12025.0157.6
LLaMA-7BPrecision (W-A-KV)=16-16-16, Size (GB)=12.6, Evaluation=5-shot, Backbone=LLaMA-7B2023.0557
SmoothQuantPrecision (W-A-KV)=8-8-16, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-7B2023.0556.8
SmoothQuantPrecision (W-A-KV)=8-8-8, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-7B2023.0556.7
SmoothQuantPrecision (W-A-KV)=4-6-16, Size (GB)=15.7, Evaluation=5-shot, Backbone=LLaMA-30B2023.0556.7
RTNPrecision (W-A-KV)=8-8-8, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-7B2023.0556.6
RTNPrecision (W-A-KV)=8-8-16, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-7B2023.0556.6
LLM-QATPrecision (W-A-KV)=8-8-8, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-7B2023.0556.1
LLM-QATPrecision (W-A-KV)=8-8-16, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-7B2023.0556.1
RAG + SFTGeneration LLM=LLaMA2-7B, #API/LM calls=12025.0154.3
LLM-QATPrecision (W-A-KV)=4-8-4, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-13B2023.0554.3
RAGGeneration LLM=LLaMA2-7B, #API/LM calls=12025.0152.5
RAG + DPOGeneration LLM=LLaMA2-7B, #API/LM calls=12025.0151.7
LLM-QATPrecision (W-A-KV)=4-8-8, Size (GB)=3.5, Evaluation=5-shot, Backbone=LLaMA-7B2023.0550.8
LLM-QATPrecision (W-A-KV)=4-8-16, Size (GB)=3.5, Evaluation=5-shot, Backbone=LLaMA-7B2023.0550.8
SmoothQuantPrecision (W-A-KV)=8-8-4, Size (GB)=12.4, Evaluation=5-shot, Backbone=LLaMA-13B2023.0549.6
LLM-QATPrecision (W-A-KV)=4-6-16, Size (GB)=3.5, Evaluation=5-shot, Backbone=LLaMA-7B2023.0549
AstuteRAGGeneration LLM=ChatGPT, Adaptive Category=Pre-Eval, #API/LM calls=2-42025.0147.6
SmoothQuantPrecision (W-A-KV)=4-8-8, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-13B2023.0546.6
SmoothQuantPrecision (W-A-KV)=4-8-16, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-13B2023.0546.6
LLM-QATPrecision (W-A-KV)=8-8-4, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-7B2023.0546.2
RTNPrecision (W-A-KV)=4-8-16, Size (GB)=3.5, Evaluation=5-shot, Backbone=LLaMA-7B2023.0544.9
SmoothQuantPrecision (W-A-KV)=4-6-16, Size (GB)=6.5, Evaluation=5-shot, Backbone=LLaMA-13B2023.0544.8
RTNPrecision (W-A-KV)=4-8-8, Size (GB)=3.5, Evaluation=5-shot, Backbone=LLaMA-7B2023.0544.5