List your company
ResearchFormerly WizWand

Social Interaction Question Answering on SIQA

86.9AccuracyHuman Performance
46.506456.993267.4877.9668May 2, 2020May 4, 2021May 6, 2022May 9, 2023May 10, 2024May 12, 2025May 15, 2026
Updated 5mo ago

Evaluation results

MethodLinks
Human Performance2024.0186.9
T5Model Architecture=T52020.0581.4
UNIFIEDQAModel Architecture=T52020.0581.4
LoRA-RITEModel=3B2026.0380.25
Stable-LoRAModel=3B2026.0380.25
DeBERTa-v3-L (Supervised)Evaluation Protocol=Supervised2024.0180.1
VERA-T5 (Multitask Supervised)Evaluation Protocol=Supervised2024.0180.1
LoRA+Model=3B2026.0379.94
AdamWModel=3B2026.0379.89
RiemannModel=3B2026.0379.89
RoBERTaModel Architecture=RoBERTa2020.0578
Stable-LoRAModel=1.5B2026.0377.64
LoRA+Model=1.5B2026.0377.38
AdamWModel=1.5B2026.0377.33
LoRA-RITEModel=1.5B2026.0377.23
RiemannModel=1.5B2026.0376.87
ROBERTa-L (Supervised)Evaluation Protocol=Supervised2024.0176.6
BART-largeModel Architecture=BART-large2020.0574
UnifiedQA-BARTModel Architecture=BART2020.0573.2
Stable-LoRAModel=1B2026.0372.26
AdamWModel=1B2026.0371.7
LoRA-RITEModel=1B2026.0371.6
LoRA+Model=1B2026.0371.19
ChatGPT + Chain-of-thoughtVersion=gpt-3.5-turbo, Evaluation Protocol=Zero-shot2024.0170.7
RiemannModel=1B2026.0370.62
ChatGPT + Self-consistent chain-of-thoughtVersion=gpt-3.5-turbo, Evaluation Protocol=Zero-shot2024.0169.7
ChatGPTVersion=gpt-3.5-turbo, Evaluation Protocol=Zero-shot2024.0169.5
Stable-LoRAModel=0.5B2026.0368.27
GPT-3.5Version=text-davinci-003, Evaluation Protocol=Zero-shot2024.0168
LoRA+Model=0.5B2026.0367.91
AdamWModel=0.5B2026.0367.04
RiemannModel=0.5B2026.0366.94
LoRA-RITEModel=0.5B2026.0366.89
ZS-FusionCSKB=CSKG, Evaluation Protocol=Zero-shot2024.0166.7
DeBERTa-v3-L (CANDLE Distilled)CSKB=CANDLE, Evaluation Protocol=Zero-shot2024.0165.9
CAR-ROBERTa-LCSKB=ATOMIC, Evaluation Protocol=Zero-shot2024.0164.8
BaselineBit width (b)=16.002025.0564.8
STL-AdapterCSKB=CSKG, Evaluation Protocol=Zero-shot2024.0164.7
STL-AdapterCSKB=ATOMIC, Evaluation Protocol=Zero-shot2024.0164.4
CAR-ROBERTa-LCSKB=AbsATM, Evaluation Protocol=Zero-shot2024.0164
CAR-DEBERTa-v3-LCSKB=AbsATM, Evaluation Protocol=Zero-shot2024.0164
CAR-DEBERTa-v3-LCSKB=ATOMIC, Evaluation Protocol=Zero-shot2024.0163.8
STL-PLMCSKB=ATOMIC, Evaluation Protocol=Zero-shot2024.0163.2
ROBERTa-L (MR)CSKB=CSKG, Evaluation Protocol=Zero-shot2024.0163.2
ROBERTa-L (MR)CSKB=ATOMIC, Evaluation Protocol=Zero-shot2024.0163.1
DeBERTa-v3-L (MR)CSKB=ATOMIC, Evaluation Protocol=Zero-shot2024.0162.1
MTLCSKB=CSKG, Evaluation Protocol=Zero-shot2024.0161.9
SEEDBackbone=LLaMA2-7B, Selection Ratio=5%2026.0561.8
ROBERTa-L (MR)CSKB=ATM10X, Evaluation Protocol=Zero-shot2024.0161
DeBERTa-v3-L (MR)CSKB=ATM10X, Evaluation Protocol=Zero-shot2024.0159.7
VERA-T5-xxl (CANDLE Distilled)CSKB=CANDLE, Evaluation Protocol=Zero-shot2024.0159.4
FULLBackbone=LLaMA2-7B, Selection Ratio=100%2026.0559.4
Tensor RMS + CBit width (b)=3.002025.0558.6
Task-specific FTBase Model=Llama-3.1-8B, Fine-tuning Technique=LoRA, Merging Strategy=None2026.0258.39
VERA-T5-xxlCSKB=ATM10X, Evaluation Protocol=Zero-shot2024.0158.2
VERA-T5-xxlCSKB=AbsATM, Evaluation Protocol=Zero-shot2024.0158.1
TSV-MBase Model=Llama-3.1-8B, Fine-tuning Technique=LoRA, Merging Strategy=TSV-M2026.0258.03
VERA-T5-xxlCSKB=ATOMIC, Evaluation Protocol=Zero-shot2024.0157.7
GPT-4Evaluation Protocol=Zero-shot2024.0157
TIESBase Model=Llama-3.1-8B, Fine-tuning Technique=LoRA, Merging Strategy=TIES2026.0256.96
Task-specific FTBase Model=Llama-3.1-8B, Fine-tuning Technique=OFT, Merging Strategy=None2026.0256.81
MICOCSKB=ATOMIC, Evaluation Protocol=Zero-shot2024.0156
RANDOMBackbone=LLaMA2-7B, Selection Ratio=5%2026.0555.6
OrthoMergeBase Model=Llama-3.1-8B, Fine-tuning Technique=OFT, Merging Strategy=OrthoMerge2026.0255.17
TSV-MBase Model=Llama-3.1-8B, Fine-tuning Technique=OFT, Merging Strategy=TSV-M2026.0255.02
ROBERTa-L (MR)CSKB=CWWV, Evaluation Protocol=Zero-shot2024.0154.8
ProQA2023.0554.5
TABase Model=Llama-3.1-8B, Fine-tuning Technique=LoRA, Merging Strategy=TA2026.0254.15
DAREBase Model=Llama-3.1-8B, Fine-tuning Technique=LoRA, Merging Strategy=DARE2026.0253.84
TIESBase Model=Llama-3.1-8B, Fine-tuning Technique=OFT, Merging Strategy=TIES2026.0253.74
ZS-FusionCSKB=CWWV, Evaluation Protocol=Zero-shot2024.0153.7
Tensor RMS + SpBit width (b)=3.052025.0553.6
KoCoModel Scale=1.6B Parameters2026.0453.4
URL Prefix (MeCo)Model Scale=1.6B Parameters2026.0452.9
Standard CPTModel Scale=1.6B Parameters2026.0452.7
Muppet2023.0552.63
Data SelectionModel Scale=1.6B Parameters2026.0452.6
OLTQAConfiguration=Full2023.0552.51
Hyperformer++2023.0552.46
TABase Model=Llama-3.1-8B, Fine-tuning Technique=OFT, Merging Strategy=TA2026.0252.2
DAREBase Model=Llama-3.1-8B, Fine-tuning Technique=OFT, Merging Strategy=DARE2026.0252.1
MTLCSKB=CWWV, Evaluation Protocol=Zero-shot2024.0152
OLTQAAblation=without Pk2023.0551.76
OLTQAKnowledge Distillation=Static MKD2023.0551.48
OLTQARetriever=EPR2023.0551.23
OLTQAKnowledge Distillation=Back KD2023.0550.72
OLTQARe-ranker=excluded2023.0550.67
OLTQARetriever=BM252023.0550.41
LLAMA2Parameters=13B, Evaluation Protocol=Zero-shot2024.0150.3
UnifiedQA2023.0550.15
COMET-DynGenCSKB=ATOMIC, Evaluation Protocol=Zero-shot2024.0150.1
OLTQAKnowledge Distillation=without MKD2023.0549.85
Distributed Lion-MaVoShots=3, Optimizer=D-Lion (MaVo), Model=LLaMA 7B, Instruction Finetuning=true2024.0349.75
G-LionShots=3, Optimizer=Global Lion, Model=LLaMA 7B, Instruction Finetuning=true2024.0349.64
OLTQAAblation=without Pm2023.0549.64
G-AdamWShots=3, Optimizer=Global AdamW, Model=LLaMA 7B, Instruction Finetuning=true2024.0348.97
LLAMA2Parameters=7B, Evaluation Protocol=Zero-shot2024.0148.3
BASEBackbone=LLaMA2-7B, Selection Ratio=0%2026.0548.3
Llama-3.1-8BBase Model=Llama-3.1-8B, Fine-tuning Technique=None, Merging Strategy=None2026.0248.11
Distributed Lion-AvgShots=3, Optimizer=D-Lion (Avg), Model=LLaMA 7B, Instruction Finetuning=true2024.0348.06