List your company
ResearchFormerly WizWand

Coherence judging on TRIP (test)

70.7AccuracySITUATIONSUPERVISION (GPT-3)
35.13244.36653.662.834Dec 20, 2022
Updated 8mo ago

Evaluation results

MethodLinks
SITUATIONSUPERVISION (GPT-3)Total examples (|X|)=80, State supervision examples (|XA|)=80, SITUATION-SUPERVISION=Aux Only, Setting=Prompting2022.1270.7
SITUATIONSUPERVISIONTotal Examples (|X|)=80, State Supervision (|XA|)=80, Supervision Type=Aux Only2022.1270.7
SITUATIONSUPERVISION (GPT-3)Total examples (|X|)=80, State supervision examples (|XA|)=20, SITUATION-SUPERVISION=Aux + Latent, Setting=Prompting2022.1267.1
SITUATIONSUPERVISIONTotal Examples (|X|)=80, State Supervision (|XA|)=20, Supervision Type=Aux+Latent2022.1267.1
GPT-3 (Baseline)Total examples (|X|)=80, State supervision examples (|XA|)=0, SITUATION-SUPERVISION=None, Setting=Prompting2022.1259.5
GPT-3Total Examples (|X|)=80, State Supervision (|XA|)=0, Supervision Type=None2022.1259.5
SITUATIONSUPERVISION (GPT-3)Total examples (|X|)=80, State supervision examples (|XA|)=20, SITUATION-SUPERVISION=Aux Only, Setting=Prompting2022.1258.2
SITUATIONSUPERVISIONTotal Examples (|X|)=80, State Supervision (|XA|)=20, Supervision Type=Aux Only2022.1258.2
SITUATIONSUPERVISION (BART)Total examples (|X|)=1k, State supervision examples (|XA|)=500, SITUATION-SUPERVISION=Aux Only, Setting=Fine-tuning2022.1243.6
SITUATIONSUPERVISION (BART)Total examples (|X|)=1k, State supervision examples (|XA|)=500, SITUATION-SUPERVISION=Aux + Latent, Setting=Fine-tuning2022.1243.6
SITUATIONSUPERVISION (BART)Total examples (|X|)=1k, State supervision examples (|XA|)=1k, SITUATION-SUPERVISION=Aux Only, Setting=Fine-tuning2022.1243
BART (Baseline)Total examples (|X|)=1k, State supervision examples (|XA|)=0, SITUATION-SUPERVISION=None, Setting=Fine-tuning2022.1236.5