List your company
ResearchFormerly WizWand

Truthfulness on TruthfulQA (Accuracy (%))

97.55Truthfulness AccuracyMetaCrit + Claude-3.5-Sonnet
-3.38222.821549.02575.2285May 17, 2025Jul 8, 2025Aug 29, 2025Oct 20, 2025Dec 11, 2025Feb 1, 2026Mar 26, 2026
Updated 5mo ago

Evaluation results

MethodLinks
MetaCrit + Claude-3.5-SonnetGroup=Ours, Base Model=Claude-3.5-Sonnet, Monitoring agent (phi_up)=included, Control agent (phi_down)=included2025.0797.55
MetaCrit + GPT-4oGroup=Ours, Base Model=GPT-4o, Monitoring agent (phi_up)=included, Control agent (phi_down)=included2025.0795.83
MetaCrit + GPT-3.5-TurboGroup=Ours, Base Model=GPT-3.5-Turbo, Monitoring agent (phi_up)=included, Control agent (phi_down)=included2025.0794.12
MetaCrit + DeepSeek-v3Group=Ours, Base Model=DeepSeek-v3, Monitoring agent (phi_up)=included, Control agent (phi_down)=included2025.0793.75
GPT-4oEvaluation Protocol=Closed-Source API, Emission (gCO2/q)=4.52†, Throughput (Tok/s)=55.22026.0392.2
Gemini 2.5 ProEvaluation Protocol=Closed-Source API, Emission (gCO2/q)=3.90†, Throughput (Tok/s)=58.42026.0390.1
Claude 3.5 SonnetEvaluation Protocol=Closed-Source API, Emission (gCO2/q)=3.85†, Throughput (Tok/s)=62.12026.0389.5
MEPGroup=Baselines, Base Model=GPT-3.5-Turbo2025.0789.35
EcoThinkEvaluation Protocol=Adaptive Inference, Emission (gCO2/q)=1.32, Throughput (Tok/s)=148.62026.0388.7
Qwen-3-8B-InstructEvaluation Protocol=Standard CoT, Emission (gCO2/q)=2.12, Throughput (Tok/s)=98.52026.0381.5
MADGroup=Baselines, Base Model=GPT-3.5-Turbo2025.0780.67
ExpertPromptingGroup=Baselines, Base Model=GPT-3.5-Turbo2025.0780.66
FrugalGPT (Cascade)Evaluation Protocol=Standard CoT, Emission (gCO2/q)=1.95, Throughput (Tok/s)=88.52026.0380.2
Llama-3.1-8B-InstructEvaluation Protocol=Standard CoT, Emission (gCO2/q)=2.15, Throughput (Tok/s)=95.82026.0378.4
Universal Self-consistencyGroup=Baselines, Base Model=GPT-3.5-Turbo2025.0777.11
MetaCrit w/o ϕ↓Group=Ablation, Base Model=GPT-3.5-Turbo, Monitoring agent (phi_up)=included, Control agent (phi_down)=excluded2025.0776.62
Self-refineGroup=Baselines, Base Model=GPT-3.5-Turbo2025.0775.89
MetaCrit w/o ϕ↑Group=Ablation, Base Model=GPT-3.5-Turbo, Monitoring agent (phi_up)=excluded, Control agent (phi_down)=included2025.0775.28
Zero-shot-CoTGroup=Baselines, Base Model=GPT-3.5-Turbo2025.0770.38
Qwen-2.5-7B-InstructDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.1163.1
Gemma-2-9B-itDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.1161.4
TI-DPOModel=LLaMA-3.2-3B2025.0557
OLMo-2-7B-1124-InstructDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.1156.3
Ministral-8B-InstructDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.1155.5
Mutual-TaughtBackbone=Llama-3-8B-Instruct, Fine-tuning method=Mutual-Taught2025.0555.21
OLMo-v1.7-7B-InstructDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.1155.2
PerSynStudent Model=Qwen2.5-3B2025.1055.14
Llama-3.1-8B-InstructDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.1155.1
Tulu 3 8BDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.1155
IndividualBackbone=Qwen-14B2025.1254.35
DTS-TBackbone=Qwen-14B2025.1254.12
DTS-DBackbone=Qwen-14B2025.1254.11
TPOModel=LLaMA-3.2-3B2025.0554
DTS-T*Backbone=Qwen-14B, Variant=Optimized2025.1253.99
DTS-D*Backbone=Qwen-14B, Variant=Optimized2025.1253.99
CARStudent Model=Qwen2.5-3B2025.1053.81
GRPOModel=LLaMA-3.2-3B2025.0553.8
T-SwitchBackbone=Qwen-14B2025.1253.72
Task-ArithmeticBackbone=Qwen-14B2025.1253.52
CPOModel=LLaMA-3.2-3B2025.0553.5
Family-StrongStudent Model=Qwen2.5-3B2025.1053.37
KTOModel=LLaMA-3.2-3B2025.0553
Iterative DPOBackbone=Llama-3-8B-Instruct, Fine-tuning method=Iterative DPO2025.0552.91
EMR-MERGINGBackbone=Qwen-14B2025.1252.91
DAREBackbone=Qwen-14B2025.1252.79
Twin-MergingBackbone=Qwen-14B2025.1252.78
PerSynStudent Model=Qwen2.5-1.5B2025.1052.22
DPOModel=LLaMA-3.2-3B2025.0552
TDPOModel=LLaMA-3.2-3B2025.0552
MixStudent Model=Qwen2.5-3B2025.1051.82
Llama-3-8B-InstructFine-tuning method=None (Base)2025.0551.64
MAP-Neo-7B-InstructDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.1151.6
SIMPOModel=LLaMA-3.2-3B2025.0551.5
Ties-MergingBackbone=Qwen-14B2025.1251.46
StrongStudent Model=Qwen2.5-3B2025.1051.21
SFTModel=LLaMA-3.2-3B2025.0551
CARStudent Model=Qwen2.5-1.5B2025.1050.98
Family-StrongStudent Model=Qwen2.5-1.5B2025.1050.45
MixStudent Model=Qwen2.5-1.5B2025.1049.73
OLMoE-1B-7B-0924-InstructDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.1149.1
StrongStudent Model=Qwen2.5-1.5B2025.1049.04
IPOModel=LLaMA-3.2-3B2025.0549
OLMo-2-7B-SFTDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.1147.8
EvaByte-SFTDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.1146.3
PerSynStudent Model=Llama-3.2-3B2025.1046.15
OLMo-7B-InstructDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.1144.5
CARStudent Model=Llama-3.2-3B2025.1044.32
PerSynStudent Model=Gemma-2-2B2025.1043.87
MixStudent Model=Llama-3.2-3B2025.1043.56
PerSynStudent Model=Qwen2.5-0.5B2025.1043.01
Family-StrongStudent Model=Llama-3.2-3B2025.1042.63
StrongStudent Model=Llama-3.2-3B2025.1042.31
CARStudent Model=Gemma-2-2B2025.1042.28
CARStudent Model=Qwen2.5-0.5B2025.1041.85
Family-StrongStudent Model=Gemma-2-2B2025.1041.64
Family-StrongStudent Model=Qwen2.5-0.5B2025.1041.43
MixStudent Model=Gemma-2-2B2025.1040.83
MixStudent Model=Qwen2.5-0.5B2025.1040.54
StrongStudent Model=Gemma-2-2B2025.1040.17
StrongStudent Model=Qwen2.5-0.5B2025.1039.89
R1 - 8B + UnsafeChain fullSize=8B, Alignment=UnsafeChain (full)2025.0724
R1 - 8BSize=8B2025.0723
R1 - 8B + STAR-1Size=8B, Alignment=STAR-12025.0720.5
R1 - 8B + UnsafeChain randomSize=8B, Alignment=UnsafeChain (random subset)2025.0720.5
R1 - 8B + UnsafeChain selectedSize=8B, Alignment=UnsafeChain (selected subset)2025.0720.5
R1 - 8B + SafeChainSize=8B, Alignment=SafeChain2025.0714
R1 - 7B + UnsafeChain fullSize=7B, Alignment=UnsafeChain (full)2025.0714
R1 - 7B + STAR-1Size=7B, Alignment=STAR-12025.0712
R1 - 7B + UnsafeChain selectedSize=7B, Alignment=UnsafeChain (selected subset)2025.0711.5
R1 - 7BSize=7B2025.0711
R1 - 7B + UnsafeChain randomSize=7B, Alignment=UnsafeChain (random subset)2025.077
R1 - 1.5B + SafeChainSize=1.5B, Alignment=SafeChain2025.077
R1 - 1.5B + UnsafeChain fullSize=1.5B, Alignment=UnsafeChain (full)2025.076.5
R1 - 1.5B + UnsafeChain randomSize=1.5B, Alignment=UnsafeChain (random subset)2025.072
R1 - 1.5B + UnsafeChain selectedSize=1.5B, Alignment=UnsafeChain (selected subset)2025.072
R1 - 1.5BSize=1.5B2025.071.5
R1 - 1.5B + STAR-1Size=1.5B, Alignment=STAR-12025.071.5
R1 - 7B + SafeChainSize=7B, Alignment=SafeChain2025.070.5