List your company
ResearchFormerly WizWand

LLM-as-a-Judge Evaluation on FLASK

0.589Pearson's rQwen3-32B REAL (ours)
0.184440.289470.39450.49953Mar 6, 2025May 7, 2025Jul 9, 2025Sep 10, 2025Nov 11, 2025Jan 13, 2026Mar 17, 2026
Updated 7mo ago

Evaluation results

MethodLinks
Qwen3-32B REAL (ours)Training=RL, Inference=RAIL2026.030.5890.5860.474
Mistral2-7B REAL (ours)Training=RL, Inference=RAIL2026.030.560.5410.411
Qwen3-32B BaseTraining=None, Inference=RAIL2026.030.5430.6040.472
Qwen3-8B REAL (ours)Training=RL, Inference=RAIL2026.030.5380.5390.431
Prometheus-2-7BTraining=SFT, Inference=RAIL2026.030.5250.5140.392
Qwen3-8B Standard RLTraining=RL, Inference=RAIL2026.030.5230.5240.395
Qwen3-32B RAFTTraining=SFT, Inference=RAIL2026.030.5210.5290.399
TRACTBackbone=Mistral-7B-Instruct, CoT=true, Train Objective=C-RAFT, Training Data=Self, Inference Method=C-RAIL2025.030.5180.501—
Mistral2-7B Standard RLTraining=RL, Inference=RAIL2026.030.5160.5050.379
Qwen3-8B TRACTTraining=SFT, Inference=RAIL2026.030.5140.5130.385
Prometheus-2-7BBackbone=Prometheus-2-7B, Inference Method=Decode2025.030.5120.493—
Prometheus-2-7BTraining=SFT, Inference=Standard2026.030.5120.4930.405
Mistral-7B-Instruct (RAFT on GPT-4)Backbone=Mistral-7B-Instruct, CoT=false, Train Objective=RAFT†, Training Data=GPT-4†, Inference Method=RAIL2025.030.5090.502—
Mistral2-7B TRACTTraining=SFT, Inference=RAIL2026.030.5070.50.372
Baseline B.4 (RAFT)Backbone=Llama3.1-8B, COT=false, Train=RAFT, Data=GPT-4+, Inf=RAIL2025.030.5060.493—
TRACTBackbone=Llama3.1-8B, COT=true, Train=C-RAFT, Data=Self, Inf=C-RAIL2025.030.50.493—
Qwen3-8B RAFTTraining=SFT, Inference=RAIL2026.030.4920.5010.375
Baseline B.2 (CE)Backbone=Llama3.1-8B, COT=true, Train=CE, Data=GPT-4, Inf=Decode2025.030.4750.484—
TRACT (Ablation: Objective CE)Backbone=Mistral-7B-Instruct, CoT=true, Train Objective=CE, Training Data=Self, Inference Method=C-RAIL2025.030.4680.436—
Prometheus-1-13BTraining=SFT, Inference=Standard2026.030.4660.4290.346
Prometheus-1-7BTraining=SFT, Inference=Standard2026.030.4570.4570.365
Qwen3-8B BaseTraining=None, Inference=RAIL2026.030.450.4830.385
TRACT (Ablation: Stage 1 Init)Backbone=Mistral-7B-Instruct, Note=Stage 2 initialized from ps, CoT=true2025.030.4480.437—
Qwen3-8B BaseTraining=None, Inference=Standard2026.030.4480.480.401
Baseline B.1 (CE)Backbone=Llama3.1-8B, COT=false, Train=CE, Data=GPT-4, Inf=Decode2025.030.4350.433—
Mistral2-7B Base (w/ warmup)Training=None, Inference=RAIL2026.030.4250.4370.323
TRACT (Ablation: Data GPT-4)Backbone=Mistral-7B-Instruct, CoT=true, Train Objective=C-RAFT, Training Data=GPT-4, Inference Method=C-RAIL2025.030.4180.419—
Mistral2-7B RAFTTraining=SFT, Inference=RAIL2026.030.4180.4190.315
Mistral2-7B Base (w/ warmup)Training=None, Inference=Standard2026.030.4150.4190.341
Mistral-7B-Instruct (CE on GPT-4 CoT)Backbone=Mistral-7B-Instruct, CoT=true, Train Objective=CE, Training Data=GPT-4, Inference Method=Decode2025.030.4130.407—
Baseline B.3 (RAIL)Backbone=Llama3.1-8B, COT=false, Inf=RAIL2025.030.4120.445—
Mistral-7B-Instruct (CE on Self-gen CoT)Backbone=Mistral-7B-Instruct, CoT=true, Train Objective=CE, Training Data=Self, Inference Method=Decode2025.030.3580.346—
Mistral-7B-Instruct (CE on GPT-4 scores)Backbone=Mistral-7B-Instruct, CoT=false, Train Objective=CE, Training Data=GPT-4†, Inference Method=Decode2025.030.3550.361—
GPT-3.5-Turbo-0613Training=None, Inference=Standard2026.030.270.2320.187
CLoud (Reward model)Backbone=Llama-3-8B, Inference Method=Decode*2025.030.2280.168—
Mistral-7B-Instruct (Zero-shot RAIL)Backbone=Mistral-7B-Instruct, CoT=false, Inference Method=RAIL2025.030.20.149—