Judge Agreement on MT-Bench Second Turn 1.0
95Agreement RateGPT-4 (Pairwise)
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| GPT-4 (Pairwise)Setup=S2 (R=50%), Reference Judge=GPT-4 (Single-answer)2023.06 | 95 | 727 | |
| Expert LabelerSetup=S2 (R=50%), Reference Judge=Human Majority Vote2023.06 | 91 | 629 | |
| GPT-4 (Single-answer)Setup=S2 (R=50%), Reference Judge=Paper Author2023.06 | 89 | 193 | |
| GPT-4 (Pairwise)Setup=S2 (R=50%), Reference Judge=Paper Author2023.06 | 88 | 205 | |
| Paper AuthorSetup=S2 (R=50%), Reference Judge=Paper Author2023.06 | 87 | 31 | |
| Paper AuthorSetup=S2 (R=50%), Reference Judge=Expert Labeler2023.06 | 86 | 273 | |
| GPT-4 (Pairwise)Setup=S2 (R=50%), Reference Judge=Expert Labeler2023.06 | 85 | 864 | |
| GPT-4 (Pairwise)Setup=S2 (R=50%), Reference Judge=Human Majority Vote2023.06 | 85 | 557 | |
| GPT-4 (Single-answer)Setup=S2 (R=50%), Reference Judge=Human Majority Vote2023.06 | 85 | 506 | |
| GPT-4 (Single-answer)Setup=S2 (R=50%), Reference Judge=Expert Labeler2023.06 | 84 | 776 | |
| Paper AuthorSetup=S2 (R=50%), Reference Judge=Human Majority Vote2023.06 | 84 | 54 | |
| Expert LabelerSetup=S1 (R=33%), Reference Judge=Human Majority Vote2023.06 | 83 | 877 | |
| Expert LabelerSetup=S2 (R=50%), Reference Judge=Expert Labeler2023.06 | 82 | 474 | |
| GPT-4 (Pairwise)Setup=S1 (R=33%), Reference Judge=GPT-4 (Single-answer)2023.06 | 70 | 1,161 | |
| GPT-4 (Pairwise)Setup=S1 (R=33%), Reference Judge=Human Majority Vote2023.06 | 68 | 812 | |
| Paper AuthorSetup=S1 (R=33%), Reference Judge=Expert Labeler2023.06 | 68 | 413 | |
| Paper AuthorSetup=S1 (R=33%), Reference Judge=Paper Author2023.06 | 67 | 49 | |
| Expert LabelerSetup=S1 (R=33%), Reference Judge=Expert Labeler2023.06 | 67 | 707 | |
| GPT-4 (Pairwise)Setup=S1 (R=33%), Reference Judge=Paper Author2023.06 | 66 | 341 | |
| GPT-4 (Pairwise)Setup=S1 (R=33%), Reference Judge=Expert Labeler2023.06 | 66 | 1,325 | |
| GPT-4 (Single-answer)Setup=S1 (R=33%), Reference Judge=Paper Author2023.06 | 65 | 331 | |
| Paper AuthorSetup=S1 (R=33%), Reference Judge=Human Majority Vote2023.06 | 63 | 87 | |
| GPT-4 (Single-answer)Setup=S1 (R=33%), Reference Judge=Human Majority Vote2023.06 | 61 | 783 | |
| GPT-4 (Single-answer)Setup=S1 (R=33%), Reference Judge=Expert Labeler2023.06 | 59 | 1,285 | |
| G4-PSetup=S2 (R = 50%), Reference Judge=G4-S2023.06 | 0.95 | 727 | |
| HumanSetup=S2 (R = 50%), Reference Judge=Human-M2023.06 | 0.91 | 629 | |
| G4-SSetup=S2 (R = 50%), Reference Judge=Author2023.06 | 0.89 | 193 | |
| G4-PSetup=S2 (R = 50%), Reference Judge=Author2023.06 | 0.88 | 205 | |
| AuthorSetup=S2 (R = 50%), Reference Judge=Human2023.06 | 0.87 | 273 | |
| AuthorSetup=S2 (R = 50%), Reference Judge=Human-M2023.06 | 0.86 | 54 | |
| G4-PSetup=S2 (R = 50%), Reference Judge=Human2023.06 | 0.85 | 864 | |
| G4-PSetup=S2 (R = 50%), Reference Judge=Human-M2023.06 | 0.85 | 557 | |
| G4-SSetup=S2 (R = 50%), Reference Judge=Human-M2023.06 | 0.85 | 506 | |
| G4-SSetup=S2 (R = 50%), Reference Judge=Human2023.06 | 0.84 | 776 | |
| HumanSetup=S1 (R = 33%), Reference Judge=Human-M2023.06 | 0.83 | 877 | |
| HumanSetup=S2 (R = 50%), Reference Judge=Human2023.06 | 0.82 | 474 | |
| G4-PSetup=S1 (R = 33%), Reference Judge=G4-S2023.06 | 0.7 | 1,161 | |
| G4-PSetup=S1 (R = 33%), Reference Judge=Human-M2023.06 | 0.68 | 812 | |
| AuthorSetup=S1 (R = 33%), Reference Judge=Human2023.06 | 0.68 | 413 | |
| HumanSetup=S1 (R = 33%), Reference Judge=Human2023.06 | 0.67 | 707 | |
| G4-PSetup=S1 (R = 33%), Reference Judge=Author2023.06 | 0.66 | 341 | |
| G4-PSetup=S1 (R = 33%), Reference Judge=Human2023.06 | 0.66 | 1,325 | |
| G4-SSetup=S1 (R = 33%), Reference Judge=Author2023.06 | 0.65 | 331 | |
| AuthorSetup=S1 (R = 33%), Reference Judge=Human-M2023.06 | 0.63 | 87 | |
| G4-SSetup=S1 (R = 33%), Reference Judge=Human-M2023.06 | 0.61 | 783 | |
| G4-SSetup=S1 (R = 33%), Reference Judge=Human2023.06 | 0.59 | 1,285 |