Human Preference Prediction on Chatbot Arena latest (test)
72.18AccuracySynthesizeMe (+ FT RM + Personas)
Evaluation results
| Method | Links | |
|---|---|---|
| SynthesizeMe (+ FT RM + Personas)Evaluation Protocol=Finetuned Reward Model, Backbone=Llama 3.3 70B2025.06 | 72.18 | |
| SynthesizeMe (+ FT RM + Personas + Demos)Evaluation Protocol=Finetuned Reward Model, Backbone=Llama 3.3 70B2025.06 | 72.18 | |
| Finetuned Reward ModelEvaluation Protocol=Bradley-Terry Reward Model, Backbone=Llama 3.3 70B2025.06 | 71.48 | |
| SkillAggregation-XJudge Model Size=~70B, Number of Runs=52024.10 | 70.72 | |
| SkillAggregationJudge Model Size=~70B, Number of Runs=52024.10 | 70.66 | |
| Average ProbabilityJudge Model Size=~70B2024.10 | 70.65 | |
| DawidSkeneJudge Model Size=~70B2024.10 | 70.64 | |
| SkillAggregation w/o. Reg.Judge Model Size=~70B, Number of Runs=52024.10 | 70.62 | |
| Majority VotingJudge Model Size=~70B2024.10 | 70.61 | |
| Train on Majority VotingJudge Model Size=~70B, Number of Runs=52024.10 | 70.53 | |
| CrowdlayerJudge Model Size=~70B, Number of Runs=52024.10 | 70.38 | |
| SynthesizeMe (+ FT RM + Personas + Demos)Evaluation Protocol=Finetuned Reward Model, Backbone=Llama 3.1 8B2025.06 | 69.72 | |
| Finetuned Reward ModelEvaluation Protocol=Bradley-Terry Reward Model, Backbone=Llama 3.2 3B2025.06 | 69.01 | |
| SynthesizeMe (+ FT RM + Personas)Evaluation Protocol=Finetuned Reward Model, Backbone=Llama 3.2 3B2025.06 | 69.01 | |
| Finetuned Reward ModelEvaluation Protocol=Bradley-Terry Reward Model, Backbone=Llama 3.1 8B2025.06 | 68.31 | |
| SynthesizeMe (+ FT RM + Personas)Evaluation Protocol=Finetuned Reward Model, Backbone=Llama 3.1 8B2025.06 | 67.25 | |
| SynthesizeMe (+ FT RM + Personas + Demos)Evaluation Protocol=Finetuned Reward Model, Backbone=Llama 3.2 3B2025.06 | 66.55 | |
| DawidSkeneJudge Model Size=7B/8B2024.10 | 64.71 | |
| SkillAggregation-XJudge Model Size=7B/8B, Number of Runs=52024.10 | 64.43 | |
| SkillAggregationJudge Model Size=7B/8B, Number of Runs=52024.10 | 64.22 | |
| SkillAggregation w/o. Reg.Judge Model Size=7B/8B, Number of Runs=52024.10 | 64.17 | |
| CrowdlayerJudge Model Size=7B/8B, Number of Runs=52024.10 | 64.06 | |
| Majority VotingJudge Model Size=7B/8B2024.10 | 63.93 | |
| Train on Majority VotingJudge Model Size=7B/8B, Number of Runs=52024.10 | 63.77 | |
| Average ProbabilityJudge Model Size=7B/8B2024.10 | 63.24 | |
| SynthesizeMe (Just Demos)Evaluation Protocol=In-Context LLM as a Judge, Backbone=Llama 3.3 70B2025.06 | 61.97 | |
| SynthesizeMe (Personas + Demos)Evaluation Protocol=In-Context LLM as a Judge, Backbone=Llama 3.3 70B2025.06 | 61.97 | |
| SynthesizeMe (Personas + Demos + Distill)Evaluation Protocol=In-Context LLM as a Judge, Backbone=Llama 3.1 8B2025.06 | 61.62 | |
| PALEvaluation Protocol=Finetuned Reward Model, Backbone=Llama 3.2 3B2025.06 | 60.56 | |
| SynthesizeMe (Personas + Distill)Evaluation Protocol=In-Context LLM as a Judge, Backbone=Llama 3.1 8B2025.06 | 59.15 | |
| MemoryEvaluation Protocol=In-Context LLM as a Judge, Backbone=Llama 3.1 8B2025.06 | 58.1 | |
| GPOEvaluation Protocol=Finetuned Reward Model, Backbone=Llama 3.3 70B2025.06 | 58.1 | |
| SynthesizeMe (Personas + Demos)Evaluation Protocol=In-Context LLM as a Judge, Backbone=Llama 3.1 8B2025.06 | 57.92 | |
| MemoryEvaluation Protocol=In-Context LLM as a Judge, Backbone=Llama 3.3 70B2025.06 | 57.57 | |
| SynthesizeMe (Just Personas)Evaluation Protocol=In-Context LLM as a Judge, Backbone=Llama 3.1 8B2025.06 | 57.39 | |
| DefaultEvaluation Protocol=In-Context LLM as a Judge, Backbone=Llama 3.3 70B2025.06 | 56.69 | |
| GPOEvaluation Protocol=Finetuned Reward Model, Backbone=Llama 3.1 8B2025.06 | 56.69 | |
| VPLEvaluation Protocol=Finetuned Reward Model, Backbone=Llama 3.2 3B2025.06 | 56.69 | |
| PALEvaluation Protocol=Finetuned Reward Model, Backbone=Llama 3.1 8B2025.06 | 56.69 | |
| SynthesizeMe (Just Demos)Evaluation Protocol=In-Context LLM as a Judge, Backbone=Llama 3.1 8B2025.06 | 55.11 | |
| VPLEvaluation Protocol=Finetuned Reward Model, Backbone=Llama 3.1 8B2025.06 | 54.93 | |
| SynthesizeMe (Personas + Distill)Evaluation Protocol=In-Context LLM as a Judge, Backbone=Llama 3.2 3B2025.06 | 54.75 | |
| DefaultEvaluation Protocol=In-Context LLM as a Judge, Backbone=Llama 3.2 3B2025.06 | 54.23 | |
| GPOEvaluation Protocol=Finetuned Reward Model, Backbone=Llama 3.2 3B2025.06 | 53.87 | |
| DefaultEvaluation Protocol=In-Context LLM as a Judge, Backbone=Llama 3.1 8B2025.06 | 53.7 | |
| SynthesizeMe (Just Personas)Evaluation Protocol=In-Context LLM as a Judge, Backbone=Llama 3.3 70B2025.06 | 53.7 | |
| SynthesizeMe (Just Demos)Evaluation Protocol=In-Context LLM as a Judge, Backbone=Llama 3.2 3B2025.06 | 53.17 | |
| SynthesizeMe (Personas + Demos)Evaluation Protocol=In-Context LLM as a Judge, Backbone=Llama 3.2 3B2025.06 | 52.46 | |
| MemoryEvaluation Protocol=In-Context LLM as a Judge, Backbone=Llama 3.2 3B2025.06 | 52.29 | |
| SynthesizeMe (Just Personas)Evaluation Protocol=In-Context LLM as a Judge, Backbone=Llama 3.2 3B2025.06 | 50.88 | |
| RandomEvaluation Protocol=Baseline2025.06 | 50 |