Instruction Following on Vicuna Eval
66.3Win Rate (A)BPO-aligned claude-instant-1.2
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| BPO-aligned claude-instant-1.2Base LLM=claude-instant-1.2, Evaluator=gpt-42023.11 | 66.3 | 5 | 28.7 | |
| BPO-aligned text-bisonBase LLM=text-bison, Evaluator=gpt-42023.11 | 65 | 10 | 25 | |
| BPO-aligned vicuna-v1.3-7bBase LLM=vicuna-v1.3-7b, Reference Baseline=vicuna-v1.3-7b original, Evaluator=gpt-42023.11 | 65 | 8.7 | 26.3 | |
| BPOBase LLM=vicuna-v1.3, Method A=7B + BPO, Method B=7B, Evaluator=Claude-v1.32023.11 | 65 | — | 35 | |
| BPO-aligned llama-2-13b-chatBase LLM=llama-2-13b-chat, Reference Baseline=llama-2-13b-chat original, Evaluator=gpt-42023.11 | 61.3 | 2.5 | 36.2 | |
| BPO-aligned llama-2-13b-chat (scaling test)Base LLM=llama-2-13b-chat, Reference Baseline=llama-2-70b-chat original, Evaluator=gpt-42023.11 | 61.3 | 0 | 38.7 | |
| BPO-aligned gpt-3.5-turboBase LLM=gpt-3.5-turbo, Evaluator=gpt-42023.11 | 60 | 8.7 | 31.3 | |
| BPO-aligned llama-2-7b-chatBase LLM=llama-2-7b-chat, Reference Baseline=llama-2-7b-chat original, Evaluator=gpt-42023.11 | 60 | 2.5 | 37.5 | |
| BPO-aligned llama-2-70b-chatBase LLM=llama-2-70b-chat, Reference Baseline=llama-2-70b-chat original, Evaluator=gpt-42023.11 | 59.3 | 5.5 | 35.2 | |
| BPO-aligned claude-2Base LLM=claude-2, Evaluator=gpt-42023.11 | 57.5 | 5 | 37.5 | |
| BPOBase LLM=vicuna-v1.3, Method A=13B + BPO, Method B=13B, Evaluator=Claude-v1.32023.11 | 57.5 | — | 42.5 | |
| BPOBase LLM=llama-2-chat, Method A=7B + BPO, Method B=7B, Evaluator=Claude-v1.32023.11 | 55 | — | 45 | |
| BPO-aligned vicuna-v1.3-13bBase LLM=vicuna-v1.3-13b, Reference Baseline=vicuna-v1.3-13b original, Evaluator=gpt-42023.11 | 52.5 | 3.7 | 43.8 | |
| BPOBase LLM=llama-2-chat, Method A=13B + BPO, Method B=13B, Evaluator=Claude-v1.32023.11 | 52.5 | — | 47.5 | |
| BPOBase LLM=llama-2-chat, Method A=70B + BPO, Method B=70B, Evaluator=Claude-v1.32023.11 | 52.5 | — | 47.5 | |
| BPO-aligned llama-2-7b-chat (scaling test)Base LLM=llama-2-7b-chat, Reference Baseline=llama-2-70b-chat original, Evaluator=gpt-42023.11 | 48.8 | 3.7 | 47.5 | |
| BPOBase LLM=llama-2-chat, Method A=7B + BPO, Method B=70B, Evaluator=Claude-v1.32023.11 | 48.8 | — | 51.2 | |
| BPOBase LLM=llama-2-chat, Method A=13B + BPO, Method B=70B, Evaluator=Claude-v1.32023.11 | 46.3 | — | 53.7 | |
| BPO-aligned gpt-4Base LLM=gpt-4, Evaluator=gpt-42023.11 | 41.3 | 23.7 | 35 |