LLM Alignment on OpenAssistant
87.08GPT-4o Win RateDPPrefSyn
Evaluation Results
| Method | Links | |
|---|---|---|
| DPPrefSynDownstream LLM=Qwen-3-4B, Generator LLM=Qwen-3-4B-Instruct, epsilon=∞2026.05 | 87.08 | |
| DPPrefSynDownstream LLM=Qwen-3-4B, Generator LLM=Qwen-3-4B-Instruct, epsilon=42026.05 | 86.52 | |
| DPPrefSynDownstream LLM=Llama-3-8B, Generator LLM=Qwen-3-4B-Instruct, epsilon=∞2026.05 | 61.52 | |
| DPPrefSynDownstream LLM=Llama-3-8B, Generator LLM=Qwen-3-4B-Instruct, epsilon=42026.05 | 60.25 | |
| DP-FTDownstream LLM=Qwen-3-4B, epsilon=∞2026.05 | 59.41 | |
| DP-FTDownstream LLM=Llama-3-8B, epsilon=∞2026.05 | 59.13 | |
| DP-FTDownstream LLM=Qwen-3-4B, epsilon=42026.05 | 54.92 | |
| DPPrefSynDownstream LLM=Qwen-3-4B, Generator LLM=Llama-3-8B-Instruct, epsilon=∞2026.05 | 52.11 | |
| DPPrefSynDownstream LLM=Qwen-3-4B, Generator LLM=Llama-3-8B-Instruct, epsilon=42026.05 | 51.69 | |
| Qwen-3-4BDownstream LLM=Qwen-3-4B, epsilon=02026.05 | 48.17 | |
| DP-FTDownstream LLM=Llama-3-8B, epsilon=42026.05 | 45.79 | |
| DPPrefSynDownstream LLM=Llama-3-8B, Generator LLM=Llama-3-8B-Instruct, epsilon=∞2026.05 | 44.1 | |
| DPPrefSynDownstream LLM=Llama-3-8B, Generator LLM=Llama-3-8B-Instruct, epsilon=42026.05 | 42.28 | |
| DPPrefSynDownstream LLM=Gemma-2-2B, Generator LLM=Qwen-3-4B-Instruct, epsilon=∞2026.05 | 37.08 | |
| DPPrefSynDownstream LLM=Gemma-2-2B, Generator LLM=Llama-3-8B-Instruct, epsilon=∞2026.05 | 36.66 | |
| DPPrefSynDownstream LLM=Gemma-2-2B, Generator LLM=Qwen-3-4B-Instruct, epsilon=42026.05 | 36.24 | |
| DPPrefSynDownstream LLM=Gemma-2-2B, Generator LLM=Llama-3-8B-Instruct, epsilon=42026.05 | 36.01 | |
| DP-FTDownstream LLM=Gemma-2-2B, epsilon=∞2026.05 | 33.85 | |
| DP-FTDownstream LLM=Gemma-2-2B, epsilon=42026.05 | 30.06 | |
| Llama-3-8BDownstream LLM=Llama-3-8B, epsilon=02026.05 | 16.15 | |
| DPPrefSynData Type=Synthetic, Preference Alignment=SFT+DPO, Privacy Budget (epsilon)=∞, Downstream Model=Pythia-2.8B2026.05 | 12.36 | |
| DPPrefSynData Type=Synthetic, Preference Alignment=SFT+DPO, Privacy Budget (epsilon)=8, Downstream Model=Pythia-2.8B2026.05 | 12.05 | |
| DPPrefSynData Type=Synthetic, Preference Alignment=SFT+DPO, Privacy Budget (epsilon)=4, Downstream Model=Pythia-2.8B2026.05 | 11.85 | |
| DPPrefSynData Type=Synthetic, Preference Alignment=SFT+DPO, Privacy Budget (epsilon)=2, Downstream Model=Pythia-2.8B2026.05 | 11.04 | |
| DPPrefSynData Type=Synthetic, Preference Alignment=SFT+DPO, Privacy Budget (epsilon)=1, Downstream Model=Pythia-2.8B2026.05 | 10.31 | |
| DPPrefSynData Type=Synthetic, Preference Alignment=SFT+DPO, Privacy Budget (epsilon)=0.5, Downstream Model=Pythia-2.8B2026.05 | 9.86 | |
| DPPrefSynData Type=Synthetic, Preference Alignment=SFT, Privacy Budget (epsilon)=∞, Downstream Model=Pythia-2.8B2026.05 | 9.8 | |
| DPPrefSynData Type=Synthetic, Preference Alignment=SFT, Privacy Budget (epsilon)=8, Downstream Model=Pythia-2.8B2026.05 | 9.63 | |
| DPPrefSynData Type=Synthetic, Preference Alignment=SFT, Privacy Budget (epsilon)=4, Downstream Model=Pythia-2.8B2026.05 | 9.33 | |
| DPPrefSynData Type=Synthetic, Preference Alignment=SFT, Privacy Budget (epsilon)=2, Downstream Model=Pythia-2.8B2026.05 | 9.1 | |
| DPPrefSynData Type=Synthetic, Preference Alignment=SFT, Privacy Budget (epsilon)=1, Downstream Model=Pythia-2.8B2026.05 | 9.07 | |
| DPPrefSynData Type=Synthetic, Preference Alignment=SFT, Privacy Budget (epsilon)=0.5, Downstream Model=Pythia-2.8B2026.05 | 8.96 | |
| DP-FTData Type=Original, Preference Alignment=SFT+DPO, Privacy Budget (epsilon)=∞, Downstream Model=Pythia-2.8B2026.05 | 8.2 | |
| Gemma-2-2BDownstream LLM=Gemma-2-2B, epsilon=02026.05 | 7.3 | |
| DP-FTData Type=Original, Preference Alignment=SFT+DPO, Privacy Budget (epsilon)=8, Downstream Model=Pythia-2.8B2026.05 | 6.32 | |
| DP-FTData Type=Original, Preference Alignment=SFT+DPO, Privacy Budget (epsilon)=2, Downstream Model=Pythia-2.8B2026.05 | 6.18 | |
| DP-FTData Type=Original, Preference Alignment=SFT+DPO, Privacy Budget (epsilon)=4, Downstream Model=Pythia-2.8B2026.05 | 6.12 | |
| DP-FTData Type=Original, Preference Alignment=SFT+DPO, Privacy Budget (epsilon)=1, Downstream Model=Pythia-2.8B2026.05 | 6.04 | |
| DP-FTData Type=Original, Preference Alignment=SFT+DPO, Privacy Budget (epsilon)=0.5, Downstream Model=Pythia-2.8B2026.05 | 5.65 | |
| DP-FTData Type=Original, Preference Alignment=SFT, Privacy Budget (epsilon)=∞, Downstream Model=Pythia-2.8B2026.05 | 4.75 | |
| DP-FTData Type=Original, Preference Alignment=SFT, Privacy Budget (epsilon)=8, Downstream Model=Pythia-2.8B2026.05 | 4.49 | |
| DP-FTData Type=Original, Preference Alignment=SFT, Privacy Budget (epsilon)=4, Downstream Model=Pythia-2.8B2026.05 | 4.46 | |
| DP-FTData Type=Original, Preference Alignment=SFT, Privacy Budget (epsilon)=2, Downstream Model=Pythia-2.8B2026.05 | 4.38 | |
| DP-FTData Type=Original, Preference Alignment=SFT, Privacy Budget (epsilon)=1, Downstream Model=Pythia-2.8B2026.05 | 4.29 | |
| DP-FTData Type=Original, Preference Alignment=SFT, Privacy Budget (epsilon)=0.5, Downstream Model=Pythia-2.8B2026.05 | 3.96 |