Persona-based Role-Playing Faithfulness Evaluation on Complicated Famous Figures (test)
0.9Spartacus ΔAPCEU
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| EUModel Family=Gemma-7B, PRP Strategy=Experience Uploading, APC-based DPO=false2024.05 | 0.9 | -1.1 | -5.5 | -3.2 | -1.6 | 6.8 | -0.7 | |
| EU + APC-based DPOModel Family=Gemma-7B, PRP Strategy=Experience Uploading, APC-based DPO=true2024.05 | 2.2 | 0.8 | -0.7 | -0.2 | -1.3 | 6.9 | 0.2 | |
| Gemma-7BModel Family=Gemma-7B, PRP Strategy=Vanilla, APC-based DPO=false2024.05 | 2.3 | 2.3 | 1.4 | 2.4 | 3.5 | 9.6 | 3.5 | |
| GPT-4Model Family=GPT, PRP Strategy=Vanilla, APC-based DPO=false2024.05 | 2.5 | 2.5 | -2 | 1.5 | 5.1 | 15.1 | 4.1 | |
| ChatGPTModel Family=GPT, PRP Strategy=Vanilla, APC-based DPO=false2024.05 | 2.6 | 1.4 | -3 | -0.6 | 1.7 | 11.9 | 2.3 | |
| Gemma-7B + APC-based DPOModel Family=Gemma-7B, PRP Strategy=Vanilla, APC-based DPO=true2024.05 | 2.9 | 3.2 | 4.8 | 2 | 3.1 | 18.1 | 5.6 | |
| RAG + APC-based DPOModel Family=Gemma-7B, PRP Strategy=Retrieval-Augmented Generation, APC-based DPO=true2024.05 | 3.4 | 3.9 | 5 | 3 | 6.4 | 19.9 | 6.9 | |
| RAGModel Family=Gemma-7B, PRP Strategy=Retrieval-Augmented Generation, APC-based DPO=false2024.05 | 3.6 | 3 | 3 | 3.4 | 5.4 | 16.3 | 5.7 |