Binary Code Summarization on BCS 500 samples (test)
0.8595Win RateC-ing Clearly
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| C-ing ClearlyOpponent=Llama-3.1-8B-Instruct, Judge=GPT-4o, Sampling=Nucleus, Temperature=0.42025.12 | 0.8595 | — | |
| No C-ingOpponent=Llama-3.1-8B-Instruct, Judge=GPT-4o, Sampling=Nucleus, Temperature=0.42025.12 | 0.826 | — | |
| Multi-turn C-ing ClearlyOpponent=Llama-3.1-8B-Instruct, Judge=GPT-4o, Sampling=Nucleus, Temperature=0.42025.12 | 0.784 | — | |
| C-ing ClearlyOpponent=Multi-turn C-ing Clearly (MCC), Judge=GPT-4o, Sampling=Nucleus, Temperature=0.42025.12 | 0.5745 | — | |
| C-ing ClearlyOpponent=No C-ing (NOC), Judge=GPT-4o, Sampling=Nucleus, Temperature=0.42025.12 | 0.5709 | — | |
| No C-ingOpponent=Multi-turn C-ing Clearly (MCC), Judge=GPT-4o, Sampling=Nucleus, Temperature=0.42025.12 | 0.524 | — | |
| Multi-turn C-ing ClearlyOpponent=No C-ing (NOC), Judge=GPT-4o, Sampling=Nucleus, Temperature=0.42025.12 | 0.4665 | — | |
| No C-ingOpponent=C-ing Clearly (CC), Judge=GPT-4o, Sampling=Nucleus, Temperature=0.42025.12 | 0.4115 | — | |
| Multi-turn C-ing ClearlyOpponent=C-ing Clearly (CC), Judge=GPT-4o, Sampling=Nucleus, Temperature=0.42025.12 | 0.4015 | — | |
| Llama-3.1-8B-InstructOpponent=Multi-turn C-ing Clearly (MCC), Judge=GPT-4o, Sampling=Nucleus, Temperature=0.42025.12 | 0.2105 | — | |
| Llama-3.1-8B-InstructOpponent=No C-ing (NOC), Judge=GPT-4o, Sampling=Nucleus, Temperature=0.42025.12 | 0.1645 | — | |
| Llama-3.1-8B-InstructOpponent=C-ing Clearly (CC), Judge=GPT-4o, Sampling=Nucleus, Temperature=0.42025.12 | 0.133 | — | |
| C-ing ClearlyOpponent=Average, Judge=GPT-4o, Sampling=Nucleus, Temperature=0.42025.12 | — | 66.8 | |
| Llama-3.1-8B-InstructOpponent=Average, Judge=GPT-4o, Sampling=Nucleus, Temperature=0.42025.12 | — | 16.93 | |
| Multi-turn C-ing ClearlyOpponent=Average, Judge=GPT-4o, Sampling=Nucleus, Temperature=0.42025.12 | — | 55.07 | |
| No C-ingOpponent=Average, Judge=GPT-4o, Sampling=Nucleus, Temperature=0.42025.12 | — | 58.71 |