Planning on UltraTool (test)
72.81n-F1GraphSAGE
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| GraphSAGELLM=CodeLlama-13B, Evaluation Protocol=1-shot, #Tok=10,6582024.05 | 72.81 | 45.26 | 43.49 | |
| BeamSearchLLM=GPT-4-turbo, Evaluation Protocol=1-shot, #Tok=20,5152024.05 | 71.99 | 44.54 | 31.62 | |
| SGCLLM=CodeLlama-13B, Evaluation Protocol=1-shot, #Tok=10,6582024.05 | 71.64 | 44 | 39.68 | |
| BeamSearchLLM=GPT-4-turbo, Evaluation Protocol=0-shot, #Tok=18,7932024.05 | 71.29 | 43.99 | 30.4 | |
| SGCLLM=GPT-4-turbo, Evaluation Protocol=0-shot, #Tok=8,3462024.05 | 70.87 | 44.01 | 31.6 | |
| GraphSAGELLM=GPT-4-turbo, Evaluation Protocol=1-shot, #Tok=8,5042024.05 | 70.75 | 47.68 | 37.22 | |
| GraphSAGELLM=GPT-4-turbo, Evaluation Protocol=0-shot, #Tok=8,3462024.05 | 70.67 | 43.83 | 34.4 | |
| GraphSAGELLM=GPT-3.5-turbo, Evaluation Protocol=1-shot, #Tok=8,5042024.05 | 70.49 | 47.79 | 39.74 | |
| SGCLLM=GPT-4-turbo, Evaluation Protocol=1-shot, #Tok=8,5042024.05 | 70.46 | 44.82 | 33 | |
| DirectLLM=GPT-4-turbo, Evaluation Protocol=1-shot, #Tok=8,6932024.05 | 69.54 | 41.79 | 28.17 | |
| DirectLLM=GPT-4-turbo, Evaluation Protocol=0-shot, #Tok=8,5132024.05 | 68.63 | 40.01 | 27.2 | |
| SGCLLM=GPT-3.5-turbo, Evaluation Protocol=1-shot, #Tok=8,5042024.05 | 64.96 | 37.96 | 29.7 | |
| BeamSearchLLM=CodeLlama-13B, Evaluation Protocol=1-shot, #Tok=23,0232024.05 | 64.93 | 36.23 | 33.47 | |
| GraphSAGELLM=GPT-3.5-turbo, Evaluation Protocol=0-shot, #Tok=8,3522024.05 | 63.97 | 42.26 | 30.35 | |
| GraphSAGELLM=CodeLlama-13B, Evaluation Protocol=0-shot, #Tok=10,4562024.05 | 63.78 | 39.91 | 27.98 | |
| DirectLLM=GPT-3.5-turbo, Evaluation Protocol=1-shot, #Tok=8,6142024.05 | 63.58 | 30.85 | 25 | |
| BeamSearchLLM=GPT-3.5-turbo, Evaluation Protocol=1-shot, #Tok=20,8132024.05 | 63.41 | 34.05 | 26.28 | |
| SGCLLM=CodeLlama-13B, Evaluation Protocol=0-shot, #Tok=10,4562024.05 | 61.07 | 37.61 | 25.31 | |
| SGCLLM=GPT-3.5-turbo, Evaluation Protocol=0-shot, #Tok=8,3522024.05 | 59.8 | 37.82 | 25.87 | |
| DirectLLM=CodeLlama-13B, Evaluation Protocol=1-shot, #Tok=10,7372024.05 | 57.64 | 30.44 | 26.25 | |
| BeamSearchLLM=GPT-3.5-turbo, Evaluation Protocol=0-shot, #Tok=21,9792024.05 | 55.4 | 28.02 | 19.76 | |
| DirectLLM=GPT-3.5-turbo, Evaluation Protocol=0-shot, #Tok=8,4622024.05 | 54.35 | 21.35 | 18.33 | |
| BeamSearchLLM=CodeLlama-13B, Evaluation Protocol=0-shot, #Tok=26,0082024.05 | 49.71 | 22.51 | 17.08 | |
| DirectLLM=CodeLlama-13B, Evaluation Protocol=0-shot, #Tok=10,5352024.05 | 38.88 | 16.42 | 13.58 |