Rule Recovery on Non-judge output abstractions
1MRR@1RuleSHAP
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| RuleSHAPLLM=GPT-4o, Rules=17292025.05 | 1 | 1 | 1 | |
| RuleFitLLM=GPT-4o, Rules=17782025.05 | 0.83 | 0.83 | 0.83 | |
| RuleSHAPLLM=GPT-3.5 Turbo, Rules=5782025.05 | 0.67 | 0.67 | 0.67 | |
| RuleFitLLM=GPT-4o mini, Rules=11582025.05 | 0.67 | 0.67 | 0.67 | |
| RuleSHAPLLM=GPT-4o mini, Rules=9612025.05 | 0.67 | 0.75 | 0.75 | |
| RuleSHAP w/o S2LLM=GPT-4o, Rules=21402025.05 | 0.67 | 0.67 | 0.67 | |
| RuleSHAPLLM=Llama-3.1 8B, Rules=11662025.05 | 0.67 | 0.75 | 0.75 | |
| RuleSHAPLLM=Llama-3.1 70B, Rules=7512025.05 | 0.67 | 0.72 | 0.72 | |
| RuleFitLLM=GPT-3.5 Turbo, Rules=6682025.05 | 0.5 | 0.5 | 0.54 | |
| RuleFitLLM=Llama-3.1 8B, Rules=13542025.05 | 0.5 | 0.67 | 0.67 | |
| RuleFitLLM=Llama-3.1 70B, Rules=7662025.05 | 0.33 | 0.42 | 0.42 | |
| RuleSHAP w/o S2LLM=Llama-3.1 70B, Rules=9212025.05 | 0.33 | 0.33 | 0.33 | |
| RuleSHAP w/o S2LLM=GPT-3.5 Turbo, Rules=5512025.05 | 0.17 | 0.17 | 0.17 | |
| RuleSHAP w/o S2LLM=GPT-4o mini, Rules=11122025.05 | 0.17 | 0.17 | 0.21 | |
| RuleSHAP w/o S2LLM=Llama-3.1 8B, Rules=14812025.05 | 0.17 | 0.17 | 0.17 | |
| TreeLLM=GPT-3.5 Turbo, Rules=7222025.05 | 0 | 0 | 0 | |
| TreeLLM=GPT-4o mini, Rules=7542025.05 | 0 | 0 | 0 | |
| TreeLLM=GPT-4o, Rules=7872025.05 | 0 | 0 | 0 | |
| TreeLLM=Llama-3.1 8B, Rules=7802025.05 | 0 | 0 | 0 | |
| TreeLLM=Llama-3.1 70B, Rules=3742025.05 | 0 | 0 | 0 |